Patent Description:
There is a technology for detecting a position and a posture of a device worn by a user by analyzing an image of a camera, and controlling display of augmented reality (AR) according to a gesture of the user. The device worn by the user will be used as an AR controller that controls the display of AR.

Patent Document <NUM> discloses a technique in which a plane on a housing is specified by detecting markers prepared at four corners of the housing attached to a bracelet worn by a user, and a virtual stereoscopic image is synthesized with the specified plane and displayed.

<CIT> discloses a wearable display device using an augmented reality interface. The disclosure comprises an interface image generator part configured to generate a graphic interface image using a recognized watch as a reference and to show a graphic interface image in the acquired image.

In the case of the above-described technology, the user needs to wear a dedicated bracelet.

Since it is necessary to print a complicated texture such as a marker on the housing, the fashion of the device is impaired. In addition, since it is necessary to detect a plurality of markers, a processing amount for extracting a feature point increases.

The present technology has been made in view of such a situation, and enables an operation to be performed using a wearable device that is less resistant to being worn at all times.

According to a first aspect, the present invention provides an information processing apparatus in accordance with independent claim <NUM>. According to a second aspect, the present invention provides an information processing method in accordance with independent claim <NUM>. According to a third aspect, the present invention provides a program in accordance with independent claim <NUM>. Further aspects of the invention are set forth in the dependent claims, the drawings, and the following description.

An information processing apparatus according to one aspect of the present technology includes: an acquisition unit that acquires a captured image obtained by capturing an image of a wearable device including a module whose appearance changes according to time; and an estimation unit that detects the module appearing in the captured image on the basis of an image indicating an appearance of the module according to a current time, and estimates at least one of a position or a posture of the wearable device.

In one aspect of the present technology, a captured image obtained by capturing an image of a wearable device including a module whose appearance changes according to time is acquired, the module appearing in the captured image is detected on the basis of an image representing an appearance of the module according to a current time, and at least one of a position or a posture of the wearable device is estimated.

Hereinafter, modes for carrying out the present technology will be described. Further, the description will be given in the following order.

<FIG> is a diagram illustrating a wearing example of a wearable device according to an embodiment of the present technology.

As illustrated in <FIG>, an information processing terminal <NUM> is a glasses-type wearable device including a transmissive display unit. The information processing terminal <NUM> causes the display unit to display various types of information read from a built-in memory or acquired via the Internet, for example.

The information processing terminal <NUM> is a wearable device equipped with an AR function for displaying various types of information superimposed on an actual scene. The user sees various types of information such as characters and images superimposed on the scenery in front of the user.

The display method by the AR function may be a virtual image projection method or a retinal projection method in which an image is directly formed on the retina of the user's eye. In addition, the image displayed to be superimposed on the actual scene may be a two-dimensional image or a three-dimensional image in which an object such as a character appears stereoscopically.

The information processing terminal <NUM> is provided with a camera that captures an image of the front of the user wearing the head. The information processing terminal <NUM> analyzes the image captured by the camera to estimate the position and posture of a wristwatch <NUM> worn by the user, and detects a gesture operation on the basis of the transition of the position and posture.

The information processing terminal <NUM> performs various processes such as controlling the content displayed by the AR function and controlling an external device according to the gesture operation of the user.

That is, the wristwatch <NUM>, which is a wearable device, is used as the AR controller of the information processing terminal <NUM> equipped with the AR function. The user can operate the information processing terminal <NUM> by moving the left arm wearing the wristwatch <NUM>.

<FIG> is a front view illustrating an appearance of the wristwatch <NUM>.

As illustrated in <FIG>, the wristwatch <NUM> is an analog type wristwatch. A long hand <NUM>-<NUM> and a short hand <NUM>-<NUM> are provided on a dial <NUM> of a case <NUM> having a substantially perfect circle. Hereinafter, when it is not necessary to distinguish the long hand <NUM>-<NUM> and the short hand <NUM>-<NUM> as appropriate, they are collectively referred to as a clock hand <NUM>.

Three chronographs are provided at positions near the center of the dial <NUM>, and indexes such as scales and numbers are provided along the periphery. Furthermore, a logo is printed above the center of the dial <NUM>. In the example of <FIG>, a logo of "AAAA" is printed.

The information processing terminal <NUM> in <FIG> estimates the position and posture of such an analog type wristwatch <NUM> on the basis of the captured image obtained by capturing with the camera. Since the direction pointed by the clock hand <NUM> changes according to the time, the wristwatch <NUM> is a wearable device having a dial <NUM> whose appearance changes according to the time.

In order to estimate the position and posture of the wristwatch <NUM>, the information processing terminal <NUM> needs to detect a predetermined module of the wristwatch <NUM>, such as the dial <NUM>, from among objects shown in the captured image. The information processing terminal <NUM> estimates the position and posture of the dial <NUM> in the three-dimensional space on the basis of the detected position and shape of the dial <NUM> in the captured image, the direction of the clock hand <NUM> provided on the dial <NUM>, and the like. The position and posture of the dial <NUM> represent the position and posture of the wristwatch <NUM>.

<FIG> is a diagram illustrating an example of a template image used to detect the dial <NUM>.

Here, the template image is an image used for matching with each part of the captured image (detection of a region in which the same object is captured). The dial <NUM> is detected by pattern matching using a template image.

In the template image illustrated in A of <FIG>, a region outside the perfect circular dial <NUM> in the image representing the appearance of the wristwatch <NUM> and a region of the clock hand <NUM> representing "<NUM>:<NUM>" in the region inside the dial <NUM> are masked. The image representing the appearance of the wristwatch <NUM> used to generate the template image is, for example, an image registered in advance at the time of calibration before detecting the dial <NUM>. In <FIG>, a region denoted by black is a mask region.

In addition, in the template image illustrated in A of <FIG>, an image of the clock hand <NUM> representing "<NUM>:<NUM>" is combined with a region of the dial <NUM>. A clock hand indicating "<NUM>:<NUM>" hatched is an image of the clock hand <NUM> synthesized by the image processing.

The template image illustrated in A of <FIG> is an image used when the dial <NUM> is detected at "<NUM>:<NUM>".

Similarly, in the template image illustrated in B of <FIG>, the region outside the dial <NUM> and the region of the clock hand <NUM> representing "<NUM>:<NUM>" in the region inside the dial <NUM> are masked.

In addition, in the template image illustrated in B of <FIG>, an image of the clock hand <NUM> representing "<NUM>:<NUM>" is combined with the region of the dial <NUM>.

The template image illustrated in B of <FIG> is an image used when the dial <NUM> is detected at "<NUM>:<NUM>".

Similarly, in the template image illustrated in C of <FIG>, the region outside the dial <NUM> and the region of the clock hand <NUM> indicating "<NUM>:<NUM>" in the region inside the dial <NUM> are masked.

In addition, in the template image illustrated in C of <FIG>, an image of the clock hand <NUM> representing "<NUM>:<NUM>" is combined with the region of the dial <NUM>.

A template image illustrated in C of <FIG> is an image used when the dial <NUM> is detected at "<NUM>:<NUM>".

As described above, in the information processing terminal <NUM>, the time at which the dial <NUM> is detected is set as the current time, and the image of the dial <NUM> representing the current time is generated as the template image for each current time and used for matching.

The information processing terminal <NUM> can estimate the position of the dial <NUM> by using the image of the dial <NUM> indicating the current time as the template image. Furthermore, the information processing terminal <NUM> can estimate the posture of the dial <NUM> from the direction of the clock hand <NUM>.

That is, the information processing terminal <NUM> can estimate the position and posture of the wristwatch <NUM>.

In addition, since a complicated texture such as a marker used for detecting the position and the posture is unnecessary, the user can use a normal wristwatch as the AR controller without impairing the fashion.

Details of processing such as generation of the template image and matching using the template image will be described later.

<FIG> is a block diagram illustrating a configuration example of the information processing terminal <NUM>.

As illustrated in <FIG>, the information processing terminal <NUM> is configured by connecting a camera <NUM>, a sensor <NUM>, a communication unit <NUM>, a display unit <NUM>, and a memory <NUM> to a controller <NUM>.

The controller <NUM> includes a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and the like. The controller <NUM> executes a program stored in the ROM or the memory <NUM> and controls the overall operation of the information processing terminal <NUM>.

For example, the controller <NUM> estimates the position and posture of the wristwatch <NUM> on the basis of the captured image supplied from the camera <NUM>, and detects the gesture operation of the user. The controller <NUM> executes various processes in accordance with the detected gesture operation.

The camera <NUM> captures a scenery in front of the user. The camera <NUM> outputs a captured image obtained by performing capturing to the controller <NUM>.

Note that the camera <NUM> is a camera for visible light detection. The image captured by the camera <NUM> is an RGB image.

The image captured by the camera <NUM> is also used, for example, to estimate the position and posture of the information processing terminal <NUM> itself.

The sensor <NUM> includes a depth sensor or light detection and ranging (LiDAR). The depth sensor and the LiDAR constituting the sensor <NUM> measure the distance to each position of the object, and output a distance image, point cloud data, and the like to the controller <NUM> as data indicating a measurement result.

The sensor <NUM> appropriately includes various sensors such as an acceleration sensor, a gyro sensor, and a positioning sensor. The measurement results of the acceleration sensor, the gyro sensor, and the positioning sensor are used for estimation of the position, posture, and the like of the information processing terminal <NUM>.

The communication unit <NUM> includes a communication module of a portable communication network, a communication module of a wireless LAN, and the like. The communication unit <NUM> communicates with an external device via a network.

The display unit <NUM> displays various types of information such as characters and images as AR content under the control of the controller <NUM>.

The memory <NUM> is a storage medium such as a flash memory. The memory <NUM> stores various data such as a program executed by the CPU of the controller <NUM>.

<FIG> is a block diagram illustrating a functional configuration example of the controller <NUM>.

At least a part of the functional units illustrated in <FIG> is realized by executing a predetermined program by the CPU constituting the controller <NUM> in <FIG>.

As illustrated in <FIG>, the controller <NUM> includes an image acquisition unit <NUM>, a time synchronization unit <NUM>, a mask image generation unit <NUM>, a template image generation unit <NUM>, a position/posture estimation unit <NUM>, and a control unit <NUM>.

The image acquisition unit <NUM> acquires a photographed image obtained by photographing by the camera <NUM>. The captured image acquired at the time of calibration is supplied to the mask image generation unit <NUM>, and the captured image acquired at the time of matching is supplied to the position/posture estimation unit <NUM>.

The time synchronization unit <NUM> synchronizes the time of the internal clock of the information processing terminal <NUM> with the time of the wristwatch <NUM>. The time information of the internal clock synchronized with the time of the wristwatch <NUM> is supplied to the template image generation unit <NUM>.

The time synchronization is established, for example, by both the information processing terminal <NUM> and the wristwatch <NUM> receiving radio waves including information on the standard time.

Both the information processing terminal <NUM> and the wristwatch <NUM> may access a network time protocol (NTP) server, and time synchronization may be established on the basis of information received from the NTP server.

In a case where the wristwatch <NUM> has a radio communication function, radio communication may be performed between the information processing terminal <NUM> and the wristwatch <NUM> to establish time synchronization.

The time of the wristwatch <NUM> may be detected by analyzing the image captured by the camera <NUM>, and a deviation from the time of the internal clock may be specified. According to the deviation between the time of the wristwatch <NUM> and the time of the internal clock, the information of the current time used when the matching is performed is corrected. Furthermore, not only the deviation between the time of the wristwatch <NUM> and the time of the internal clock, but also the design of the hand or the panel of the wristwatch owned by the user may be stored.

The mask image generation unit <NUM> generates a mask image used for generating a template image on the basis of the captured image supplied from the image acquisition unit <NUM>. The mask image generated by the mask image generation unit <NUM> is supplied to the template image generation unit <NUM>.

On the basis of the mask image supplied from the mask image generation unit <NUM>, the template image generation unit <NUM> generates a template image for each time at which the dial <NUM> is detected. The template image generated by the template image generation unit <NUM> is supplied to the position/posture estimation unit <NUM>.

The position/posture estimation unit <NUM> performs matching between each part of the captured image supplied from the image acquisition unit <NUM> and the template image generated by the template image generation unit <NUM>, and detects the dial <NUM> appearing in the captured image. The position/posture estimation unit <NUM> estimates the position and posture of the wristwatch <NUM> on the basis of the position and shape of the dial <NUM> appearing in the captured image, the direction of the clock hand <NUM>, and the like. Information indicating the estimation results of the position and posture of the wristwatch <NUM> is supplied to the control unit <NUM>. Instead of estimating both the position and posture of the wristwatch <NUM>, only one of them may be estimated.

The control unit <NUM> performs various processes such as controlling the content displayed by the AR function and controlling an external device on the basis of the estimation result of the position and posture of the wristwatch <NUM>. The control unit <NUM> functions as a display control unit that controls display of various types of information by AR, and also functions as an operation control unit that controls operation of an external device.

Here, the operation of the information processing terminal <NUM> having the above configuration will be described.

First, the calibration process will be described with reference to the flowchart of <FIG>.

The calibration process is, for example, a process performed in advance before the position and posture of the wristwatch <NUM> are estimated.

In step S11, the controller <NUM> causes the display unit <NUM> to display a guide image serving as a guide of the dial region.

In step S12, the image acquisition unit <NUM> causes the camera <NUM> to capture the wristwatch <NUM> whose position is guided according to the guide image.

<FIG> is a diagram illustrating a display example of a guide image.

An image G displayed at a position P1 in the upper part of <FIG> is a guide image serving as a guide of the dial region. In the example of <FIG>, a circular image G of a predetermined color is displayed as a guide image.

Such a guide image is displayed to be superimposed on the scenery in front while the display position is fixed. The user moves his/her arm so that the dial <NUM> of the wristwatch <NUM> fits inside the guide image.

As indicated by the tip of arrow A1, photographing is performed in a state where the dial <NUM> of the wristwatch <NUM> is inside the guide image. The photographed image obtained by photographing the wristwatch <NUM> is supplied to the mask image generation unit <NUM> together with the information of the range of the guide image.

In step S13 of <FIG>, the mask image generation unit <NUM> registers the captured image supplied from the image acquisition unit <NUM>. The captured image showing the wristwatch <NUM> is managed as a registered image.

In step S14, the mask image generation unit <NUM> detects the long hand <NUM>-<NUM> and the short hand <NUM>-<NUM> by analyzing the region of the dial <NUM> on which the guide image is superimposed in the registered image. The mask image generation unit <NUM> generates a mask image that masks the region outside the dial <NUM> and the region of the clock hand <NUM> inside the dial <NUM>.

<FIG> is a diagram illustrating an example of generating a mask image.

As illustrated on the left side of <FIG>, the registered image shows the wristwatch <NUM> photographed by aligning the position of the dial <NUM> with the guide image.

The clock hand <NUM> representing "<NUM>:<NUM>" is detected on the basis of such a registered image, and a mask image that masks the region outside the dial <NUM> and the region of the clock hand <NUM> as indicated by a tip of arrow A11 is generated.

Note that, in a case where the calibration time is, for example, "<NUM>:<NUM>", the clock hand <NUM> is at a position indicating "<NUM>:<NUM>". The clock hand <NUM> may be detected in a predetermined range based on a position indicating "<NUM>:<NUM>". Therefore, the detection accuracy of the clock hand <NUM> can be enhanced.

After the mask image is generated, the calibration process ends.

Next, a position/posture estimation process will be described with reference to the flowchart of <FIG>.

The position/posture estimation process in <FIG> is started, for example, after the calibration process is performed and when a gesture operation of the user is detected.

In step S31, the time synchronization unit <NUM> synchronizes the time of the internal clock of the information processing terminal <NUM> with the time of the wristwatch <NUM>.

In step S32, the image acquisition unit <NUM> causes the camera <NUM> to capture an image and acquires a captured image. The captured image acquired by the image acquisition unit <NUM> is supplied to the position/posture estimation unit <NUM>.

In step S33, the template image generation unit <NUM> refers to the time information supplied from the time synchronization unit <NUM> and generates a template image on the basis of the current time.

<FIG> is a diagram illustrating an example of generation of a template image.

In a case where the registered image illustrated on the left side of <FIG> is prepared, the template image generation unit <NUM> masks the region outside the dial <NUM> and the region of the clock hand <NUM> on the basis of the mask image as indicated by a tip of arrow A21.

The masked image includes information on the region inside the dial <NUM> other than the region of the clock hand <NUM> representing, for example, "<NUM>:<NUM>" that is the calibration time.

In addition, the template image generation unit <NUM> generates a template image by synthesizing an image of the clock hand <NUM> representing the current time with the masked image as indicated by a tip of arrow A22.

For example, in a case where the current time is "<NUM>:<NUM>", an image of the clock hand <NUM> representing "<NUM>:<NUM>" is synthesized. In the template image illustrated at the right end of <FIG>, a hand indicating "<NUM>:<NUM>" hatched is an image synthesized by image processing.

Returning to the description of <FIG>, in step S34, the position/posture estimation unit <NUM> performs matching with each portion of the captured image using the template image generated by the template image generation unit <NUM>. For example, a perfect circular template image as illustrated in <FIG> is deformed so as to represent appearance when viewed from various angles, and matching is performed using the deformed template image.

In step S35, the position/posture estimation unit <NUM> determines whether or not the dial <NUM> has been detected by matching using the template image.

In a case where the dial <NUM> is not detected, the processing returns to step S32, and the above processing is repeated. The template image is updated, and matching is repeated using the updated template image.

In a case where it is determined in step S35 that the dial <NUM> has been detected, in step S36, the position/posture estimation unit <NUM> estimates the position and posture of the dial <NUM> on the basis of the position and shape of the dial <NUM> appearing in the captured image, the direction of the clock hand <NUM>, and the like.

In step S37, the position/posture estimation unit <NUM> updates information on the position and posture of the wristwatch <NUM> represented by the position and posture of the dial <NUM>. Thereafter, the processing returns to step S32, and the above processing is repeated.

By repeating the above processing, the control unit <NUM> detects the gesture operation of the user on the basis of the time series of the position and posture of the wristwatch <NUM>.

Through the above processing, the information processing terminal <NUM> can easily estimate the position and posture of the wristwatch <NUM>.

For example, in a case where the processing is performed on the basis of only the position and shape (perfect circle or ellipse) of the dial <NUM> appearing in the captured image, the information processing terminal <NUM> can estimate the position of the dial <NUM>, but cannot estimate the direction of the wristwatch <NUM> in the vertical direction or the like.

By considering the direction of the hand indicating the current time, the information processing terminal <NUM> can also estimate the direction of the wristwatch <NUM> in the vertical direction or the like.

In addition, the user can use a normal wristwatch as the AR controller.

In a case where it is assumed that the information processing terminal <NUM> is always worn indoors and outdoors, the user is required to always wear the AR controller. By using the wristwatch <NUM> that is less resistant to being worn on a steady basis as the AR controller, it is possible to wear the AR controller on a steady basis without difficulty in a state where fashion is maintained. The user can wear the AR controller at all times, and can obtain an interactive experience using gesture operations and display of the AR object.

Although the estimation of the position and posture of the wristwatch <NUM> is performed on the basis of the RGB image captured by the camera <NUM>, which is a camera for visible light detection, it is also possible to perform similar processing on the basis of an IR image captured by the IR camera.

In this case, the information processing terminal <NUM> is provided with an IR light irradiation device and an IR camera that detects reflected light of the IR light emitted by the irradiation device and generates an IR image as a captured image.

<FIG> is an enlarged view of the dial <NUM> of the wristwatch <NUM>.

As indicated by a broken line on the left side of <FIG>, a reflecting portion 21A including a retroreflective material for IR light is formed on the dial <NUM>. In the example of <FIG>, the reflecting portion 21A is formed in a perfect circular range including a range in which the long hand <NUM>-<NUM> and the short hand <NUM>-<NUM> rotate.

The retroreflective material is a material that reflects light at the same angle as the incident angle. The IR light with which the reflecting portion 21A is irradiated is reflected at the same angle as the incident angle.

The long hand <NUM>-<NUM> and short hand <NUM>-<NUM> provided on the dial <NUM> on which the reflecting portion 21A is formed include a non-reflective material for IR light.

In a case where the dial <NUM> having such a configuration is irradiated with IR light and captured by an IR camera, an IR image as indicated by a tip of arrow A31 is acquired as a captured image.

In the IR image shown in <FIG>, a portion other than the long hand <NUM>-<NUM> and the short hand <NUM>-<NUM> indicating "<NUM>:<NUM>" inside the reflecting portion 21A is shown with predetermined brightness. The outside of the reflecting portion 21A and the portion of the clock hand <NUM> indicating "<NUM>:<NUM>" appear in a dark state.

In the information processing terminal <NUM>, a template image for each posture of the dial <NUM> is generated on the basis of the shape of the reflecting portion 21A appearing in the IR image.

<FIG> is a diagram illustrating an example of a template image.

As illustrated in <FIG>, a template image for each posture is generated by deforming the dial <NUM> representing the current time so as to represent appearance when viewed from various angles.

For example, an image of the clock hand <NUM> representing the current time is synthesized with a circle representing the shape of the reflecting portion 21A, and the synthesized image is deformed to represent appearance when viewed from various angles, thereby generating a template image.

By performing such matching between the template image and each part of the IR image, the dial <NUM> appearing in the IR image is detected.

The configuration of the information processing terminal <NUM> illustrated in <FIG> is different from the configuration of <FIG> in that an IR light irradiation unit <NUM> and an IR camera <NUM> are provided. In the configuration illustrated in <FIG>, the same components as those illustrated in <FIG> are denoted by the same reference numerals as those used in <FIG>. Overlapping description will be omitted as appropriate.

The IR light irradiation unit <NUM> is an irradiation device that emits IR light. The IR light irradiation unit <NUM> emits IR light when a gesture operation of the user is detected.

The IR camera <NUM> detects reflected light of IR light emitted by the IR light irradiation unit <NUM> and generates an IR image as a captured image. The IR image generated by the IR camera <NUM> is supplied to the controller <NUM>. The IR camera <NUM> is provided, for example, in the vicinity of the IR light irradiation unit <NUM>.

In the controller <NUM> of <FIG>, the same configuration as each configuration of <FIG> is realized by executing a predetermined program.

Here, the position/posture estimation process will be described with reference to the flowchart of <FIG>.

The position/posture estimation process in <FIG> is started when a gesture operation of the user is detected.

In step S51, the time synchronization unit <NUM> synchronizes the time of the internal clock of the information processing terminal <NUM> with the time of the wristwatch <NUM>.

In step S52, the image acquisition unit <NUM> causes the IR camera <NUM> to capture an image and acquires an IR image. The IR image acquired by the image acquisition unit <NUM> is supplied to the position/posture estimation unit <NUM>.

In step S53, the template image generation unit <NUM> refers to the time information supplied from the time synchronization unit <NUM>, and generates a template image corresponding to each posture on the basis of the current time.

In step S54, the position/posture estimation unit <NUM> performs matching with each portion of the IR image using the template image generated by the template image generation unit <NUM>.

In step S55, the position/posture estimation unit <NUM> determines whether or not the dial <NUM> has been detected by matching using the template image.

In a case where the dial <NUM> is not detected, the processing returns to step S52, and the above processing is repeated. The template image is updated, and matching is repeated using the updated template image.

In a case where it is determined in step S55 that the dial <NUM> has been detected, in step S56, the position/posture estimation unit <NUM> estimates the position and posture of the dial <NUM> on the basis of the position and shape of the dial <NUM> appearing in the IR image, the direction of the clock hand <NUM>, and the like.

In step S57, the position/posture estimation unit <NUM> updates information on the position and posture of the wristwatch <NUM> represented by the position and posture of the dial <NUM>. Thereafter, the processing returns to step S52, and the above processing is repeated.

By using the IR image as described above, the estimation accuracy of the position and posture of the wristwatch <NUM> can be improved.

The configuration of the dial <NUM> illustrated on the left side of <FIG> is different from the configuration illustrated in <FIG> in that the short hand <NUM>-<NUM> includes a semi-transmissive material or a retroreflective material.

The reflecting portion 21A including a retroreflective material for IR light is formed on the dial <NUM>. In addition, the long hand <NUM>-<NUM> includes a non-reflective material for IR light.

In a case where the IR light is reflected on such a dial <NUM> and captured by an IR camera, an IR image as indicated by a tip of arrow A41 is acquired as a captured image.

In the IR image illustrated in <FIG>, a portion other than the long hand <NUM>-<NUM> inside the reflecting portion 21A is shown in a state of having predetermined brightness. The outside of the reflecting portion 21A and the portion of the long hand <NUM>-<NUM> appear in a dark state.

In the information processing terminal <NUM>, the position and posture of the dial <NUM> are estimated by such image processing on the IR image in which one long hand <NUM>-<NUM> appears together with the reflecting portion 21A. In this example, matching using the template image is not performed, and the position and posture of the dial <NUM> are estimated on the basis of the relationship between the shape of the reflecting portion 21A appearing in the IR image and the position of the long hand <NUM>-<NUM>.

<FIG> is a diagram illustrating an example of image processing.

In a case where the IR image illustrated at the left end of <FIG> is captured, binarization processing is performed on the IR image. In the IR image to be binarized, a flat elliptical shape representing the dial <NUM> is shown. A straight line representing long hand <NUM>-<NUM> is shown on the flat elliptical shape representing dial <NUM>.

By performing the binarization processing on the IR image, a binarized image in which an edge is emphasized as indicated by a tip of arrow A51 is generated.

As indicated at the tip of arrow A52, elliptical fitting is applied to the binarized image. The elliptical fitting is a process of aligning an ellipse having a predetermined size with an edge that is a boundary of the dial <NUM> appearing in the binarized image.

On the basis of the shape of the ellipse matched to the edge of the dial <NUM> by elliptical fitting, a plane in three-dimensional space where the dial <NUM> exists is detected as indicated by a dashed parallelogram at the tip of arrow A53.

On the other hand, as indicated by a tip of arrow A54, edge calculation is performed on an inner region of an ellipse matched with the edge of the dial <NUM> by elliptical fitting, and an edge E of a straight line representing the long hand <NUM>-<NUM> is detected.

As indicated by tips of arrows A55 and A56, a line segment representing the edge E is projected on a plane in a three-dimensional space where the dial <NUM> exists, and the position and posture of the dial <NUM> are estimated on the basis of the direction of the line segment representing the edge E. The direction of the line segment representing the edge E coincides with the direction of the minute of the current time. The position and posture of the dial <NUM> are such that long hand <NUM>-<NUM> indicating the minute of the current time looks like a line segment representing the edge E.

In this example, the position and posture of the dial <NUM> are estimated on the basis of the direction of one long hand <NUM>-<NUM> appearing in the IR image, but the position and posture of the dial <NUM> may be estimated on the basis of the direction of each of two of the long hand <NUM>-<NUM> and the short hand <NUM>-<NUM>. In this case, not only the long hand <NUM>-<NUM> but also the short hand <NUM>-<NUM> includes a non-reflective material.

The long hand <NUM>-<NUM> may include a semi-transmissive material or a retroreflective material, and the short hand <NUM>-<NUM> may include a non-reflective agent.

In the example of <FIG>, the IR light irradiation unit <NUM> as an IR light irradiation device is provided in the information processing terminal <NUM>, but the IR light irradiation device may be provided on the wristwatch <NUM> side.

As illustrated in A of <FIG>, the reflecting portion 21A includes a light diffusion sheet. As illustrated in a cross section in B of <FIG>, an IR-LED which is an irradiation device of IR light is provided on the back side of the reflecting portion 21A. The IR light emitted from the IR-LED is diffused by the reflecting portion 21A.

By detecting the IR light diffused by the reflecting portion 21A, the IR camera <NUM> of the information processing terminal <NUM> generates the IR image as described with reference to <FIG>, and estimates the position and posture of the dial <NUM> on the basis of the IR image.

In this manner, the IR light irradiation device can be provided on the wristwatch <NUM> side. In this case, the IR light irradiation unit <NUM> does not need to be provided in the information processing terminal <NUM>.

Note that, as described with reference to <FIG>, the short hand <NUM>-<NUM> of the long hand <NUM>-<NUM> and the short hand <NUM>-<NUM> may include a semi-transmissive material or a retroreflective material.

Although the shape of the dial <NUM> is a perfect circle, it may be a quadrangle.

In a case where the shape of the dial <NUM> is a quadrangle, the quadrangular template image is deformed to represent appearance when viewed from various angles, and matching is performed using the deformed template image.

For back projection from a quadrangle to a three-dimensional plane, for example, a general AR marker region detection method is used.

<FIG> is a diagram illustrating another example of the wristwatch.

As illustrated in <FIG>, a wristwatch whose time display type is a digital type can also be used as the AR controller.

In this case, instead of the clock hand <NUM>, a template image in which a digital numerical value representing the current time is synthesized is generated, and matching is performed.

In a case where the position and posture are estimated using the IR image, a region including a retroreflective material for IR light is formed on the liquid crystal back surface. In addition, IR light may be used as light of the backlight.

Since there is little resistance to wearing the wristwatch at all times, the user can widely use the wristwatch as the AR controller in daily life.

Example <NUM>: When the user stretches his/her hand to food in a convenience store, the information processing terminal <NUM> displays the price, ingredients, best-before date, reviews, and the like of the food on a display.

Example <NUM>: The information processing terminal <NUM> can give a warning to the user in a case where it is determined that there is an object that has not been put in the bag by photographing and storing an object that has been taken in and out of the bag by the user with the camera <NUM>.

(<NUM>) It can be used anytime, anywhere, and immediately when intended to be used, without the trouble of mounting the device.

Example <NUM>: In a case where the user finds that the user has no time to clean while purchasing for a visitor, the user can remotely instruct the cleaning robot where to clean while displaying a scenery in the house on the display of the information processing terminal <NUM>.

Example <NUM>: In a case where a message of a VR game participation request arrives from a friend in a train on commuting, the user can participate in the game by causing the information processing terminal <NUM> to detect a hand gesture and a touch operation on the dial <NUM> of the wristwatch <NUM>.

Example <NUM>: In a case where the user participates in a video conference using the information processing terminal <NUM>, the user can speak while indicating a document displayed on the display of the information processing terminal <NUM> with a hand gesture (pointer).

Example <NUM>: The user can photograph a scene ahead of the user's hand with the camera <NUM> of the information processing terminal <NUM>.

(<NUM>) Information related to time can be confirmed on the information processing terminal <NUM>.

Example <NUM>: A user may confirm information such as a calendar, a schedule, a mail, an SNS, and the like.

Example <NUM>: The user can display a past mail, an SNS message, or the like on the display of the information processing terminal <NUM> by turning the time adjustment dial provided on the wristwatch <NUM> in a predetermined direction.

Example <NUM>: The user can display the future schedule on the display of the information processing terminal <NUM> by turning the time adjustment dial of the wristwatch <NUM> in the opposite direction.

Example <NUM>: Instead of operating the dial, it may be possible to operate the clock face with a finger.

<FIG> is a diagram illustrating another example of the information processing terminal <NUM>.

Instead of the glasses-type wearable device, an information processing terminal such as a video transmissive-type head mounted display (HMD) 1A illustrated in A of <FIG> or a portable terminal such as a smartphone 1B illustrated in B of <FIG> may be used as a display device that displays content by the AR function.

In a case where the video transmissive-type HMD 1A is used as the display device, the video of the AR content reproduced by the HMD 1A is displayed in superposition with the image of the scenery in front of the HMD 1A captured by the camera provided in the HMD 1A. In front of the field of view of the user wearing the HMD 1A, a display that displays the AR content in superposition with the image captured by the camera is provided.

Furthermore, in a case where the smartphone 1B is used, the video of the AR content reproduced by the smartphone 1B is displayed to be superimposed on the image of the scenery in front of the smartphone 1B captured by the camera provided on the back surface of the smartphone 1B.

At least one of the configurations described with reference to <FIG> may be realized in an external device such as a PC connected to the information processing terminal <NUM>.

A series of processes described above may be performed by hardware, or may be performed by software. In a case where the series of processing is executed by software, a program constituting the software is installed from a program recording medium to a computer incorporated in dedicated hardware, a general-purpose personal computer, or the like.

<FIG> is a block diagram illustrating a hardware configuration example of the computer which performs a series of processes described above by a program.

A central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, and a random access memory (RAM) <NUM> are mutually connected by a bus <NUM>.

An input/output interface <NUM> is further connected to the bus <NUM>. An input unit <NUM> including a keyboard, a mouse, and the like, and an output unit <NUM> including a display, a speaker, and the like are connected to the input/output interface <NUM>. Furthermore, a storage unit <NUM> including a hard disk, a nonvolatile memory, or the like, a communication unit <NUM> including a network interface or the like, and a drive <NUM> that drives a removable medium <NUM> are connected to the input/output interface <NUM>.

In the computer configured as described above, for example, the CPU <NUM> loads a program stored in the storage unit <NUM> into the RAM <NUM> via the input/output interface <NUM> and the bus <NUM> and executes the program, whereby the above-described series of processing is performed.

The program executed by the CPU <NUM> is provided, for example, by being recorded in the removable medium <NUM> or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit <NUM>.

Note that the program executed by the computer may be a program in which processing is performed in time series in the order described in the present specification, or may be a program in which processing is performed in parallel or at necessary timing such as when a call is made.

Further, in this specification, the system means a set of a plurality of components (devices, modules (parts), etc.) regardless of whether or not all the components are provided in the same housing. Therefore, the plurality of devices which is stored in separate housings and connected through the network, and one device in which a plurality of modules is stored in one housing may be a system.

The effects described in the present specification are merely examples and are not limited, and other effects may be provided.

Embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made within the scope of the invention as defined in the appended claims.

For example, the present technology may be configured by a cloud computing system in which the plurality of devices shares one function through a network to perform the process together.

In addition, the steps described in the above flowchart may be being shared and performed by the plurality of devices instead of one device.

Claim 1:
An information processing apparatus (<NUM>) comprising:
an acquisition unit (<NUM>) that acquires a captured image obtained by capturing an image of a wearable device (<NUM>) including a module whose appearance changes according to time and whose appearance represents a current time; and
an estimation unit (<NUM>) that detects the module appearing in the captured image on the basis of a generated template image representing an appearance of the module according to the current time and estimates at least one of a position or a posture of the wearable device (<NUM>).