Patent Description:
Some video games involve capturing a user's body and markers with a camera and having a region of the image replaced with another image for display on a display (for example, see PTL <NUM>). Also known are user interface systems by which movements of the user's mouth or hands captured by a camera are interpreted as instructions to operate an application. Such a technology for capturing the real world to display a virtual world reacting to the captured movements or to perform some kind of information processing on the images has been used extensively on diverse scales ranging from mobile terminals to leisure facilities.

[PTL <NUM>]
<CIT>. A further known arrangement is disclosed in <CIT>.

The above-mentioned technology faces the important challenge of how to accurately acquire information associated with the real world from captured images. For example, when there are a large number of objects in the field of view, an object other than an object whose position and posture are to be recognized is sometimes detected as the object in question. Further, the luminance distribution of captured images largely changes depending on the brightness in the image-capturing environment or the lighting arrangement, to thereby affect the analysis accuracy in some cases. Thus, a technology that is highly robust to changes in environment or circumstance and that can acquire accurate information from captured images has been demanded.

The present invention has been made in view of such a challenge and has an object to provide a technology capable of acquiring information associated with objects at stable accuracy by using captured images.

A mode of the present invention relates to an information processing apparatus. The information processing apparatus includes a captured image acquiring section configured to acquire data on a moving image obtained by an imaging apparatus capturing a device including a light-emitting marker with an exposure time shorter than a period of one frame, a synchronous processing section configured to request the device to cause the light-emitting marker to emit light in a predetermined flashing pattern, with each flash being within a minimum unit being a time obtained by dividing the period of the one frame by a predetermined number and configured to identify the exposure time on a time axis of the device, based on whether or not an image of the light-emitting marker appears in a predetermined number of frames of the moving image obtained by capturing the device, a control information transmitting section configured to request the device to cause the light-emitting marker to emit light at a light emission time point corresponding to the identified exposure time, a position and posture acquiring section configured to acquire position and posture information regarding the device based on the image in a frame of the moving image of the light-emitting marker emitting light at the light emission time point for a fixed period of time equal to or shorter than the exposure time, and an output data generating section configured to generate and output data based on the position and posture information.

Another mode of the present invention relates to an information processing system. The information processing system includes a device including a light-emitting marker, an imaging apparatus configured to capture the device with an exposure time shorter than a period of one frame, and an information processing apparatus configured to acquire position and posture information regarding the device by using data on a moving image captured by the imaging apparatus. The information processing apparatus includes a synchronous processing section configured to request the device to cause the light-emitting marker to emit light in a predetermined flashing pattern, with each flash being within a minimum unit being a time obtained by dividing the period of the one frame by a predetermined number and configured to identify the exposure time on a time axis of the device, based on whether or not an image of the light-emitting marker appears in a predetermined number of frames of the moving image obtained by capturing the device, a control information transmitting section configured to request the device to cause the light-emitting marker to emit light at a light emission time point corresponding to the identified exposure time, a position and posture acquiring section configured to acquire the position and posture information regarding the device based on the image in a frame of the moving image of the light-emitting marker emitting light at the light emission time point for a fixed period of time equal to or shorter than the exposure time, and an output data generating section configured to generate and output data based on the position and posture information.

Still another mode of the present invention relates to a device for position and posture acquisition. The device for position and posture acquisition is captured by an imaging apparatus with an exposure time shorter than a period of one frame, and position and posture information regarding the device is acquired by an information processing apparatus using a captured moving image. The device includes a light-emitting marker, a control information acquiring section configured to acquire requests associated with light emission from the information processing apparatus, and a control section configured to cause, in response to a first one of the requests, the light-emitting marker to repetitively emit light in a predetermined flashing pattern, with each flash being within a minimum unit being a time obtained by dividing the period of the one frame by a predetermined number, and to cause, in response to a second one of the requests, the light-emitting marker to emit light at a light emission time point corresponding to the exposure time indicated on a time axis of the device for a fixed period of time equal to or shorter than the identified exposure time.

Yet another mode of the present invention relates to a device information acquisition method. The device information acquisition method includes the steps of acquiring data on a moving image obtained by an imaging apparatus capturing a device including a light-emitting marker with an exposure time shorter than a period of one frame, requesting the device to cause the light-emitting marker to emit light in a predetermined flashing pattern, with each flash being within a minimum unit being a time obtained by dividing the period of the one frame by a predetermined number, identifying the exposure time on a time axis of the device, based on whether or not an image of the light-emitting marker appears in a predetermined number of frames of the moving image obtained by capturing the device, requesting the device to cause the light-emitting marker to emit light at a light emission time point corresponding to the identified exposure time, acquiring position and posture information regarding the device based on the image in a frame of the moving image of the light-emitting marker emitting light at the light emission time point for a fixed period of time equal to or shorter than the exposure time, and generating and outputting data based on the position and posture information.

Note that, optional combinations of the above-mentioned components or the above expressions of the present invention converted between different forms, such as methods, apparatuses, systems, computer programs, and recording media having recorded thereon the computer programs, are also effective as modes of the present invention.

According to the present invention, it is possible to acquire information regarding objects at stable accuracy by using captured images.

<FIG> illustrates a configuration example of an information processing system to which the present embodiment is applicable. An information processing system <NUM> includes a light-emitting device <NUM>, an imaging apparatus <NUM> configured to capture a space including the light-emitting device <NUM>, an information processing apparatus <NUM> configured to analyze captured images to perform information processing, and a display apparatus <NUM> configured to output results of processing by the information processing apparatus <NUM>.

In this example, the light-emitting device <NUM> communicates with the information processing apparatus <NUM> by known wireless communication means such as Bluetooth (registered trademark). Further, the imaging apparatus <NUM> and the display apparatus <NUM> communicate with the information processing apparatus <NUM> with cables. However, the components may communicate with each other by communication connection means other than the ones described above. Further, the illustrated modes are exemplary and not intended to limit the shapes and configurations of the apparatuses. For example, the information processing apparatus <NUM> and the imaging apparatus <NUM>, or the information processing apparatus <NUM>, the imaging apparatus <NUM>, and the display apparatus <NUM> may be implemented as an integrated apparatus.

The display apparatus <NUM> may not be a flat-panel display as illustrated in <FIG> and may be, for example, a head-mounted display configured to display images in front of the eyes of the user wearing the head-mounted display. In this case, the imaging apparatus <NUM> may be provided to the head-mounted display to capture an image corresponding to the line of sight of the user. Alternatively, the light-emitting device <NUM> may be mounted on the external surface of the head-mounted display, and the imaging apparatus <NUM> may capture the light-emitting device <NUM>.

The light-emitting device <NUM> provides, when being held by the user, for example, its position, posture, movement, and the like as input information through captured images. In the example illustrated in <FIG>, the light-emitting device <NUM> includes a ring-like casing with a large number of point-like light-emitting markers (for example, light-emitting markers <NUM>) arranged on the surface thereof. The light-emitting markers <NUM> each include an element capable of being switched between the on state and the off state such as an LED (Light Emitting Diode). The information processing apparatus <NUM> can control the switching of the light-emitting markers <NUM>.

Note that, the shape of the light-emitting device <NUM> and the shape, size, and number of the light-emitting markers <NUM> are not limited to the ones illustrated in <FIG>. For example, a user-holdable object having an optional shape may be connected to one or a plurality of light-emitting bodies having an optional shape and including light-emitting elements. Alternatively, an apparatus including a general game controller provided with light-emitting markers may serve as the light-emitting device <NUM>. Still alternatively, one or a plurality of the light-emitting devices <NUM> may directly be mounted on the user's body. Further, the light-emitting device <NUM> preferably includes a gyroscope and an accelerometer configured to acquire the angular velocity and acceleration of the light-emitting device <NUM> itself. In the following, these sensors are collectively referred to as an "IMU sensor.

The imaging apparatus <NUM> includes a camera configured to capture a space including the light-emitting device <NUM> at a predetermined frame rate, and a mechanism configured to perform general processing, such as demosaicing, on an output signal from the camera to generate output data on the captured image and to transmit the output data to the information processing apparatus <NUM>. The camera includes a visible light sensor used in a general digital video camera, such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. However, a sensor capable of detecting light in an optional wavelength band may be used as long as the sensor allows the camera to acquire the images of the light-emitting markers <NUM> of the light-emitting device <NUM>.

Further, the imaging apparatus <NUM> of the present embodiment uses an exposure time shorter than that in general image-capturing conditions to accurately detect the images of the light-emitting markers <NUM> emitting light. Thus, in a case where captured images obtained with a general exposure time are required to generate display images, for example, the exposure time may be changed in units of frames or a dedicated imaging apparatus may be separately introduced. That is, the number of the imaging apparatus <NUM> included in the information processing system <NUM> is not limited.

Further, the imaging apparatus <NUM> may include only one camera or incorporate what is called a stereo camera in which two cameras are disposed side by side with a known distance therebetween. In the case where a stereo camera is introduced, the light-emitting device <NUM> is captured from the left and right different point of views, and a distance to the light-emitting device <NUM> is obtained by the principles of triangulation on the basis of the position shifts on the captured images.

The information processing apparatus <NUM> performs required information processing by using the data on the captured image transmitted from the imaging apparatus <NUM>, to thereby generate output data such as images and sound. In the present embodiment, the information processing apparatus <NUM> at least identifies the position and posture of the light-emitting device <NUM> on the basis of the images of the light-emitting markers <NUM> appearing in the captured image. When the shape of the light-emitting device <NUM> and the positions of the light-emitting markers <NUM> on the surface of the light-emitting device <NUM> are known, the position and posture of the light-emitting device <NUM> can be estimated on the basis of the distribution of the images of the light-emitting markers <NUM> on the captured image. Also in a case where a single light-emitting body is used as a light-emitting marker, the position and the like of the light-emitting device <NUM> can be acquired from the size and shape of the image of the light-emitting marker.

By repeating this processing in units of frames of the captured image, the movements of the light-emitting device <NUM>, namely, the movements of the user, can be identified, so that information processing such as a video game can be performed using the movements as command inputs. Alternatively, an image in which the image of the light-emitting device <NUM> in the captured image is replaced by a virtual object or an image in which the light-emitting device <NUM> is interacting with surrounding real objects can be generated. The information processing apparatus <NUM> performs any processing by utilizing the information regarding the position and posture of the light-emitting device <NUM> and may perform processing appropriately determined depending on functions that the user wants or the contents of applications.

The display apparatus <NUM> includes a general display configured to display images, such as a liquid-crystal display or an organic EL (Electroluminescence) display, a speaker configured to output sound, and the like. The display apparatus <NUM> outputs images and sound generated by the information processing apparatus <NUM> as information processing results. As described above, the display apparatus <NUM> may be a head-mounted display or the like and may have any form as long as the display apparatus <NUM> can output images and sound. Further, the results of processing performed by the information processing apparatus <NUM> may be recorded on a recording medium or a storage apparatus or may be transmitted to another apparatus via a network, which is not illustrated. That is, the information processing system <NUM> does not necessarily include the display apparatus <NUM> and may include those output mechanisms instead of the display apparatus <NUM>.

<FIG> is a diagram illustrating a method by which the information processing apparatus <NUM> acquires the position and posture of the light-emitting device <NUM> in the present embodiment. The information processing apparatus <NUM> acquires a captured image <NUM> from the imaging apparatus <NUM> at a predetermined rate. The captured image <NUM> depicts the images of the light-emitting markers in a known color at relatively high luminance. Thus, the information processing apparatus <NUM> extracts such images of the light-emitting markers, thereby identifying the three-dimensional position and posture of the light-emitting device <NUM> by using the following general conversion formula. <NUM>] <MAT>.

Here, (u, v) indicates the position of the image of the light-emitting marker in the captured image, (fx, fy) indicates the focal length of the imaging apparatus <NUM>, (cx, cy) indicates the principal point of the image, the matrix with elements of r<NUM> to r<NUM> and t<NUM> to t<NUM> is a rotation and translation matrix, and (X, Y, Z) indicates the three-dimensional position of the light-emitting marker when the light-emitting device <NUM> is at a reference position with a reference posture. The equation is solved for the plurality of light-emitting markers with (u, v), (fx, fy), (cx, cy), and (X, Y, Z) having known values, so that a rotation and translation matrix common to the light-emitting markers is obtained. The position and posture of the light-emitting device <NUM> are obtained on the basis of the angle and translation amount represented by this matrix. This processing is referred to as "image analysis processing <NUM>.

Meanwhile, the information processing apparatus <NUM> also acquires, from the IMU sensor incorporated in the light-emitting device <NUM>, the angular velocity and acceleration of the light-emitting device <NUM> at a predetermined rate, to thereby acquire the position and posture of the light-emitting device <NUM>. This is referred to as "sensor value analysis processing <NUM>. " In <FIG>, an actual change in position or posture of the light-emitting device <NUM> over time and the processing results of the image analysis processing <NUM> and the sensor value analysis processing <NUM> are schematically illustrated in the single graph. The actual change of the light-emitting device <NUM> is indicated by the solid line, the results obtained by the image analysis processing <NUM> are indicated by the white circles, and the results obtained by the sensor value analysis processing <NUM> are indicated by the dotted lines.

Information obtained from the IMU sensor includes the rotational speed and translational acceleration of the light-emitting device <NUM>. Thus, in the sensor value analysis processing <NUM>, with initial points being a position and a posture at a previous time point, the amounts of change in position and posture obtained by the integration of the speed and the acceleration are added, so that a position and a posture at a next time point are calculated. As the initial points here, the results of the image analysis processing <NUM> can be utilized. However, while the imaging apparatus <NUM> captures images at approximately <NUM> (<NUM> frame/second), the IMU sensor can perform measurement at a high frequency of approximately <NUM>,<NUM>. Thus, as compared to the results of the image analysis processing <NUM> that are discrete as indicated by the white circles, the sensor value analysis processing <NUM> can follow the position and the posture at remarkably short intervals as indicated by the dotted lines.

However, since the sensor value analysis processing <NUM> including the integration of the speed and the acceleration accumulates errors, the results tend to deviate from the actual position and posture indicated by the solid line over time. The results of the image analysis processing <NUM> are thus integrated with a Kalman filter to obtain the position and posture of the light-emitting device <NUM> continuously and accurately. The method that achieves high accuracy by integrating the results of analysis by the imaging apparatus <NUM> and the IMU sensor including the plurality of sensors in this way is known as "sensor fusion.

In sensor fusion, time points at which values are acquired by each sensor are required to be indicated on a common time axis. In the case of the present embodiment, on the time axis that is the horizontal axis of <FIG>, the time points at which the images used in the image analysis processing <NUM> have been captured, and the time points at which the angular velocities and acceleration used in the sensor value analysis processing <NUM> have been measured are accurately set, so that the position and posture of the light-emitting device <NUM> is obtained with a higher accuracy by the integration of the results.

Meanwhile, to enhance the accuracy of the image analysis processing <NUM>, it is effective to shorten the exposure time of the imaging apparatus <NUM> and the light emission time of the light-emitting markers as much as possible. That is, in a general captured image, the images of the light-emitting markers are expressed at luminance similar to that of the images of light reflected by other objects or lighting, and it is thus difficult to tell which is which in some cases. Further, the difficulty in distinction changes depending on surrounding brightness or the like. The shorter the exposure time, the larger a difference in luminance between the images of objects emitting strong light and other objects, with the result that the images of the light-emitting markers can be detected with high robustness.

Further, the light-emitting device <NUM> is supposed to be moved, and hence when the light emission time of the light-emitting markers is long, the images are blurred, resulting in a difficulty in detection and large errors of the position coordinates of the images that are used for analysis. When less blurred images can be captured with the shortest possible light emission time, the positions on the captured image can accurately be acquired, and the accuracy of the analysis result can thus be enhanced. Further, the image-time point correspondence can be clarified, which is effective in the sensor fusion described above. Further, since the light-emitting device <NUM> is caused to emit light for a short period of time with small power consumption, the light-emitting device <NUM> can be used as the battery power supply for a long period of time.

<FIG> exemplifies a relation between the exposure time of the imaging apparatus <NUM> and the light emission time of the light-emitting markers <NUM> in the present embodiment. In <FIG>, the horizontal axis indicates elapsed time and the time divisions with the dashed-dotted lines indicate image-capturing periods per frame. For example, in the case of a frame rate of <NUM> frame/second, the image-capturing period is approximately <NUM> msec. Meanwhile, in the present embodiment, in the frames, exposure times 60a, 60b, and 60c are set to <NUM>µsec, and light emission times 62a, 62b, and 62c of the light-emitting markers <NUM> are set to <NUM>µsec, for example.

However, the exposure time and the light emission time are only required to be optimized depending on light emission luminance or surrounding brightness and are not limited to specific times. In any case, the exposure times 60a, 60b, and 60c and the light emission times 62a, 62b, and 62c of the light-emitting markers <NUM> are desirably the shortest possible times that allow the light-emitting markers <NUM> to be expressed at luminance high enough to achieve image detection from a captured image. Qualitatively, necessary conditions include determining a light emission time that allows light emitted at a certain moment to be expressed as an image, and determining an exposure time equal to or longer than the light emission time.

Further, as illustrated in <FIG>, the light emission times 62a, 62b, and 62c are set to the times entirely included in the exposure times 60a, 60b, and 60c, so that the images of the light-emitting markers <NUM> always appear at similar luminance in a captured image. For example, the exposure time is set to a time approximately several times longer than the light emission time, so that the subtle shifts of the light emission time point can be prevented from affecting images.

Further, in a case where the sensor fusion described above is implemented, correlations between the time points of measurement by the IMU sensor and exposure time points are desirably acquired with high accuracy. Since the light-emitting device <NUM> and the imaging apparatus <NUM> operate with the respective independent clocks, in the present embodiment, the clocks are set on a common time axis through control. Specifically, first, the reference time points of the exposure times 60a, 60b, and 60c and the reference time points of the light emission times 62a, 62b, and 62c are adjusted to each other.

In the example illustrated in <FIG>, as indicated by the arrows, the intermediate time points (time points at the middle of the entire times) of the exposure times 60a, 60b, and 60c are set as references, and the intermediate time points of the light emission time 62a, 62b, and 62c are adjusted thereto. In the following, the intermediate time point of the light emission time is sometimes simply referred to as a "light emission time point. " With this, a light emission time point recognized by the light-emitting device <NUM> is an image-capturing time point. Further, even when the light emission time 62a, 62b, or 62c slightly shifts, the light emission time does not easily get out of the exposure time 60a, 60b, or 60c.

<FIG> exemplifies a temporal relation between light emission and measurement by the IMU sensor in the light-emitting device <NUM> of the present embodiment. In the upper part of <FIG>, the exposure time 60a and the light emission time 62a in the image-capturing period for one frame on the time axis illustrated in <FIG> are illustrated. In this image-capturing period, the IMU sensor measures the angular velocity and acceleration of the light-emitting device <NUM> at high frequency. In the middle part of <FIG>, on the same time axis as in the upper part, the light emission time point of the light-emitting markers <NUM> (white circle) and the time points of measurement by the IMU sensor (black circles) are illustrated in parallel.

The light-emitting device <NUM> acquires the light emission time point and the measurement time points on the internal time axis. Thus, the light-emitting device <NUM> transmits, as illustrated in the lower part of <FIG>, the time point at which the light-emitting markers <NUM> have been caused to emit light together with the results of measurement by the IMU sensor and the measurement time points to the information processing apparatus <NUM>. The light emission time point is adjusted to the intermediate time point of the exposure time as illustrated in <FIG>, so that the information processing apparatus <NUM> can express, on the basis of the transmitted light emission time point, the image-capturing time point and the time points of measurement by the IMU sensor on the same time axis.

With such a mechanism, the sensor value analysis processing <NUM> and the image analysis processing <NUM> can be integrated as illustrated in <FIG>, and the position and posture of the light-emitting device <NUM> can thus continuously and accurately be acquired. The information processing apparatus <NUM> achieves the state as illustrated in <FIG>, in which the image capturing and the light emission are synchronized with each other, before operating a video game, for example, and monitors shifts as background processing during operation to maintain the state. The information processing apparatus <NUM> corrects the shifts as needed.

<FIG> illustrates the internal circuit configuration of the information processing apparatus <NUM>. The information processing apparatus <NUM> includes a CPU (Central Processing Unit) <NUM>, a GPU (Graphics Processing Unit) <NUM>, and a main memory <NUM>. These sections are connected to each other via a bus <NUM>. An input/output interface <NUM> is also connected to the bus <NUM>. A communication section <NUM>, a storage section <NUM>, an output section <NUM>, an input section <NUM>, and a recording medium driving section <NUM> are connected to the input/output interface <NUM>. The communication section <NUM> includes a peripheral interface such as a USB (Universal Serial Bus) or IEEE (Institute of Electrical and Electronics Engineers) <NUM> or a wired or wireless LAN (Local Area Network) network interface. The storage section <NUM> is a hard disk drive, a nonvolatile memory, or the like. The output section <NUM> outputs data to the display apparatus <NUM> or the light-emitting device <NUM>. The input section <NUM> receives data from the imaging apparatus <NUM> or the light-emitting device <NUM>. The recording medium driving section <NUM> drives removable recording media such as magnetic disks, optical discs, or semiconductor memories.

The CPU <NUM> controls the whole information processing apparatus <NUM> by executing the operating system stored in the storage section <NUM>. The CPU <NUM> also executes various programs that are either read from a removable recording medium and loaded into the main memory <NUM> or downloaded via the communication section <NUM>. The GPU <NUM> has a geometry engine function and a rendering processor function. The GPU <NUM> performs drawing processing in accordance with drawing instructions from the CPU <NUM> and stores the display images in a frame buffer, which is not illustrated. Then, the GPU <NUM> converts the display images stored in the frame buffer into video signals and outputs the video signals to the output section <NUM>. The main memory <NUM> includes a RAM (Random Access Memory) and stores programs and data required for processing.

<FIG> illustrates the internal circuit configuration of the light-emitting device <NUM>. The light-emitting device <NUM> includes a CPU <NUM>, a memory <NUM>, a communication section <NUM>, an IMU sensor <NUM>, an LED <NUM>, and a clock generating circuit <NUM>. The CPU <NUM> controls the circuits of the light-emitting device <NUM> and data transmission between the circuits. In the present embodiment, in particular, the CPU <NUM> receives requests from the information processing apparatus <NUM> to control the on and off of the LED <NUM> or measurement by the IMU sensor <NUM>, or to correct the internal clock. The memory <NUM> stores data required for processing in the CPU <NUM>. In the present embodiment, in particular, the memory <NUM> stores the light emission timing patterns of the LED <NUM> and various correction values.

The communication section <NUM> is an interface for data transmission with the information processing apparatus <NUM> and can be implemented with a known wireless communication technology such as Bluetooth (registered trademark). The IMU sensor <NUM> includes a gyroscope and an accelerometer and acquires the angular velocity and acceleration of the light-emitting device <NUM>. Values output from the sensor are transmitted to the information processing apparatus <NUM> via the communication section <NUM>. The LED <NUM> is an element configured to emit light in a predetermined color or a set of such elements, and forms the light-emitting marker <NUM> illustrated in <FIG>. The clock generating circuit <NUM> is a circuit configured to generate the clock of the light-emitting device <NUM> and has a function of correcting the clock frequency, under the control of the CPU <NUM>.

<FIG> illustrates the functional block configurations of the information processing apparatus <NUM> and the light-emitting device <NUM>. The functional blocks illustrated in <FIG> can be implemented, in terms of hardware, by the configurations illustrated in <FIG> and <FIG>, such as the CPU, the GPU, or the memory. The functional blocks can be implemented, in terms of software, by a program loaded from a recording medium or the like into the memory to achieve various functions such as a data input function, a data retaining function, an image processing function, and an input/output function. Thus, it is understood by those skilled in the art that the functional blocks can be implemented in various forms from only hardware, only software, or a combination thereof and are not limited to any of them.

The information processing apparatus <NUM> includes a device data acquiring section <NUM>, a captured image acquiring section <NUM>, a position and posture acquiring section <NUM>, an information processing section <NUM>, an output data generating section <NUM>, a synchronous processing section <NUM>, and a control information transmitting section <NUM>. The device data acquiring section <NUM> acquires, from the light-emitting device <NUM>, data on values measured by the sensor and various time points. The captured image acquiring section <NUM> acquires data on captured images from the imaging apparatus <NUM>. The position and posture acquiring section <NUM> acquires the position and posture of the light-emitting device <NUM> by using values measured by the IMU sensor and captured images. The information processing section <NUM> performs information processing on the basis of the position and posture of the light-emitting device <NUM>. The output data generating section <NUM> generates and outputs data indicating the results of information processing. The synchronous processing section <NUM> synchronizes the exposure of the imaging apparatus <NUM> and the light emission of the light-emitting device <NUM>. The control information transmitting section <NUM> transmits, to the light-emitting device <NUM>, control information associated with light emission and time point correction.

The device data acquiring section <NUM> is implemented by the input section <NUM>, the communication section <NUM>, the CPU <NUM>, and the main memory <NUM> of <FIG>, for example. The device data acquiring section <NUM> acquires, from the light-emitting device <NUM>, values measured by the IMU sensor <NUM>, that is, angular velocity and acceleration at the predetermined rate. At this time, the device data acquiring section <NUM> also acquires time points at which the measured values have been obtained and time points at which the light-emitting markers have been caused to emit light in parallel to the measurement. The device data acquiring section <NUM> supplies the acquired information to the position and posture acquiring section <NUM>. The captured image acquiring section <NUM> is implemented by the input section <NUM>, the CPU <NUM>, and the main memory <NUM> of <FIG>, for example. The captured image acquiring section <NUM> acquires sequentially data on a captured image obtained by image capturing by the imaging apparatus <NUM> at a predetermined frame rate and supplies the data to the position and posture acquiring section <NUM>.

The position and posture acquiring section <NUM> is implemented by the CPU <NUM>, the GPU <NUM>, and the main memory <NUM> of <FIG>, for example. The position and posture acquiring section <NUM> detects the images of the light-emitting markers from a captured image and acquires the position and posture of the light-emitting device <NUM> with Expression <NUM>. The position and posture acquiring section <NUM> acquires the position and posture of the light-emitting device <NUM> also by integration based on angular velocity and acceleration measured by the IMU sensor. Then, both results are integrated as described with <FIG>, so that position and posture information is generated at a predetermined rate. Here, as described above, the position and posture acquiring section <NUM> expresses the time points of measurement by the IMU sensor <NUM> and the light emission time point of the light-emitting markers, namely, the image-capturing time point, on the same time axis, to thereby enhance the accuracy of position and posture acquisition.

The information processing section <NUM> is implemented by the CPU <NUM> and the main memory <NUM> of <FIG>, for example. The information processing section <NUM> performs predetermined information processing on the basis of the position and posture of the light-emitting device <NUM>. As described above, the details of the processing are not particularly limited. The information processing section <NUM> may cause a video game to progress with input information being the movements of the user or may replace the light-emitting device <NUM> by a virtual object and perform physical calculation to achieve augmented reality. It will be understood by those skilled in the art that various types of information processing other than the above can be implemented.

The output data generating section <NUM> is implemented by the GPU <NUM>, the main memory <NUM>, the output section <NUM>, and the communication section <NUM> of <FIG>, for example. The output data generating section <NUM> generates data on images and sound to be output as the results of information processing by the information processing section <NUM>. The contents of data to be generated depend on the details of information processing. In a case where images and sound are generated, the output data generating section <NUM> outputs generated data to the display apparatus <NUM> at a predetermined rate. However, as described above, the output destination is not limited to the display apparatus and may be a recording medium, a storage apparatus, a network, or the like.

The synchronous processing section <NUM> is implemented by the CPU <NUM> and the main memory <NUM> of <FIG>, for example. The synchronous processing section <NUM> synchronizes image capturing by the imaging apparatus <NUM> and the light emission of the light-emitting markers <NUM>. That is, the synchronous processing section <NUM> performs adjustment processing so that the intermediate time point of the exposure time and the light emission time point of the light-emitting markers <NUM> match each other as illustrated in <FIG>. Specifically, the synchronous processing section <NUM> causes the light-emitting markers <NUM> to emit light in a predetermined pattern in the time direction and checks how the light appears in a captured image, to thereby determine a time point on the time axis inside the light-emitting device <NUM> at which the light-emitting markers <NUM> are to be caused to emit light.

This processing includes a first stage inf which whether light is emitted at least within the exposure time is evaluated in units of time similar to the exposure time, and a second stage in which the intermediate time point of the exposure time and the light emission time point are finely adjusted to each other. The first stage is performed as initial calibration before the processing of operating a video game, for example. The second stage, in which light is emitted for the same light emission time as during operation, can be performed as the background processing during operation. In the background processing, shifts due to a clock difference between the imaging apparatus <NUM> and the light-emitting device <NUM> are monitored to correct the clock. The details of the processing are described later.

The control information transmitting section <NUM> is implemented by the CPU <NUM>, the output section <NUM>, and the communication section <NUM> of <FIG>, for example. The control information transmitting section <NUM> transmits, to the light-emitting device <NUM>, information or requests required for the synchronous processing section <NUM> to achieve synchronization. Specifically, the control information transmitting section <NUM> transmits light emission pattern specification, light emission time point correction requests, clock correction requests, or the like. The control information transmitting section <NUM> may also transmit control information associated with the start/stop of measurement by the IMU sensor or the start/stop of image capturing by the imaging apparatus <NUM> to the light-emitting device <NUM> or the imaging apparatus <NUM>. The exposure time of the imaging apparatus <NUM> may be set to the imaging apparatus <NUM> itself in advance or may be specified by the control information transmitting section <NUM>.

The light-emitting device <NUM> includes a control information acquiring section <NUM>, a control section <NUM>, a light-emitting section <NUM>, a measurement section <NUM>, and a data transmitting section <NUM>. The control information acquiring section <NUM> acquires control information from the information processing apparatus <NUM>. The control section <NUM> controls light emission, measurement, and the clock on the basis of control information. The light-emitting section <NUM> causes the LED <NUM> to emit light. The measurement section <NUM> measures angular velocity and acceleration. The data transmitting section <NUM> transmits necessary data such as measured values to the information processing apparatus <NUM>. The control information acquiring section <NUM> is implemented by the communication section <NUM> and the CP1U80 of <FIG>, for example. The control information acquiring section <NUM> acquires control information transmitted from the information processing apparatus <NUM>.

The control section <NUM> is implemented by the CPU <NUM>, the memory <NUM>, and the clock generating circuit <NUM> of <FIG>, for example. The control section <NUM> causes, on the basis of the control information, the light-emitting section <NUM> and the measurement section <NUM> to operate, and generates the clock. The control information from the information processing apparatus <NUM> includes identification information specifying light emission patterns indicating temporal changes in flashing state. Thus, the control section <NUM> holds therein information having the identification information and the light emission patterns associated with each other.

Besides, the control information may include various types of information regarding light emission time point correction values, the start/stop of light emission, light emission luminance, light emission colors, or the start/stop of measurement by the IMU sensor. The control section <NUM> appropriately causes, on the basis of those pieces of information, the light-emitting section <NUM> and the measurement section <NUM> to operate. Further, when the control information includes a clock correction request, the control section <NUM> corrects the frequency of the clock generated inside the control section <NUM>.

The light-emitting section <NUM> is implemented by the LED <NUM> of <FIG>. The light-emitting section <NUM> causes, under the control of the control section <NUM>, the element to emit light in a specified pattern or at a specified timing. The measurement section <NUM> is implemented by the IMU sensor <NUM> of <FIG>. The measurement section <NUM> measures, under the control of the control section <NUM>, the angular velocity and acceleration of the light-emitting device <NUM> at a predetermined frequency. The data transmitting section <NUM> is implemented by the CPU <NUM>, the memory <NUM>, and the communication section <NUM> of <FIG>, for example. The data transmitting section <NUM> sequentially transmits, to the information processing apparatus <NUM>, the values of angular velocity and acceleration measured by the measurement section <NUM> together with the measurement time points. The data transmitting section <NUM> also sequentially transmits, to the information processing apparatus <NUM>, time points at which the light-emitting section <NUM> has emitted light.

Now, the details of synchronous processing that the synchronous processing section <NUM> of the information processing apparatus <NUM> performs in cooperation with the light-emitting device <NUM> are described. <FIG> illustrates the stages of the processing that is performed by the synchronous processing section <NUM>. The synchronous processing includes, as illustrated in <FIG>, processing in three stages of a pre-scan phase (S10), a broad phase (S12), and a background phase (S14). The pre-scan phase and the broad phase are performed before the operation of video game processing, for example. Of those, the pre-scan phase is a stage for detecting the light-emitting device <NUM>. Thus, the light-emitting markers <NUM> are caused to emit light in the whole period.

The broad phase is a stage for roughly adjusting the light emission time point of the light-emitting device <NUM> detected in the pre-scan phase to the exposure time. Thus, the time is divided into time grids each of which is approximately equal to the exposure time, and the light-emitting markers <NUM> are caused to emit light in flash patterns set to the respective time grids. The light emission pattern is set over a time corresponding to a plurality of frames. With this, the range of a light emission time point that matches the exposure time is identified on the basis of how the images of the light-emitting markers <NUM> appear in the time direction.

The background phase is a stage for checking light emission time point shifts regularly and rigorously and correcting the light emission time point or the clock frequency of the light-emitting device <NUM> as needed before and during operation. Thus, the light emission time point is intentionally shifted by the light emission time during operation that is shorter than the time grid in the broad phase, and a transition timing between the state in which the light-emitting markers <NUM> appear in the captured image and the state in which the light-emitting markers <NUM> do not appear in the captured image is acquired, so that a shift of the original light emission time point from the intermediate time point of the exposure time is obtained. Then, the light emission time point is corrected to eliminate the shift. Further, the clock frequency is corrected in a direction that eliminates the shift, on the basis of the amount of shift generated per unit time.

When the images of the light-emitting markers <NUM> are not obtained for a predetermined period of time in the broad phase, the processing returns to the pre-scan phase and starts from the light-emitting device detection. When effective shift amounts are no longer obtained in the background phase, the processing returns to the broad phase and starts again from the light emission time point rough adjustment. In this way, the plurality of phases are provided and light is emitted in the patterns suitable therefor so that synchronization can efficiently be performed and constant monitoring during operation can be achieved.

<FIG> schematically illustrates a light emission pattern in the pre-scan phase. The rectangle illustrated in <FIG> indicates an image-capturing period having a horizontal length corresponding to one frame. Basically, this time is divided into time grids each having a predetermined time, and the light-emitting device is caused to flash in units of time grids. For example, in the case of a frame rate of <NUM> frame/second, the image-capturing period is <NUM> msec. When the image-capturing period is divided into <NUM> time grids, one time grid is <NUM> usec. In a case where the exposure time is set to <NUM>µsec as described above and time grids each of which is approximately <NUM>µsec are used, the exposure time and the light emission time have similar granularities, with the result that whether or not the light emission time overlaps the exposure time is roughly determined.

However, in the pre-scan phase, which is the stage for detecting the light-emitting device <NUM> in the field of view of the imaging apparatus <NUM>, the light-emitting markers are on in all the time grids by PWM (Pulse Width Modulation) control. In <FIG>, the time grids are painted out to indicate that the light-emitting markers are on. In the pre-scan phase, the synchronous processing section <NUM> acquires a captured image from the captured image acquiring section <NUM> and detects the images of the light-emitting markers <NUM>, to thereby confirm that the light-emitting device is present in the field of view. Since the light-emitting markers are on in the whole period, the images of the light-emitting markers <NUM> are detected even with exposure for a short period of time of <NUM>µsec as long as the light-emitting device <NUM> is present in the field of view.

When the presence of the light-emitting device <NUM> is confirmed, the processing transitions to the broad phase. <FIG> schematically illustrates an exemplary light emission pattern in the broad phase. A manner of representation in <FIG> is similar to that in <FIG>. However, with regard to this phase in which a light emission pattern is set over a plurality of frames, changes between the frames are illustrated in the vertical direction. In the case of <FIG>, one light emission pattern includes <NUM> frames of the frame <NUM> to the frame <NUM>.

That is, the light-emitting markers <NUM> repetitively emit light in the light emission pattern in question in units of <NUM> frames. In a manner similar to that during operation, the imaging apparatus <NUM> repeats exposure for a short period of time (for example, approximately <NUM>µsec) at a predetermined timing in the image-capturing period in each frame. For example, it is assumed that the exposure time is set to a time indicated by a solid rectangle 70a corresponding to a time grid "<NUM>. " In this case, in the first five frames (frames <NUM> to <NUM>), the light-emitting markers <NUM> are off and no image thus appears. In the next frame <NUM>, the light-emitting markers <NUM> are on in the whole period, and hence the images of the light-emitting markers <NUM> appear in the exposure time corresponding to the rectangle 70a.

In the next three frames (frames <NUM> to <NUM>), the light-emitting markers <NUM> are off at this timing, so that no image appears. In the next two frames (frames <NUM> and <NUM>), the light-emitting markers <NUM> are on so that the images of the light-emitting markers <NUM> appear. Of the successive <NUM> frames, the frames in which the images of the light-emitting markers <NUM> appear are indicated by "<NUM>," and the frames in which the images of the light-emitting markers <NUM> do not appear are indicated by "<NUM>. " In the exposure time corresponding to the rectangle 70a, a numerical sequence of {<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>} is obtained in order from the frame <NUM>. In the following, such a numerical sequence is referred to as an "image appearance pattern.

Note that, when the image exposure time extends over the on time and the off time, the luminance of images varies depending on the on time in the exposure time. A threshold is accordingly set to the luminance. Images having luminance equal to or larger than the threshold are determined as "positive," and images having luminance smaller than the threshold are determined as "negative. " Further, in a case where there are a plurality of light-emitting markers, a case where at least one of the light-emitting markers appears may be determined as "positive," or a case where the number of appearing light-emitting markers is equal to or larger than a threshold may be determined as "positive.

With those reference combinations, whether the images of the light-emitting markers appear or not are indicated by the binary digits of "<NUM>" and "<NUM>. " As a similar method, blob detection that detects the presence/absence of objects by image analysis is known. However, in the present embodiment, since the light-emitting markers are intentionally turned off, "negative" results are not taken as errors and the image appearance pattern acquisition continues.

The light emission pattern is set so that the different image detection patterns are obtained in all the time grids as illustrated in <FIG>, and hence relations between the time grids of the light-emitting device <NUM> and the exposure time are obtained. Note that, the reason why the light-emitting markers are off in the whole period of the first five frames but are on in the whole period of the next frame is to distinguish a light emission pattern for one unit from adjacent light emission patterns. This period is referred to as a "sync pattern <NUM>. " Thus, in reality, a captured image is continuously acquired, and when an image appearance pattern having the sync pattern <NUM> of {<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>} is obtained, relations between the time grids of the light-emitting device <NUM> and the exposure time are identified on the basis of image appearance patterns in the next five frames.

The light emission patterns following the sync pattern <NUM> are set so that a light emission pattern in a certain time grid does not match those in other time grids, and that the adjacent time grids are different from each other only in one frame in terms of the flashing state as described above. For example, when the time grid corresponding to the rectangle 70a (time grid "<NUM>") is a reference, only in the frame <NUM>, in which the light-emitting markers are off in the previous time grid "<NUM>," the light-emitting markers are turned on. Further, in the following time grid "<NUM>," the light-emitting markers are changed from the on state to the off state only in the frame <NUM>.

With the light emission pattern in which the light-emitting markers are gradually changed between the on state and the off state, even when the exposure time extends over two time grids, an error is at most one time grid. For example, in a case where the exposure time is at the timing of a dashed rectangle 70b delayed from the rectangle 70a by approximately <NUM>µsec, an image appearance pattern following the sync pattern is the same up to {<NUM>, <NUM>, <NUM>, <NUM>} but has an intermediate value ranging from <NUM> to <NUM> as the last value depending on the degree of delay. When the value is determined with a threshold by blob detection, a numerical value of <NUM> or <NUM> is set. In any case, however, such a numerical sequence is obtained only in successive two time grids, and the exposure time can therefore be identified with almost no error. Such a feature of patterns is also seen in Gray codes that are used for expressing numerical values in digital circuits.

Note that, the light emission pattern illustrated in <FIG> is an example, and any light emission pattern may be employed as long as image appearance patterns are different between the time grids and the adjacent time grids are different from each other only in one frame in terms of the flashing state. Further, the size of the time grids may appropriately be adjusted on the basis of the exposure time. Qualitatively, as the number of time grids increases, the number of frames required for a light emission pattern for one unit increases.

<FIG> is a flowchart illustrating a processing procedure by which the synchronous processing section <NUM> identifies a time grid of the light-emitting device <NUM> that corresponds to the exposure time in the broad phase. This flowchart starts under a state where the control information transmitting section <NUM> has requested the light-emitting device <NUM> to emit light in the light emission pattern for the broad phase, and the imaging apparatus <NUM> is capturing, with short-time exposure, a scene in which the light-emitting markers <NUM> are repetitively emitting light in the pattern as illustrated in <FIG>, in response to the request.

Further, the synchronous processing section <NUM> performs the processing of acquiring data on the captured image to detect the images of the light-emitting markers in parallel. With a shorter exposure time, the images of the light-emitting markers can be more easily detected with light emission in the exposure time. Under that state, the image detection processing is repeated until an image appearance pattern that matches the sync pattern is obtained (N in S20). When such a pattern is detected (Y in S20), from the next frame, image appearance patterns for identifying the time grid are acquired (S22).

In a case where, when the frames for the light emission pattern for one unit have been captured, an image appearance pattern matches one of the variations of light emission patterns determined in advance for the respective time grids (Y in S24), it is determined that the exposure time overlaps the time grid corresponding to the pattern in question (S26). Thus, the synchronous processing section <NUM> generates control information that achieves light emission in the time grid in question and transmits the control information to the light-emitting device <NUM> through the control information transmitting section <NUM>, to thereby set a rough light emission time period (S28).

In a case where the image appearance patterns match none of the light emission patterns set for the respective time grids in S24, the processing starts again from the sync pattern detection (N in S24). Note that, the accuracy may be enhanced as follows: the processing in S20 to S26 is repeated a plurality of times, and of the obtained results, a result obtained at the highest frequency or a result obtained a predetermined number of times first is determined as the final value of the time grid corresponding to the exposure time. Further, when no result is obtained for a predetermined period of time, the processing returns to the pre-scan phase and starts again from the light-emitting device detection.

In the broad phase, the period of time in which light is to be emitted can be determined at the granularity similar to that of the exposure time. With this, at least the exposure time and the light emission can be synchronized with each other, so that the images of the light-emitting markers certainly appear in the captured image. In a case where strict time point adjustment is not required since the results are used for sensor fusion, for example, the following background phase may be omitted and the broad phase may be regularly performed. However, when the light emission time point is adjusted to the intermediate time point of the exposure time in the background phase, high robustness against the shifts of the light emission time point due to the clock difference can be achieved.

<FIG> is a diagram illustrating the principle of synchronous processing in the background phase. In <FIG>, the horizontal axis indicates elapsed time, the solid rectangle indicates an exposure time <NUM> for one frame, and the dashed rectangles indicate the light emission times of the light-emitting markers <NUM> (for example, light emission time <NUM>). However, the illustrated light emission times are for a plurality of frames. Since the background phase is basically performed during operation, the light emission time <NUM> is a short light emission time during operation (for example, <NUM> psec). As described above, the background phase is the processing for strictly adjusting the light emission time point to the intermediate time point of the exposure time.

In the broad phase, a timing corresponding to the exposure time <NUM> is determined on the time axis of the light-emitting device <NUM> at a resolution of approximately <NUM>µsec. When the light emission time is shorter than <NUM>µsec, however, light is possibly emitted at a time point shifted from the intermediate time point of the exposure time. Further, it is conceivable that due to the clock frequency difference between the apparatus, the light emission time point is relatively shifted from the intermediate time point of the exposure time over time.

In the example illustrated in <FIG>, an intermediate time point TL of the light emission time <NUM> comes earlier than an intermediate time point TE of the exposure time <NUM>. A temporal shift amount d is accordingly obtained to be added to the set light emission time point, so that the intermediate time point TL of the light emission time is adjusted to the exposure time point TE. To achieve this, the synchronous processing section <NUM> intentionally shifts the light emission time point, thereby acquiring the transition timing between the state in which the light-emitting markers <NUM> appear in a captured image and the state in which the light-emitting markers <NUM> do not appear in the captured image.

In the case of <FIG>, in a period in which the light emission time <NUM> of the light-emitting markers <NUM> is included in the exposure time <NUM>, the images of the light-emitting markers <NUM> appear in the captured image. When the light emission time <NUM> gets out of the exposure time <NUM>, the images of the light-emitting markers <NUM> no longer appear. A light emission time point getting out of the exposure time <NUM> in a shorter period of time as a result of gradual shifts means that the original light emission time point is biased toward the shift direction in the exposure time. In the example illustrated in <FIG>, when the light emission time <NUM> is advanced as indicated by the hollow arrow, the light emission time <NUM> gets out of the exposure time <NUM> relatively soon, so that the images do not appear in the captured image. The shift amount d of the original light emission time point TL is obtained from a time taken for the light emission time to get out of the exposure time, the amount of shift of the light emission time point in each frame, and the exposure time <NUM>.

<FIG> schematically illustrates an exemplary light emission pattern in the background phase. A manner of representation in <FIG> is similar to that in <FIG>. However, the time width on the horizontal axis is shorter than the time corresponding to one frame and is, for example, approximately <NUM>,<NUM>µsec. In the background phase, as described above, attention is paid to a timing at which the light emission time enters or gets out of the exposure time, and hence it is sufficient that the light emission time point is shifted by a time width approximately twice as long as the exposure time with the center being the set light emission time point indicated by the hollow arrow. Further, the state in which the light emission time is out of the exposure time does not last long so that a drop in accuracy of tracking the position and posture of the light-emitting device <NUM> is prevented. In the case where the light emission time point comes earlier than the intermediate time point of the exposure time as in <FIG>, the exposure time is in the range of a rectangle <NUM>, for example.

Further, when the light emission time is set to a short period of time, the shift amount d has a detection error at the corresponding granularity. For example, as illustrated in <FIG>, when the light emission time is <NUM>µsec, the detection error is within the range of ±<NUM>µsec. In the example illustrated in <FIG>, one light emission pattern includes <NUM> frames ranging from the frame <NUM> to the frame <NUM>. The light-emitting markers <NUM> repetitively emit light in the light emission pattern in question in units of <NUM> frames.

For the same purpose as the broad phase, a sync pattern <NUM> is provided to the first seven frames (frames <NUM> to <NUM>). In this example, the sync pattern <NUM> is a pattern in which the light-emitting markers <NUM> are alternately on and off. Since the set light emission time point is within the exposure time as a result of adjustment in the broad phase, in the sync pattern <NUM>, light is emitted at the set time point without shifting. Then, the light emission time point is shifted by a predetermined period of time from the frame (frame <NUM>) following the period of the sync pattern <NUM>. In the example illustrated in <FIG>, the light emission time point is advanced by <NUM> usec.

Then, when the amount of shift of the light emission time point reaches a predetermined upper limit time, light is emitted at a time point shifted in the opposite direction by the predetermined upper limit time. In the example illustrated in <FIG>, the light emission time point is advanced by <NUM>µsec over the <NUM> frames (frames <NUM> to <NUM>), and then, in the next frame (frame <NUM>), light is emitted at a time point delayed by <NUM>µsec from the original set time point. Thereafter, the light emission time point is shifted by the predetermined period of time in a similar manner. A time point just before the original light emission time point is the end of the light emission pattern for one unit (frames <NUM> to <NUM>).

When light emission in such a pattern is captured in the exposure time corresponding to the rectangle <NUM>, an image appearance pattern after the sync pattern <NUM> is {<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>}. That is, the light emission time gets out of the exposure time in the frame <NUM> and enters the exposure time again in the frame <NUM>, so that no image appears in the period of the frames <NUM> to <NUM>. Such an image appearance pattern changes depending on a relation between the exposure time and the original light emission time point.

<FIG> illustrates a relation between the original light emission time point and image appearance patterns in the background phase. From the light emission pattern illustrated in <FIG>, there are <NUM> image appearance patterns based on a temporal relation between the original light emission time point and the exposure time. However, the variations of image appearance patterns depend on the length of the exposure time or the light emission time, or how much the light emission time point is shifted at a time. When the original light emission time point matches the intermediate time point of the exposure time, the number of frames by which the light emission time gets out of the exposure time and the number of frames by which the light emission time enters the exposure time again to restore the initial state are the same. The pattern of "position <NUM>" corresponds to this case.

The case where the exposure time is in the rectangle <NUM> in <FIG> corresponds to the pattern of "position <NUM>. " Thus, the data as illustrated in <FIG> is prepared so that a shift of the original light emission time point from the exposure time can be obtained at a granularity of approximately <NUM>µsec on the basis of the actual image appearance patterns.

<FIG> is a flowchart illustrating a processing procedure by which the synchronous processing section <NUM> adjusts the light emission time point to the intermediate time point of the exposure time in the background phase. This flowchart starts under a state where the control information transmitting section <NUM> has requested the light-emitting device <NUM> to emit light in the light emission pattern for the background phase, and the imaging apparatus <NUM> is capturing, with short-time exposure, a scene in which the light-emitting markers <NUM> are repetitively emitting light in the pattern as illustrated in <FIG>, in response to the request.

Further, the synchronous processing section <NUM> performs the processing of acquiring data on the captured image to detect the images of the light-emitting markers in parallel. Further, the position and posture acquiring section <NUM> of the information processing apparatus <NUM> may acquire the position and posture of the light-emitting device <NUM> on the basis of the detected images, values measured by the IMU sensor, and the like, and the information processing section <NUM> may appropriately perform information processing on the basis of the position and the posture. Under this state, the image detection processing is repeated until an image appearance pattern that matches the sync pattern is obtained (N in S30). When the sync pattern is detected (Y in S30), from the next frame, image appearance patterns for identifying a light emission time point corresponding to the exposure time are acquired (S32).

In a case where, when the frames in the light emission pattern for one unit have been captured, the image appearance patterns match none of the estimated patterns as illustrated in <FIG>, the processing starts again from the sync pattern detection (N in S34). When an image appearance pattern matches one of the estimated patterns (Y in S34), as in "position <NUM>" of <FIG>, identification information indicating the original light emission time point that is associated with the pattern in question is acquired (S36).

The processing in S30 to S36 is repeated until the same result is obtained a predetermined number of times in a row (N in S38). When the same result is obtained by the predetermined number of times in a row (Y in S38), a correction amount corresponding to the light emission time point is acquired (S40). For example, on the basis of "position <NUM>" obtained in <FIG>, the temporal shift amount d illustrated in <FIG> is acquired as a correction amount. To achieve this, a table or conversion rule in which light emission time point identification information and correction amounts are associated with each other is prepared in advance. In a case where the acquired correction amount indicates that correction is required (Y in S42), the correction amount in question is added to the set original light emission time point so that the light emission time point is corrected (S44).

In a case where correction is not required, no change is made to the set light emission time point (N in S42). The processing procedure illustrated in <FIG> is regularly repeated as the background processing of operation processing so that the light emission time point can always be prevented from being shifted from the intermediate time point of the exposure time. <FIG> exemplifies setting information that is referred to in the acquisition of a correction amount from light emission time point identification information in S40 of <FIG>. In a correction amount table <NUM>, the correspondences between light emission time point identification information (light emission time point ID (Identification)) and times to be used for correction (correction amounts) are illustrated. The correction amounts can be derived from the image appearance patterns, the exposure time, the light emission time, and how much the light emission time point is shifted at a time, and are thus calculated in advance.

In the example illustrated in <FIG>, the <NUM> different light emission time points are defined. Thus, in the correction amount table <NUM>, correction amounts are set to the <NUM> pieces of identification information. In the example illustrated in <FIG>, the correction amounts are illustrated in units of <NUM>/<NUM> of the exposure time (=light emission time). As illustrated in <FIG>, in the case where the light emission time is shifted by <NUM>/<NUM> of the light emission time at a time, the correction amounts can be set at a granularity of <NUM> as illustrated in <FIG>. With correction at this granularity, "position <NUM>" and "position <NUM>," which are adjacent to "position <NUM>" at which the original light emission time point matches the intermediate time point of the exposure time as described above, have a correction amount "<NUM>" and do not require correction.

The positions farther from "position <NUM>" have the larger correction amounts, and the maximum correction amount is ±<NUM>. The set time point is delayed when a positive correction amount is added to the original set time point, and the set time point is advanced when a negative correction amount is added to the original set time point. For example, the example of <FIG> corresponds to "position <NUM>," and hence when the set time point is delayed by "shift amount d=<NUM>×light emission time," the light emission time point almost matches the intermediate time point of the exposure time. When the light emission time is <NUM>µsec, the light emission time point is delayed by <NUM>µsec in correction.

It is conceivable that, even with such adjustment, due to a subtle clock frequency difference between the imaging apparatus <NUM> and the light-emitting device <NUM>, the light emission time point is gradually shifted from the intermediate time point of the exposure time. Meanwhile, the clock frequency difference is obtained by observing the temporal change in such a shift, and hence the shift increase rate can be reduced through the artificial correction of the frequency itself.

<FIG> exemplifies the temporal change in a deviation in shift of the light emission time point from the intermediate time point of the exposure time. Here, a deviation D is a value obtained by the cumulative addition of a correction amount acquired in the i-th processing in S40 in the loop of the flowchart of <FIG>, that is, a shift amount di from the intermediate time point of the exposure time. The deviation D is defined by the following expression. <NUM>] <MAT>.

By the correction described above, the actual shift amount is regularly returned to <NUM>, but the acquired deviation D with the shift amount di added thereto monotonously increases or monotonously decreases due to the clock frequency difference. That is, in the case of the time course on the order of seconds as illustrated in <FIG>, the deviation linearly changes, and the slope thereof indicates the clock frequency difference. Thus, for example, when a deviation slope is obtained in a period A, the clock frequency is corrected in a direction that eliminates the slope. This changes the deviation slope as in a period B.

Then, when a deviation slope is obtained even in the period B, the clock frequency is corrected again to eliminate the slope. Similar processing is performed again also in a next period C to gradually reduce the temporal change in the deviation. In this way, the clock frequency is corrected to achieve an ideal state with no inclination like a deviation <NUM> indicated by the dotted line. The original clock of the light-emitting device is artificially corrected so that a state in which the intermediate time point of the exposure time and the light emission time point almost match each other can be maintained for a long period of time.

<FIG> is a flowchart illustrating a processing procedure by which the synchronous processing section <NUM> corrects the clock of the light-emitting device <NUM> in the background phase. This flowchart is performed in parallel to the light emission time point adjustment processing illustrated in <FIG>, but may be performed at frequency different from that of the light emission time point adjustment processing. First, as illustrated in <FIG>, the deviations D are plotted on a time t, and a correlation coefficient r thereof is obtained as follows (S50). <NUM>] <MAT>.

Here, t and D with overlines are average values. When a predetermined criterion indicating there is a correlation, such as that the correlation coefficient r has an absolute value larger than <NUM>, is not satisfied (N in S52), the plotting of the deviations D and the acquisition of the correlation coefficient r continue (S50). When the predetermined criterion indicating that there is a correlation is satisfied (Y in S52), the slope, which is a slope A, is obtained (S54). This processing is actually the processing of obtaining a slope a in a predetermined period immediately before the period by regression analysis as described below, and adding the slope a to the already obtained slope A. <NUM>] <MAT>.

Deviation addition parameters m and n are obtained as described below by using the thus obtained slope A and are transmitted to the light-emitting device <NUM> as control information so that the set value is updated (S56). Note that, the numerator of the parameter m is for the conversion of the time unit from see to usec. Further, the deviation addition parameters m and n are parameters that provide a clock correction amount (correction time) per unit time with n/m=A·<NUM>-<NUM>. <NUM>] <MAT>.

<FIG> is a diagram illustrating a method of correcting the clock frequency of the light-emitting device <NUM> with the deviation addition parameters m and n. The control section <NUM> of the light-emitting device <NUM> includes a correction circuit configured to correct the clock frequency from an oscillator. The correction circuit itself may be a general correction circuit. In the graph of <FIG>, the horizontal axis indicates real time and the vertical axis indicates the clock of the light-emitting device <NUM>. Further, a dashed-dotted line <NUM> indicates the initial state of the clock.

The control section <NUM> of the light-emitting device <NUM> corrects the clock on the basis of the deviation addition parameters m and n transmitted from the information processing apparatus <NUM> so that n is added to the time every time m. As indicated by a straight line <NUM> of <FIG>, when n is positive, the clock is advanced as illustrated in <FIG>. The clock is delayed when n is negative. With this, the slope changes by n/m=A·<NUM>-<NUM>, so that the change in deviation illustrated in <FIG> is theoretically eliminated. However, even such correction includes errors, and hence the deviation addition parameters m and n are obtained regularly and updated to reduce gradually the deviation.

According to the experimentation, with the introduction of the correction processing including the deviation addition, the time taken for the light emission time to get out of the exposure time was significantly extended. The light emission time point is triggered by a timer interrupt in the µsec order, but the clock correction illustrated in <FIG> allows the light emission time point to be controlled in smaller units.

<FIG> illustrates the sequence of the processing of synchronizing the exposure time and the light emission time point that the information processing apparatus <NUM> performs in cooperation with the light-emitting device <NUM>. First, in the pre-scan phase, the information processing apparatus <NUM> transmits, to the light-emitting device <NUM>, an instruction on light emission in the light emission pattern in the pre-scan phase (S100). In response to this, the light-emitting device <NUM> causes the light-emitting markers to emit light in the light emission pattern in the pre-scan phase illustrated in <FIG>, that is, in the whole period (S102). The information processing apparatus <NUM> detects the light-emitting device <NUM> by using an image obtained by capturing the scene (S104).

When the light-emitting device can be detected, the processing transitions to the broad phase where the information processing apparatus <NUM> transmits an instruction on light emission in the light emission pattern in the broad phase (S106). In response to this, the light-emitting device <NUM> causes the light-emitting markers to emit light in the pattern as illustrated in <FIG>, that is, causes the light-emitting markers to repetitively flash in a minimum unit being a relatively longer period of time similar to the exposure time (S108). The information processing apparatus <NUM> detects a time grid corresponding to the exposure time in the clock of the light-emitting device <NUM> (time axis) by using an image obtained by capturing the scene (S110).

When the time grid in question can be detected, the processing transitions to the background phase where the information processing apparatus <NUM> transmits an instruction on light emission in the light emission pattern in the background phase (S112). In response to this, the light-emitting device <NUM> causes the light-emitting markers to emit light in the pattern as illustrated in <FIG>, that is, causes the light-emitting markers to repetitively emit light for a short period of time suitable for the operation at time points shifted by frames (S114). With the instruction in S112, identification information regarding the time grid corresponding to the exposure time that is detected in the broad phase is specified so that the light emission time point can be set at least within the exposure time in the light-emitting device <NUM>.

The information processing apparatus <NUM> detects, by using an image obtained by capturing a scene in which the light emission time point is shifted with the basic point being the set time point in question, a shift of the set point from the intermediate time point of the exposure time as described above (S116). Then, the information processing apparatus <NUM> transmits a refresh instruction for correcting the set time point to the light-emitting device <NUM> as needed (S118). Further, the information processing apparatus <NUM> adds the obtained shift amount to acquire the temporal change in the deviation (S120). Then, the information processing apparatus <NUM> transmits a deviation correction instruction including the deviation addition parameters m and n to the light-emitting device <NUM> (S122).

The light-emitting device <NUM> corrects, on the basis of the transmitted refresh instruction and deviation correction instruction, the set light emission time point and the internal clock (S124). With this, a state in which the light emission time point is synchronized with the intermediate time point of the exposure time can be regularly created. As described above, the time taken for the light emission time point to get out of the exposure time is long, so that light emitted by the light-emitting device <NUM> that had gotten out of the field of view of the imaging apparatus <NUM> for a few minutes, for example, and has entered the field of view again can positively be captured, with the result that the possibility of losing the sight of the light-emitting device <NUM> can be significantly reduced.

Note that, since the clock frequencies are unique to the apparatus, when data on the frequency difference, that is, for example, the deviation addition parameters m and n, is acquired once, the data can be used for appropriate clock correction for the imaging apparatus <NUM> and the light-emitting device <NUM> corresponding to the obtained data. In this case, the deviation correction processing in the background phase can be omitted.

For example, a table in which the combinations of individual identification information regarding the imaging apparatus <NUM> and the light-emitting device <NUM> and deviation addition parameters are associated with each other is prepared in the synchronous processing section <NUM>. Then, deviation addition parameters are read on the basis of individual identification information regarding the imaging apparatus <NUM> and the light-emitting device <NUM> that are actually connected to each other so that correction can be made without deriving the parameters in question during operation. The synchronized state can therefore be maintained easily.

According to the present embodiment described above, in the technology for extracting the images of the light-emitting markers from a captured image, to thereby acquire the three-dimensional position and posture of the device including the light-emitting markers, the exposure time of the imaging apparatus and the light emission time of the light-emitting markers are shortened. This can make the images of the light-emitting markers stand out in terms of luminance in the captured image, and thus reduce a possibility that surrounding lighting or strong reflected light is erroneously recognized as the light-emitting markers. Further, the images of the light-emitting markers are prevented from being blurred due to the movements of the device, so that the position-time point correspondence can be clarified. Further, since the light-emitting device can cause the markers to emit light with small power consumption, the light-emitting device can be used as the battery power supply for a long period of time.

Further, results obtained by capturing the markers flashing in a unique light emission pattern are analyzed so that the exposure time of the imaging apparatus and the light emission time of the device, the apparatus and the device operating on the individual time axes, are adjusted to match each other at their central time points. With this, the images can positively be captured with a short light emission time. Further, the light emission time point can be handled as an image-capturing time point, and hence the measurement time points of the IMU sensor incorporated in the device and the image-capturing time point can be identified on the same time axis. As a result, information obtained by the image analysis and information obtained from values measured by the IMU sensor can accurately be integrated, and the accuracy of the finally obtained position and posture can therefore be enhanced.

The adjustment described above includes the phase in which the light-emitting markers are on in the whole period so that the device is detected, the phase in which the light-emitting markers are caused to flash in units of time similar to the exposure time so that the light emission time is included in the exposure time, and the phase in which the intermediate time point of an optimal light emission time is adjusted to the intermediate time point of the exposure time. With the separate phases, the last phase can be performed in parallel to the processing of acquiring the position and posture of the device, so that constant and long-term monitoring and adjustment that do not prevent the information processing, which is the primary processing, can be achieved. Thus, the deviation of the clock unique to the apparatus can be evaluated for a long term, and the accumulation of time shifts can adaptively be eliminated, so that the synchronized state can be maintained for a long period of time with the less often performed correction processing.

The present invention is described above on the basis of the embodiment. The above-mentioned embodiment is exemplary, and it will be understood by those skilled in the art that various modifications can be made to the combinations of the components and the processing processes and that such modifications are also within the scope of the present invention as defined by the claims.

For example, in the present embodiment, the light emission in the broad phase has the two states of the on state and the off state, and the image appearance patterns are indicated by the two values, that is, the numerical values of <NUM> and <NUM>. A plurality of light emission luminance stages may, however, be employed so that the image appearance patterns are indicated by three or more values depending on luminance. To identify a time grid corresponding to an exposure time, the time grids are completely different from each other in numerical sequence indicating an image appearance pattern. Thus, as the number of time grids obtained by division increases, the number of frames required for identifying the time grids increases. With numerical sequences including three or more values, the time grids can be identified with a fewer number of frames, so that the broad phase can be shortened.

Claim 1:
An information processing apparatus (<NUM>) comprising:
a captured image acquiring section (<NUM>) configured to acquire data on a moving image obtained by an imaging apparatus (<NUM>) capturing a device (<NUM>) including a light-emitting marker (<NUM>) with an exposure time shorter than a period of one frame;
a synchronous processing section (<NUM>) configured to request the device to cause the light-emitting marker to emit light in a predetermined flashing pattern, with each flash being within a minimum unit being a time obtained by dividing the period of the one frame by a predetermined number and configured to identify the exposure time on a time axis of the device, based on whether or not an image of the light-emitting marker appears in a predetermined number of frames of the moving image obtained by capturing the device;
a control information transmitting section (<NUM>) configured to request the device to cause the light-emitting marker to emit light at a light emission time point corresponding to the identified exposure time;
a position and posture acquiring section (<NUM>) configured to acquire position and posture information regarding the device, based on the image in a frame of the moving image of the light-emitting marker emitting light at the light emission time point for a fixed period of time equal to or shorter than the exposure time; and
an output data generating section (<NUM>) configured to generate and output data based on the position and posture information.