Head mounted display device, method of controlling head mounted display device, system, synchronous control apparatus, and method of controlling synchronous control apparatus

A head mounted display device including a first image capturing unit and a second image capturing unit different from the first image capturing unit comprises a first generation unit configured to generate, based on a signal representing an image output timing of the first image capturing unit, a first signal for controlling a start of exposure of the second image capturing unit, and supply the generated first signal to the second image capturing unit, and a second generation unit configured to generate, based on the signal representing the image output timing of the first image capturing unit, a second signal for controlling a start of measurement of a sensor that measures a position and orientation of the sensor, and supply the generated second signal to the sensor.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a mixed reality presentation technique.

Description of the Related Art

These days, an MR (mixed Reality) technique is known as a technique of seamlessly blending physical and virtual worlds in real time. One of MR techniques is an MR system using a video see-through HMD (Head Mounted Display: to be referred to as an “HMD” if necessary). In the MR system, an HMD-incorporated image capturing unit captures an image of an object almost coincident with an object observed from the pupil position of an HMD wearer. Then, CG (Computer Graphics) is superimposed and displayed on the captured image, and the resultant image is presented to the HMD wearer. The HMD wearer can experience the MR space.

The MR system obtains the position and orientation of the HMD by performing arithmetic processing using captured images and various kinds of sensor information. Image capturing units and various sensors desirably operate as synchronously as possible. For example, Japanese Patent Laid-Open No. 2000-341719 discloses a technique of establishing synchronization by supplying a common driving signal and sync signal to a plurality of image capturing units. Japanese Patent Laid-Open No. 2006-005608 discloses a technique of establishing synchronization by making the barycenters of exposure times coincide with each other for a plurality of image capturing units having different exposure times.

However, the conventional techniques have the following problems. In an arrangement typified by Japanese Patent Laid-Open Nos. 2000-341719 and 2006-005608, only image capturing units are synchronized. In a system using various sensors in addition to image capturing units, like the MR system, if the image capturing units and the sensors operate asynchronously, no satisfactory arithmetic accuracy may be obtained. In this case, misalignment may occur between a captured image and CG.

SUMMARY OF THE INVENTION

The present invention provides a technique for synchronously operating an image capturing unit and an orientation sensor.

According to the first aspect of the present invention, there is provided a head mounted display device including a first image capturing unit and a second image capturing unit different from the first image capturing unit, comprising: a first generation unit configured to generate, based on a signal representing an image output timing of the first image capturing unit, a first signal for controlling a start of exposure of the second image capturing unit, and supply the generated first signal to the second image capturing unit: and a second generation unit configured to generate, based on the signal representing the image output timing of the first image capturing unit, a second signal for controlling a start of measurement of a sensor that measures a position and orientation of the sensor, and supply the generated second signal to the sensor.

According to the second aspect of the present invention, there is provided a method of controlling a head mounted display device including a first image capturing unit and a second image capturing unit different from the first image capturing unit, the method comprising: generating, based on a signal representing an image output timing of the first image capturing unit, a first signal for controlling a start of exposure of the second image capturing unit, and supplying the generated first signal to the second image capturing unit: and generating, based on the signal representing the image output timing of the first image capturing unit, a second signal for controlling a start of measurement of a sensor that measures a position and orientation of the head mounted display device, and supplying the generated second signal to the sensor.

According to the third aspect of the present invention, there is provided a system comprising a head mounted display device including a first image capturing unit and a second image capturing unit different from the first image capturing unit, and an image processing apparatus, the head mounted display device including: a first generation unit configured to generate, based on a signal representing an image output timing of the first image capturing unit, a first signal for controlling a start of exposure of the second image capturing unit, and supply the generated first signal to the second image capturing unit; and a second generation unit configured to generate, based on the signal representing the image output timing of the first image capturing unit, a second signal for controlling a start of measurement of a sensor that measures a position and orientation of the sensor, and supply the generated second signal to the sensor, and the image processing apparatus including: an obtaining unit configured to obtain, from the head mounted display device, an image captured by the first image capturing unit, an image captured by the second image capturing unit in response to reception of the first signal, and a position and orientation measured by the sensor in response to reception of the second signal; a unit configured to generate an image of a virtual space based on the image captured by the second image capturing unit and the position and orientation measured by the sensor, and generate a composite image of the generated image of the virtual space and the image captured by the first image capturing unit; and a unit configured to output the composite image to the head mounted display device.

According to the fourth aspect of the present invention, there is provided a synchronous control apparatus comprising: an image capturing unit: a sensor configured to measure a position and orientation of the sensor; an obtaining unit configured to obtain a signal out of an external sync input signal externally input to control the image capturing unit, and a sync output signal output from the image capturing unit: and a control unit configured to control to execute measurement processing of the sensor at an arbitrary timing in an exposure time in the image capturing unit based on the signal obtained by the obtaining unit.

According to the fifth aspect of the present invention, there is provided a method of controlling a synchronous control apparatus including: an image capturing unit: and a sensor configured to measure a position and orientation of the sensor, the method comprising: obtaining a signal out of an external sync input signal externally input to control the image capturing unit, and a sync output signal output from the image capturing unit; and controlling to execute measurement processing of the sensor at an arbitrary timing in an exposure time in the image capturing unit based on the obtained signal.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note, the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but limitation is not made to an invention that requires all such features, and multiple such features may be combined as appropriate.

Furthermore, in the attached drawings, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.

First Embodiment

First, the arrangement of an MR system according to the first embodiment will be exemplified with reference toFIG.1. As shown inFIG.1, the MR system according to the embodiment includes an HMD101serving as an example of a head mounted display device, a computer apparatus103that generates an image of a mixed reality space (space obtained by blending physical and virtual spaces) to be displayed on the HMD101, and a controller102that mediates between the HMD101and the computer apparatus103.

First, the HMD101will be described. The HMD101includes an image capturing unit that captures an image of a physical space, a sensor that measures (measurement processing) the position and orientation of the HMD101, and a display unit that displays an image of a mixed reality space transmitted from an image processing apparatus104. The HMD101also functions as a synchronous control apparatus for these devices. The HMD101transmits to the controller102an image captured by the image capturing unit and the position and orientation of the HMD101measured by the sensor. The HMD101receives from the controller102an image of the mixed reality space generated by the computer apparatus103based on the captured image and the position and orientation, and displays the received image on the display unit. The image of the mixed reality space is presented in front of the eyes of a user wearing the HMD101on his/her head.

The HMD101may operate by power supplied from the controller102or by power supplied from the battery of the HMD101. That is, a method of supplying power to the HMD101is not limited to a specific one.

InFIG.1, the HMD101and the controller102are connected by wire. However, the connection form between the HMD101and the controller102is not limited to wire, and may be wireless or a combination of wireless and wire. That is, the connection form between the HMD101and the controller102is not limited to a specific one.

Next, the controller102will be described. The controller102performs various image processes (for example, resolution conversion, color space conversion, distortion correction of the optical system of the image capturing unit of the HMD101, and encoding) on a captured image transmitted from the HMD101. Then, the controller102transmits to the computer apparatus103the captured image having undergone the image processes and a position and orientation transmitted from the HMD101. The controller102performs similar image processes on an image of the mixed reality space transmitted from the computer apparatus103, and transmits the processed image to the HMD101.

Next, the computer apparatus103will be described. The computer apparatus103obtains the position and orientation (position and orientation of the image capturing unit of the HMD101) of the HMD101based on a captured image and a position and orientation received from the controller102, and generates an image of a virtual space viewed from a viewpoint having the obtained position and orientation. The computer apparatus103generates a composite image (image of the mixed reality space) of the image of the virtual space and the captured image transmitted from the HMD101via the controller102, and transmits the generated composite image to the controller102.

Processing of generating a composite image from a captured image and an image of the virtual space will be explained with reference toFIG.2. A captured image201includes a marker202(the number of markers is one inFIG.2for descriptive convenience, but a plurality of markers are included in practice) artificially arranged in the physical space. The computer apparatus103extracts the marker202from the captured image201, and obtains the position and orientation of the HMD101based on the extracted marker202and a position and orientation received from the controller102. The computer apparatus103then generates an image203of the virtual space viewed from a viewpoint having the obtained position and orientation. The image203includes a virtual object204.

The computer apparatus103generates an image205of the mixed reality space as a composite image of the captured image201and the image203of the virtual space. The computer apparatus103transmits the generated image205to the HMD101. Note that the captured image and the image of the virtual space are composited using information about the depth in the 3D space and information about the transparency of a virtual object. This enables generating a composite image considering the positional relationship in depth between a physical object and the virtual object, or a composite image in which the virtual object is composited in a semitransparent state.

The computer apparatus103and the controller102are separate apparatuses inFIG.1, but may be integrated. In the embodiment, a form will be explained in which the computer apparatus103and the controller102are integrated. An apparatus constituted by integrating the computer apparatus103and the controller102will be referred to as the image processing apparatus104.

The image capturing unit of the HMD can selectively use a rolling shutter image sensor and a global shutter image sensor in consideration of various factors such as the number of pixels, image quality, noise, sensor size, power consumption, and cost, or can use them in combination depending on the intended use. For example, the rolling shutter image sensor capable of obtaining a higher-quality image is used to capture an image that is composited with an image of the virtual space, and the global shutter image sensor free from image flow is used to capture an image of a marker. The image flow is a phenomenon arising from a rolling shutter operation principle that starts exposure processing sequentially for respective lines in the scanning direction. More specifically, the image flow is known as a phenomenon in which a time lag is generated between the timings of exposure of respective lines, as shown inFIG.3, and when an image capturing unit or an object moves during the exposure time, the object is deformed and recorded as if it flowed. InFIG.3, the abscissa represents the time, and “exposure time (rolling)” represents the exposure time of each line subjected to image capturing by the rolling shutter image sensor. “Exposure time (global)” represents the exposure time of each line subjected to image capturing by the global shutter image sensor. In the global shutter type, exposure processing is performed simultaneously on all lines, so no time lag is generated between the exposure timings of respective lines and no image flow occurs.

As shown inFIG.3, when the image capturing unit receives an external sync input (external sync input signal), exposure of each pixel starts after a processing time tExp_Readytill the start of exposure, and exposure of each pixel is performed during an exposure time tExpset in the image capturing unit. Output of a sync signal (sync output, sync output signal) and an image signal (image output) from the image capturing unit starts tImg_Outafter the end of exposure of each line. As shown inFIG.3, in the global shutter image capturing unit, the time till an exposure start time tExp_Startafter a time tSync_Inat which an external sync input is input, an exposure center time tExp_Center, and an exposure end time tExp_Endare uniquely determined as values unique to each image sensor. To the contrary, in the rolling shutter image capturing unit, the exposure start time, the exposure center time, and the exposure end time can take different values within a range equivalent to an exposure start timing difference ΔtExp_Startbetween start and final lines depending on a line whose exposure time is set as a reference.

As for a sensor (orientation sensor) that measures the position and orientation of the HMD, when an external sync input is received, measurement (data obtainment) of the position and orientation starts after a processing time tSens_Readytill the start of measurement, and output (data output) of the position and orientation starts a time tSense_Outafter the end of measurement.

FIG.4is a timing chart for explaining synchronization between the image capturing unit and the orientation sensor based on an external sync input. InFIG.4, devices are synchronized by supplying a common external sync input at the time tSync_Into a rolling shutter image capturing unit1, a global shutter image capturing unit2, and the orientation sensor. As described with reference toFIG.3, the image capturing unit1, the image capturing unit2, and the orientation sensor have unique processing times, so lags are generated in the time when the image capturing unit actually performs exposure, and the time when the orientation sensor performs measurement in response to an external sync input. For example, when the exposure center time of the image capturing unit1or2is regarded as a reference, the position and orientation measurement time of the orientation sensor is tSens_Meas, the exposure center time of the image capturing unit1is tExp_Center_CAM1, and the exposure center time of the image capturing unit2is tExp_Center_CAM2. These times do not coincide with each other.

An arrangement will be considered in which the image capturing unit1is used to obtain a captured image that is composited with an image of the virtual space, and the image capturing unit2is used to capture an image of a marker. An image of the virtual space is generated based on a captured image of the marker obtained by the image capturing unit2and a position and orientation obtained by the orientation sensor, and is influenced by an error arising from the lag between the obtaining timings of the captured image and the position and orientation. Further, the obtaining timing of a captured image obtained by the image capturing unit1differs from the obtaining timing of the captured image by the image capturing unit2and the obtaining timing of the position and orientation by the orientation sensor. The influence on the positional accuracy of the image of the virtual space superimposed on the captured image becomes larger. To prevent this, it can be said that it is desirable that the MR system performs a synchronous operation on the image capturing unit1, the image capturing unit2, and the orientation sensor so that the exposure timing of the image capturing unit1, the exposure timing of the image capturing unit2, and the position and orientation obtaining timing of the orientation sensor coincide with each other at higher precision.

Next, the functional arrangements of the HMD101and the image processing apparatus104will be exemplified with reference to the block diagram ofFIG.5. First, the HMD101will be explained. An image capturing unit501captures an image of the physical space that is composited with an image of the virtual space. The image capturing unit501includes a left-eye image capturing portion and a right-eye image capturing portion. The left-eye image capturing portion captures a moving image of the physical space corresponding to the left eye of the wearer of the HMD101, and outputs an image (captured image) of each frame in the moving image. The right-eye image capturing portion captures a moving image of the physical space corresponding to the right eye of the wearer of the HMD101, and outputs an image (captured image) of each frame in the moving image. That is, the image capturing unit501obtains captured images as stereo images having a parallax that almost coincide with the positions of the left and right eyes of the wearer of the HMD101. Note that in the HMD of the MR system, the central optical axis of the image capturing range of the image capturing unit is desirably arranged to almost coincide with the line-of-sight direction of the wearer of the HMD.

Each of the left- and right-eye image capturing portions includes an optical system and an image capturing device. Light incoming from the outside enters the image capturing device via the optical system, and the image capturing device outputs an image corresponding to the entering light as a captured image. As the image capturing device of the image capturing unit501, a rolling shutter image sensor is used. The image capturing unit501periodically outputs a captured image and also outputs a sync signal representing the output start timing (image output timing) of the captured image.

An image capturing unit502includes a plurality of image capturing portions for capturing a marker arranged in the physical space, and obtains captured images as stereo images having a parallax. Each image capturing portion captures a moving image of the physical space and outputs an image (captured image) of each frame in the moving image. Each image capturing portion of the image capturing unit502includes an optical system and an image capturing device. Light incoming from the outside enters the image capturing device via the optical system, and the image capturing device outputs an image corresponding to the entering light as a captured image. As the image capturing device of the image capturing unit502, a global shutter image sensor is used. A plurality of image capturing portions of the image capturing unit502start exposure every time they receive a sync signal from a generation unit506, and end the exposure after the lapse of an exposure time of one frame.

An orientation sensor503measures the position and orientation of the HMD101every time it receives a sync signal from the generation unit506, and outputs the measured position and orientation. The orientation sensor503is implemented by a magnetic sensor, an ultrasonic sensor, an acceleration sensor, an angular velocity sensor, or the like.

A display unit504includes a right-eye display portion and a left-eye display portion. The left-eye display portion displays a left-eye image of the mixed reality space received from the image processing apparatus104via an/F508. The right-eye display portion displays a right-eye image of the mixed reality space received from the image processing apparatus104via the I/F508. Each of the left- and right-eye display portions includes a display optical system and a display element. The display optical system may be not only an eccentric optical system such as a free-form surface prism, but also a normal co-axial optical system or an optical system having a zoom mechanism. The display element is, for example, a compact liquid crystal display, an organic EL display, or a MEMS retina scanning device. Light traveling from an image displayed on the display element enters the eyes of the wearer of the HMD101via the display optical system.

A detection unit505detects a sync signal (signal representing the start timing of image output from the image capturing unit501) output from the image capturing unit501, and upon detecting the sync signal, notifies the generation unit506of the detection.

When the generation unit506receives the notification from the detection unit505, it generates sync signals to be supplied to the image capturing unit502and the orientation sensor503based on the sync signal detected by the detection unit505, and supplies the generated sync signals to the image capturing unit502and the orientation sensor503. A setting unit507sets various parameters used in the operation of the HMD101.

All of a captured image output from the image capturing unit501, a captured image output from the image capturing unit502, and a position and orientation output from the orientation sensor503are transmitted to the image processing apparatus104via the I/F508.

Next, the image processing apparatus104will be described. The image processing apparatus104receives via an I/F509the captured images and the position and orientation transmitted from the HMD101. A processing unit510performs various image processes on the captured images received from the HMD101via the I/F509.

A generation unit511extracts (recognizes) markers from the left- and right-eye captured images having undergone the image processes by the processing unit510. The generation unit511obtains the positions and orientations of the left- and right-eye image capturing portions based on the markers and the position and orientation received from the HMD101via the I/F509. Processing for obtaining the position and orientation of an image capturing portion based on a marker in an image, and a position and orientation measured by a sensor included in an HMD together with the image capturing unit is generally known, so a description of this technique will be omitted.

Various data (virtual space data) necessary to render an image of the virtual space is saved in a content DB (DataBase)512. The virtual space data includes, for example, data defining each virtual object constituting the virtual space (for example, data defining the geometric shape, color, texture, arrangement position and orientation, and the like of the virtual object). Also, the virtual space data includes, for example, data defining a light source arranged in the virtual space (for example, data defining the type, position and orientation, and the like of the light source).

A composition unit513builds a virtual space using the virtual space data saved in the content DB512. The composition unit513generates an image L of the virtual space viewed from a viewpoint having the position and orientation of the left-eye image capturing portion obtained by the generation unit511. The composition unit513generates an image R of the virtual space viewed from a viewpoint having the position and orientation of the right-eye image capturing portion obtained by the generation unit511. The composition unit513generates a composite image L as a left-eye image L of the mixed reality space by compositing the image L of the virtual space and an image captured by the left-eye image capturing portion. The composition unit513generates a composite image R as a right-eye image R of the mixed reality space by compositing the image R of the virtual space and an image captured by the right-eye image capturing portion.

A processing unit514performs various image processes on the image L of the mixed reality space and the image R of the mixed reality space generated by the composition unit513. The processing unit514transmits, to the HMD101via the I/F509, the image L of the mixed reality space and the image R of the mixed reality space having undergone the image processes. A setting unit515sets various parameters used in the operation of the image processing apparatus104.

Next, an example of generating by the generation unit506a sync signal to be supplied to the image capturing unit502will be explained with reference to a timing chart on the upper part ofFIG.6. In the following description, an “image capturing unit1” corresponds to the image capturing unit501, and an “image capturing unit2” corresponds to the image capturing unit502. InFIG.6, the abscissa represents the time.

When the detection unit505detects a sync signal (sync output) from the image capturing unit1, it notifies the generation unit506of the detection. Upon receiving the notification, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure of the image capturing unit2” after an offset601from the detection timing of the sync signal. Here, t61is the time till a “center time (exposure center time) in the exposure time of the next frame at the detection timing in the image capturing unit1” after the detection timing of the sync signal, t62is a “processing time till the start of exposure” unique to the image capturing unit2, and t63is the exposure time of one frame in the image capturing unit2. At this time, the offset601can be calculated according to the following equation:
offset 601=t61−t62−t63/2

Assume that the offset601is obtained and set in advance by the setting unit507. Note that the setting unit507may obtain and set the offset601periodically or irregularly. The generation unit506supplies the generated “sync signal for controlling the start of exposure of the image capturing unit2” to the image capturing unit2.

Upon receiving the generated “sync signal for controlling the start of exposure of the image capturing unit2”, the image capturing unit2starts exposure. Since the center time of the exposure time coincides with the center time in the exposure time of the image capturing unit1, the image capturing units1and2perform exposure synchronously as a result. That is, the generation unit506generates a sync signal to be supplied to the image capturing unit2so that the center time of the exposure time of the image capturing unit1coincides with that of the exposure time of the image capturing unit2.

Even if devices to perform a synchronous operation include a device having no external sync input function, the device is set as the reference of the synchronous operation and the synchronous operation between the devices can be implemented.

A timing chart on the lower part ofFIG.6shows generation of an external sync input to the image capturing unit2when an external sync input to the image capturing unit1is set as a reference. When the HMD101detects an external sync input to the image capturing unit1, it generates an external sync input to the image capturing unit2after an offset601′. Letting64be the time till a “center time in the exposure time of the next frame at the detection timing in the image capturing unit1” after the detection timing of the external sync input to the image capturing unit1, the offset601′ can be calculated according to the following equation:
offset 601′=t64−t62−t63/2

In the above description, the synchronous timing in the synchronous operation is the center time in the exposure time of the image capturing unit1.

However, the setting of the synchronous timing in the embodiment is not limited to this and can be an arbitrary timing in the exposure time. A synchronous operation when a sync signal from the image capturing unit1in the timing chart shown in the upper part ofFIG.6is set as a reference will be described. However, the synchronous operation can be implemented similarly even when an external sync input to the image capturing unit1in the timing chart shown in the lower part ofFIG.6is set as a reference.

The synchronous operation of the image capturing unit501, the image capturing unit502, and the orientation sensor503according to the embodiment will be described with reference toFIG.7. InFIG.7, the abscissa represents the time.

When the detection unit505detects a sync signal (sync output) from the image capturing unit1, it notifies the generation unit506of the detection. Upon receiving the notification, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure of the image capturing unit2” after an offset701from the detection timing of the sync signal. The offset701is set by the setting unit507similarly to the offset601. Upon receiving the generated “sync signal for controlling the start of exposure of the image capturing unit2”, the image capturing unit2starts exposure and outputs, in accordance with the sync output of the image capturing unit2, data of each line captured by the exposure.

Upon receiving the notification, the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement of a position and orientation by the orientation sensor503” after an offset702from the detection timing of the sync signal. The offset702can be calculated using the time t61and a “processing time tSens_Readytill the start of measurement of a position and orientation after the orientation sensor503receives an external sync input” according to the following equation:
offset 702=t61−tSens_Ready

Assume that the offset702is obtained and set in advance by the setting unit507. The generation unit506supplies the generated “sync signal for controlling the start of measurement of a position and orientation by the orientation sensor503” to the orientation sensor503. Upon receiving the generated “sync signal for controlling the start of measurement of a position and orientation by the orientation sensor503”, the orientation sensor503starts measurement (data obtainment) of a position and orientation. That is, the generation unit506generates a sync signal to be supplied to the image capturing unit2and a sync signal to be supplied to the orientation sensor503so that the center time of the exposure time of the image capturing unit1, that of the exposure time of the image capturing unit2, and the timing (measurement timing) of data obtainment by the orientation sensor503coincide with each other.

Control of the synchronous operation of the image capturing unit501, the image capturing unit502, and the orientation sensor503by the HMD101according to the embodiment will be described with reference to the flowchart ofFIG.8. If the detection unit505detects a sync signal (sync output) from the image capturing unit501, it notifies the generation unit506of the detection and the process advances to step S802via step S801. If the detection unit505does not detect a sync signal (sync output) from the image capturing unit501, the process stands by in step S801.

In step S802, upon receiving the notification from the detection unit505, the generation unit506generates a “sync signal to be supplied to the image capturing unit502” after the offset701from the detection timing of the sync signal, and supplies the generated sync signal to the image capturing unit502. Further, the generation unit506generates a “sync signal to be supplied to the orientation sensor503” after the offset702from the detection timing of the sync signal, and supplies the generated sync signal to the orientation sensor503.

As described above, according to the first embodiment, the synchronous operation of the image capturing unit501, the image capturing unit502, and the orientation sensor503can be implemented based on the detection timing of a sync signal from the image capturing unit501or an external sync input to the image capturing unit501. Since the processing times and the like of respective devices are considered, the image capturing and data obtaining timings of the respective devices can coincide with an arbitrary timing in the exposure time. A more real MR experience free from misalignment between a captured image and an image of the virtual space can be provided. Even when the image capturing unit501does not have the external sync input function, the synchronous operation can be implemented and the choice of devices can be widened.

Second Embodiment

In the following embodiments including the second embodiment, differences from the first embodiment will be explained, and the remaining parts are similar to the first embodiment, unless otherwise specified. In the first embodiment, an arrangement has been described in which sync signals to respective devices are generated in consideration of the processing times and the like of the respective devices with reference to the detection timing of a sync signal from the image capturing unit501. In the second embodiment, an arrangement will be explained in which the synchronous operation between devices is performed when the setting of the exposure time of an image capturing unit501or502is changed or the setting of the synchronous timing is changed.

First, the functional arrangements of an HMD101and an image processing apparatus104will be exemplified with reference to the block diagram ofFIG.9. A generation unit506generates a sync signal to be supplied to the image capturing unit502and a sync signal to be supplied to an orientation sensor503under the control of a synchronous control unit901.

The synchronous control unit901controls generation of a sync signal by the generation unit506in accordance with a change of the setting of the exposure time of the image capturing unit501or502by a setting unit507. The synchronous control unit901controls generation of a sync signal by the generation unit506in accordance with a change of the setting of a time (line corresponding to the time) in the exposure time of the image capturing unit501with which the generation timing of a sync signal to be supplied to the image capturing unit502or the orientation sensor503is synchronized.

Next, generation processing of generating sync signals to be supplied to the image capturing unit502and the orientation sensor503in consideration of a lag between the exposure times of lines of an image captured by a rolling shutter image sensor will be described with reference toFIG.10.

When a detection unit505detects a sync signal (sync output) from an image capturing unit1, it notifies the generation unit506of the detection. Upon receiving the notification, the generation unit506generates a sync signal to be supplied to an image capturing unit2and a sync signal to be supplied to the orientation sensor503in accordance with the detection timing of the sync signal under the control of the synchronous control unit901.

Assume that the exposure time of the image capturing unit1and that of the image capturing unit2are equal, and the setting unit507sets the “exposure start time of the start line of an image captured by the image capturing unit1as the reference of the synchronous timing”. In this case, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure of the image capturing unit2” after an offset1001from the detection timing of the sync signal. The offset1001can be obtained as a result of subtracting the time t62from a time till the “exposure start time of the start line of an image captured by the image capturing unit1” after the detection timing of the sync signal. Also, the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement of a position and orientation by the orientation sensor” after an offset1004from the detection timing of the sync signal. The offset1004can be obtained as a result of subtracting the processing time tSens_Ready from a time till the “exposure start time of the start line of an image captured by the image capturing unit1” after the detection timing of the sync signal.

Assume that the exposure time of the image capturing unit1and that of the image capturing unit2are equal, and the setting unit507sets the “exposure center time of the center line of an image captured by the image capturing unit1as the reference of the synchronous timing”. In this case, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure of the image capturing unit2” after an offset1002from the detection timing of the sync signal. The offset1002can be obtained as a result of subtracting the time t62from a time till the “exposure start time of the center line of an image captured by the image capturing unit1” after the detection timing of the sync signal. Also, the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement of a position and orientation by the orientation sensor” after an offset1005from the detection timing of the sync signal. The offset1005can be obtained as a result of subtracting the processing time tSens_Ready from a time till the “exposure start time of the center line of an image captured by the image capturing unit1” after the detection timing of the sync signal.

Assume that the exposure time of the image capturing unit1and that of the image capturing unit2are equal, and the setting unit507sets the “exposure end time of the final line of an image captured by the image capturing unit1as the reference of the synchronous timing”. In this case, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure of the image capturing unit2” after an offset1003from the detection timing of the sync signal. The offset1003can be obtained as a result of subtracting the time t62from a time till the “exposure start time of the final line of an image captured by the image capturing unit1” after the detection timing of the sync signal. Also, the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement of a position and orientation by the orientation sensor” an offset1006after the detection timing of the sync signal. The offset1006can be obtained as a result of subtracting the processing time tSens_Ready from a time till the “exposure start time of the final line of an image captured by the image capturing unit1” after the detection timing of the sync signal.

In this manner, the synchronous control unit901supplies offsets corresponding to the reference of the synchronous timing to the generation unit506so that sync signals to be supplied to the image capturing unit2and the orientation sensor are generated based on the offsets corresponding to the reference of the synchronous timing.

As for arbitrary timings such as the exposure start time of the final line and the exposure end time of the start line, the synchronous operation can be performed by adjusting offsets to the image capturing unit2and the orientation sensor based on various setting contents by the setting unit507.

Next, generation processing of generating sync signals to be supplied to the image capturing unit2and the orientation sensor using, as the reference of the synchronous timing, the exposure center time of the center line of an image captured by a rolling shutter image sensor will be described with reference toFIG.11.

When the detection unit505detects that the image capturing unit1has output a sync signal corresponding to a captured image (frame (N−1)) of the (N−1)th frame, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of a frame N) of the image capturing unit2” after an offset1101, and supplies it to the image capturing unit2. The offset1101can be obtained by the calculation method described with reference toFIG.10(offset calculation method when the “exposure center time of the center line of an image captured by the image capturing unit1is set as the reference of the synchronous timing”). When the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N−1), the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame N) of a position and orientation by the orientation sensor” after an offset1104, and supplies it to the orientation sensor. The offset1104can be obtained by the calculation method described with reference toFIG.10(offset calculation method when the “exposure center time of the center line of an image captured by the image capturing unit1is set as the reference of the synchronous timing”).

Assume that the setting unit507performs a setting (setting change2) of changing the exposure time of the image capturing unit2until the detection unit505detects a sync signal corresponding to the frame N after detecting a sync signal corresponding to the frame (N−1).

At this time, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame N, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of a frame (N+1)) of the image capturing unit2” after an offset1102, and supplies it to the image capturing unit2. The offset1102can be obtained by a method similar to that of the offset1101. Further, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame N, the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame (N+1)) of a position and orientation by the orientation sensor” after an offset1104, and supplies it to the orientation sensor.

Assume that the setting unit507performs a setting (setting change1) of changing the exposure time of the image capturing unit1until the detection unit505detects a sync signal corresponding to the frame (N+1) after detecting a sync signal corresponding to the frame N.

At this time, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N+1), the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of a frame (N+2)) of the image capturing unit2” after an offset1103, and supplies it to the image capturing unit2. The offset1103can be obtained by a method similar to that of the offset1101. Further, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N+1), the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame (N+2)) of a position and orientation by the orientation sensor” after an offset1105, and supplies it to the orientation sensor. The offset1105can be obtained by a method similar to that of the offset1104.

After that, no setting change by the setting unit507is performed, and no offset switching occurs. In this way, the synchronous control unit901obtains offsets and supplies them to the generation unit506so that sync signals to be supplied to the image capturing unit2and the orientation sensor are generated based on the offsets corresponding to a setting change of the exposure time of the image capturing unit by the setting unit507.

The above-described processing can be performed to implement a synchronous operation in which the exposure center time of the image capturing unit1, that of the image capturing unit2, and the data obtaining timing of the orientation sensor coincide with each other even when a setting change is performed.

Next, generation processing of generating sync signals to be supplied to the image capturing unit2and the orientation sensor using, as the reference of the synchronous timing, the exposure start time of the start line of an image captured by a rolling shutter image sensor will be described with reference toFIG.12.

When the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N−1), the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of the frame N) of the image capturing unit2” after an offset1201, and supplies it to the image capturing unit2. The offset1201can be obtained by the calculation method described with reference toFIG.10(offset calculation method when the “exposure start time of the start line of an image captured by the image capturing unit1is set as the reference of the synchronous timing”). When the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N−1), the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame N) of a position and orientation by the orientation sensor” after an offset1203, and supplies it to the orientation sensor. The offset1203can be obtained by the calculation method described with reference toFIG.10(offset calculation method when the “exposure start time of the start line of an image captured by the image capturing unit1is set as the reference of the synchronous timing”).

Assume that the setting unit507performs the setting (setting change2) of changing the exposure time of the image capturing unit2until the detection unit505detects a sync signal corresponding to the frame N after detecting a sync signal corresponding to the frame (N−1).

Even if the setting change2is performed, the relationship between the exposure start time of the image capturing unit1and the processing time till the exposure start time after the external sync input of the image capturing unit2does not change, so the offset need not be changed.

Hence, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame N, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of the frame (N+1)) of the image capturing unit2” after the offset1201, and supplies it to the image capturing unit2. Further, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame N, the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame (N+1)) of a position and orientation by the orientation sensor” after the offset1203, and supplies it to the orientation sensor.

Assume that the setting unit507performs the setting (setting change1) of changing the exposure time of the image capturing unit1until the detection unit505detects a sync signal corresponding to the frame (N+1) after detecting a sync signal corresponding to the frame N.

At this time, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N+1), the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of the frame (N+2)) of the image capturing unit2” after an offset1202, and supplies it to the image capturing unit2. The offset1202can be obtained by a method similar to that of the offset1201. Also, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N+1), the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame (N+2)) of a position and orientation by the orientation sensor” after an offset1204, and supplies it to the orientation sensor. The offset1204can be obtained by a method similar to that of the offset1203.

Thereafter, no setting change by the setting unit507is performed, and no offset switching occurs. In this fashion, the synchronous control unit901supplies offsets to the generation unit506so that sync signals to be supplied to the image capturing unit2and the orientation sensor are generated based on the offsets corresponding to a setting change of the exposure time of the image capturing unit by the setting unit507.

The above-described processing can be performed to implement a synchronous operation in which the exposure start time of the first line of an image captured by the image capturing unit1, that of the first line of an image captured by the image capturing unit2, and the data obtaining timing of the orientation sensor coincide with each other even when a setting change is performed.

In the case in which the synchronous timing is set at the exposure start time, offsets for generating external sync inputs to the image capturing unit2and the orientation sensor are changed only when the setting change1of the exposure time of the image capturing unit1is performed. This also applies to a case in which the synchronous timing is set not only at the exposure start time of the start line but also at the exposure start time of an arbitrary line. This can be utilized to simplify offset change processing when a setting change is performed, and reduce the processing load and circuit scale of the HMD101.

Next, generation processing of generating sync signals to be supplied to the image capturing unit2and the orientation sensor using, as the reference of the synchronous timing, the exposure end time of the center line of an image captured by a rolling shutter image sensor will be described with reference toFIG.13.

When the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N−1), the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of the frame N) of the image capturing unit2” after an offset1301, and supplies it to the image capturing unit2. The offset1301can be obtained as a result of subtracting the “exposure time of the image capturing unit2” and the “time t62” from a “time till the exposure end time of the center line of an image captured by the image capturing unit1after the detection timing of the sync signal of the image capturing unit1”. When the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N−1), the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame N) of a position and orientation by the orientation sensor” after an offset1303, and supplies it to the orientation sensor. The offset1303can be obtained as a result of subtracting the “processing time tSens_Ready” from a “time till the exposure end time of the center line of an image captured by the image capturing unit1after the detection timing of the sync signal of the image capturing unit1”.

Assume that the setting unit507performs the setting (setting change2) of changing the exposure time of the image capturing unit2until the detection unit505detects a sync signal corresponding to the frame N after detecting a sync signal corresponding to the frame (N−1).

At this time, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame N, the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of the frame (N+1)) of the image capturing unit2” after an offset1302, and supplies it to the image capturing unit2. The offset1302can be obtained by a method similar to that of the offset1301. Further, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame N, the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame (N+1)) of a position and orientation by the orientation sensor” after the offset1303, and supplies it to the orientation sensor.

Assume that the setting unit507performs the setting (setting change1) of changing the exposure time of the image capturing unit1until the detection unit505detects a sync signal corresponding to the frame (N+1) after detecting a sync signal corresponding to the frame N.

At this time, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N+1), the generation unit506generates a “sync signal (external sync input) for controlling the start of exposure (for image capturing of the frame (N+2)) of the image capturing unit2” after the offset1302, and supplies it to the image capturing unit2. Also, when the detection unit505detects that the image capturing unit1has output a sync signal corresponding to the frame (N+1), the generation unit506generates a “sync signal (external sync input) for controlling the start of measurement (for the frame (N+2)) of a position and orientation by the orientation sensor” after the offset1303, and supplies it to the orientation sensor.

After that, no setting change by the setting unit507is performed, and no offset switching occurs. In this manner, the synchronous control unit901supplies offsets to the generation unit506so that sync signals to be supplied to the image capturing unit2and the orientation sensor are generated based on the offsets corresponding to a setting change of the exposure time of the image capturing unit by the setting unit507.

The above-described processing can be performed to implement a synchronous operation in which the exposure end time of the center line of an image captured by the image capturing unit1, that of the center line of an image captured by the image capturing unit2, and the data obtaining timing of the orientation sensor coincide with each other even when a setting change is performed.

In the case in which the synchronous timing is set at the exposure end time, an offset for generating an external sync input to the image capturing unit2is changed only when the setting change2of the exposure time of the image capturing unit2is performed. In addition, in the case in which the synchronous timing is set at the exposure end time, an offset for generating an external sync input to the orientation sensor is not changed. This also applies to a case in which the synchronous timing is set not only at the exposure end time of the center line but also at the exposure end time of an arbitrary line. This can be utilized to simplify offset change processing when a setting change is performed, and reduce the processing load and circuit scale of the HMD101.

Further, in the case in which the synchronous timing is set at the exposure end time, the cycle of the sync output and image output of the image capturing unit2and the cycle of the data output of the orientation sensor become advantageously constant in synchronization with the cycle of the sync output and image output of the image capturing unit1. This also applies to a case in which the synchronous timing is set not only at the exposure end time of the center line but also at the exposure end time of an arbitrary line.

For example, it can be confirmed inFIGS.11and12that when the setting change1of the image capturing unit1and the setting change2of the image capturing unit2are performed, the cycle of the sync output and image output of the image capturing unit2and the cycle of the data output of the orientation sensor vary. If a captured image is not output in a constant cycle, an IC (Integrated Circuit) corresponding to such an output cycle needs to be selected as a subsequent captured image processing unit. Also, a frame memory or the like needs to be added to adjust the image cycle constant. To the contrary, when a captured image is output in a constant cycle, as shown inFIG.13, the choice of subsequent captured image processing units is widened, and no frame memory for cycle adjustment is required.

Control of the synchronous operation of the image capturing unit501, the image capturing unit502, and the orientation sensor503by the HMD101according to the embodiment will be described with reference to the flowchart ofFIG.14. In step S1401, the synchronous control unit901determines whether the setting unit507has performed a setting change as described above.

If it is determined that the setting unit507has performed a setting change as described above, the process advances to step S1402. If the setting unit507has not performed a setting change as described above, the process advances to step S1407.

In step S1402, the synchronous control unit901obtains from the setting unit507the setting of which line of an image captured by which capturing unit to set as the reference of the synchronous timing. In step S1403, the synchronous control unit901obtains from the setting unit507the setting of which time in the exposure time to set as the reference of the synchronous timing.

In step S1404, the synchronous control unit901obtains from the setting unit507the settings (including at least parameters regarding the image capturing unit1necessary to obtain an offset) of the image capturing unit1. As described with reference toFIG.13, when the reference line of the synchronous timing does not change and the reference exposure time is the exposure end time, the obtaining processing in step S1404may be skipped because the exposure time setting of the image capturing unit1does not influence the decision of an offset.

In step S1405, the synchronous control unit901obtains from the setting unit507the settings (including at least parameters regarding the image capturing unit2necessary to obtain an offset) of the image capturing unit2. As described with reference toFIG.12, when the reference line of the synchronous timing does not change and the reference start time is the exposure start time, the obtaining processing in step S1405may be skipped because the exposure time setting of the image capturing unit2does not influence the decision of an offset.

In step S1406, the synchronous control unit901performs the above-described processing based on the pieces of information obtained in steps S1402to S1406, obtaining an offset corresponding to the image capturing unit2and an offset corresponding to the orientation sensor.

In step S1407, it is determined whether the detection unit505has detected a sync signal (sync output) from the image capturing unit501. If it is determined that the detection unit505has detected a sync signal (sync output) from the image capturing unit501, the process advances to step S1408. If it is determined that the detection unit505has not detected a sync signal (sync output) from the image capturing unit501, the process returns to step S1401.

In step S1408, if the generation unit506receives from the detection unit505a notification that the sync signal (sync output) has been received from the image capturing unit501, it generates a “sync signal to be supplied to the image capturing unit502” after the offset corresponding to the image capturing unit2from the detection timing of the sync signal, and supplies the generated sync signal to the image capturing unit502. Further, the generation unit506generates a “sync signal to be supplied to the orientation sensor503” after the offset corresponding to the orientation sensor503from the detection timing of the sync signal, and supplies the generated sync signal to the orientation sensor503.

As described above, according to the second embodiment, the synchronous operation of the image capturing unit501, the image capturing unit502, and the orientation sensor503can be implemented. In addition, a setting change of the exposure time of the image capturing unit501or502or a setting change of the exposure timing can be coped with. Since the image capturing and data obtaining timings of the respective devices coincide with an arbitrary timing in the exposure time, a more real MR experience free from misalignment between a captured image and an image of the virtual space can be provided.

Third Embodiment

The functional units in the HMD101and the image processing apparatus104shown inFIG.5or9may be implemented by hardware, or some functional units may be implemented by software (computer program).

In the latter case, the image capturing unit501, the image capturing unit502, the orientation sensor503, the display unit504, and the I/F508in the HMD101may be implemented by hardware, and the remaining functional units may be implemented by software. In this case, the software is stored in the memory of the HMD101and executed by the processor of the HMD101to implement the functions of corresponding functional units.

The hardware arrangement of an HMD101will be exemplified with reference to the block diagram ofFIG.15A. A processor1510executes various processes using computer programs and data stored in a RAM1520. By these processes, the processor1510controls the operation of the whole HMD101, and executes or controls each process that is performed by the HMD101in the above description.

The RAM1520has an area for storing computer programs and data loaded from a nonvolatile memory1530, and an area for storing data received from an image processing apparatus104via an I/F1570. Further, the RAM1520has a work area used when the processor1510executes various processes. The RAM1520can properly provide various areas.

The nonvolatile memory1530stores computer programs and data for causing the processor1510to execute or control the operation of the HMD101. The computer programs stored in the nonvolatile memory1530include computer programs for causing a CPU1501to execute the functions of the functional units (except image capturing units501and502, an orientation sensor503, a display unit504, and an I/F508) of the HMD101shown inFIG.5or9. The computer programs and data stored in the nonvolatile memory1530are properly loaded to the RAM1520under the control of the processor1510, and processed by the processor1510.

An image capturing unit1540includes the above-described image capturing units501and502. An orientation sensor1550includes the above-described orientation sensor503. A display unit1560includes the above-described display unit504. The I/F1570includes the above-described I/F508. All the processor1510, the RAM1520, the nonvolatile memory1530, the image capturing unit1540, the orientation sensor1550, the display unit1560, and the I/F1570are connected to a bus1580. Note that the arrangement shown inFIG.15Ais an example of the arrangement applicable to the HMD101and can be properly changed/deformed.

As for the image processing apparatus104, any computer apparatus capable of executing software corresponding to functional units except an1F509and a content DB512is applicable to the image processing apparatus104. The hardware arrangement of the computer apparatus applicable to the image processing apparatus104will be exemplified with reference to the block diagram ofFIG.15B.

The CPU1501executes various processes using computer programs and data stored in a RAM1502and a ROM1503. The CPU1501controls the operation of the whole computer apparatus, and executes or controls each process that is performed by the computer apparatus-applied image processing apparatus104in the above description.

The RAM1502has an area for storing computer programs and data loaded from the ROM1503and an external storage device1506, and an area for storing data received from the HMD101via the I/F1507. In addition, the RAM1502has a work area used when the CPU1501executes various processes. The RAM1502can properly provide various areas. The ROM1503stores setting data, startup programs, and the like of the computer apparatus.

An operation unit1504is a user interface including a keyboard, a mouse, a touch panel, and the like. By operating the operation unit1504, the user can input various instructions to the CPU1501.

A display unit1505is formed from a liquid crystal screen, a touch panel screen, or the like, and can display the result of processing of the CPU1501using an image or a text. Note that the display unit1505may be a projection apparatus such as a projector that projects an image or a text.

The external storage device1506is amass information storage device such as a hard disk drive. The external storage device1506stores an OS (Operating System). The external storage device1506stores computer programs and data for causing the CPU1501to execute the functions of the functional units (except an I/F509and the content DB512) of the image processing apparatus104shown inFIG.5or9. The external storage device1506also includes the above-described content DB512.

The computer programs and data stored in the external storage device1506are properly loaded to the RAM1502under the control of the CPU1501, and processed by the CPU1501.

An/F1507is a communication interface for performing data communication with the HMD101and functions as the above-described I/F509. That is, the computer apparatus performs data communication with the HMD101via the I/F1507.

All the CPU1501, the RAM1502, the ROM1503, the operation unit1504, the display unit1505, the external storage device1506, and the I/F1507are connected to a bus1508. Note that the arrangement shown inFIG.15Bis an example of the arrangement applicable to the image processing apparatus104and can be properly changed/deformed.

Fourth Embodiment

In each of the above-described embodiments, a marker artificially arranged in the physical space is used to obtain the position and orientation of an image capturing unit. However, in addition to or instead of the marker, a natural feature (for example, the corner of furniture such as a chair or desk, or the corner of a building, car, or the like forming a landscape) originally present in the physical space may be used to obtain the position and orientation of an image capturing unit.

The arrangement of the MR system shown inFIG.5or9is merely an example. For example, a plurality of apparatuses may share and execute processes that are performed by the HMD101in the above description, or share and execute processes that are performed by the image processing apparatus104in the above description.

Instead of ahead mounted display device, a “portable device including one or more image capturing units, an orientation sensor, and a display device” such as a smartphone may be used. Also, such a portable device may be added to the MR system together with the head mounted display device. In this case, an image processing apparatus104generates an image of the mixed reality space corresponding to the position and orientation of the head mounted display device, and distributes it to the head mounted display device. Further, the image processing apparatus104generates an image of the mixed reality space corresponding to the position and orientation of the portable device, and distributes it to the portable device. A method of generating an image of the mixed reality space is the same as those in the above-described embodiments.

For example, a smartphone has an application that superimposes and displays AR (Augmented Reality) information on video based on a feature amount (for example, natural feature or QR Code®) detected from an image captured by the image capturing unit. In some cases, orientation information of the smartphone itself detected by the orientation sensor is reflected in the AR display form. In such a case, the smartphone serves as a synchronous control apparatus to synchronize another device in accordance with the exposure time of the image capturing unit, as in the above-described embodiments. This can implement high-precision synchronous processing.

The HMD101and the image processing apparatus104may be integrated. Instead of the head mounted display device, the portable device and the image processing apparatus104may be integrated.

In the above-described embodiments, the HMD101includes the orientation sensor503. However, the present invention is not limited to this, and necessary information may be obtained from an image captured by an objective camera installed near the wearer of the HMD101.

Numerical values, arithmetic methods, processing execution timings, and the like used in the above-described embodiments are merely examples for concrete descriptions, and it is not intended to limit the embodiments to these examples.

Some or all of the above-described embodiments may be combined and used. Some or all of the above-described embodiments may be selectively used.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2020-009447 filed Jan. 23, 2020, which is hereby incorporated by reference herein in its entirety.