Image adjustment system, image adjustment device, and image adjustment

An image adjustment system includes a camera, an image adjustment device, an image display device, and a controller. The image display device displays a captured image adjusted by the image adjustment device. The image adjustment device includes an image generator and an image processor. The image generator generates a spherical surface image. The image processor acquires the spherical surface image from the image generator to display the spherical surface image on the image display device on the basis of instruction information output from the controller. The image processor rotates the spherical surface image on the basis of the instruction information. The image processor adjusts a right-eye image or a left-eye image of the captured image displayed on the image display device in accordance with a rotation of the spherical surface image.

BACKGROUND

The present invention relates to an image adjustment system, an image adjustment device, and an image adjustment method.

A head-mounted display has recently attracted attention as an image display device. The head-mounted display, while being mounted on the head of the user, can provide a user with a sense (immersion) of entering a virtual space by displaying an image. As described in Japanese Patent Application Laid-Open No. 2005-56295 (Patent Document 1), the head-mounted display can display an image captured by an external camera through a network.

The head-mounted display displays an image captured by using a stereo camera capable of capturing a right-eye image and a left-eye image as an omnidirectional camera capable of capturing the range of 360 degrees, and thus the user can view the image displayed on the head-mounted display in three dimensions.

However, when the right-eye image and the left-eye image have a large misalignment therebetween, the user may have a symptom similar to motion sickness called VR (Virtual Reality) sickness. VR sickness is likely to occur when there is a large misalignment in the up-down direction between the right-eye image and the left-eye image, a large parallax between the right-eye image and the left-eye image, or a large difference in tilt between the right-eye image and the left-eye image.

SUMMARY

According to a first aspect of the embodiments, there is provided an image adjustment system, including: a camera configured to capture a right-eye image and a left-eye image; an image adjustment device configured to acquire the right-eye image and the left-eye image from the camera as a captured image and adjust the captured image; an image display device configured to display the captured image adjusted by the image adjustment device; and a controller configured to output instruction information to the image adjustment device, wherein the image adjustment device includes: an image generator configured to generate a spherical surface image; and an image processor configured to acquire the spherical surface image from the image generator to display the spherical surface image on the image display device on the basis of the instruction information, rotate the spherical surface image on the basis of the instruction information, and adjust the right-eye image or the left-eye image of the captured image displayed on the image display device in accordance with a rotation of the spherical surface image.

According to a second aspect of the embodiments, there is provided an image adjustment device, including: an image generator configured to generate a spherical surface image; and an image processor configured to acquire the spherical surface image from the image generator on the basis of instruction information acquired from a controller to display the spherical surface image on an image display device, rotate the spherical surface image on the basis of the instruction information, and adjust a right-eye image or a left-eye image of a captured image captured by a camera and displayed on the image display device in accordance with a rotation of the spherical surface image.

According to a third aspect of the embodiments, there is provided a method of adjusting an image, including: capturing a right-eye image and a left-eye image by a camera; acquiring the right-eye image and the left-eye image from the camera as a captured image by an image processor; displaying the captured image by an image display device; acquiring instruction information from a controller by the image processor; acquiring a spherical surface image from an image generator by the image processor on the basis of the instruction information; displaying the spherical surface image by the image display device; rotating the spherical surface image by the image processor on the basis of the instruction information; and adjusting the right-eye image or the left-eye image of the captured image displayed on the image display device by the image processor in accordance with a rotation of the spherical surface image.

DETAILED DESCRIPTION

First Embodiment

An example of the configuration of an image adjustment system according to a first embodiment is described below with reference toFIG.1. An image adjustment system101includes a camera102, a communication unit103, an image display device104, a controller105, an image adjustment device110, and a server106. The image adjustment device110includes an image processing unit (image processor)111, an image generating unit (image generator)112, and a communication unit113.

The camera102is an omnidirectional camera (360-degree camera) capable of capturing the range of 360 degrees and is a stereo camera capable of capturing a right-eye image and a left-eye image. The communication unit103and the communication unit113are connected through a network. The image adjustment device110can acquire a right-eye image IMR1and a left-eye image IML1taken by the camera102in the range of 360 degrees as a captured image IM1through the communication units103,113and the network.

Computer equipment may be used as the image adjustment device110. A CPU may be used as the image processing unit111and the image generating unit112. The image generating unit112and the communication unit113may be provided outside the image adjustment device110.

The server106is connected to the camera102through the network and the communication unit103and connected to the image adjustment device110through the network and the communication unit113. The server106may acquire the captured image IM1from the camera102through the communication unit103and the network, and the image adjustment device110may acquire the captured image IM1from the server106through the network and the communication unit113.

The captured image IM1acquired by the image adjustment device110is input to the image processing unit111. The image adjustment device110performs image processing, such as correction of distortion of the captured image IM1, and outputs the captured image IM1that is image-processed to the image display device104. The image display device104is, for example, a head-mounted display. The controller105is, for example, a glove type controller used for VR or the like.

FIG.2schematically illustrates the image display device104being mounted on the head of a user US and the controller105being attached to the hand of the user US. The zenith is indicated by a symbol ZE inFIG.2. It is desirable that the zenith of the camera102and the zenith of the user US coincide. With the image display device104mounted on the head of the user US, the image adjustment device110displays the right-eye image IMR1in an area corresponding to the right eye of the user US and the left-eye image IML1in an area corresponding to the left eye of the user US, so that the user US can view the captured image IM1as a three-dimensional image.

The server106may correct distortion of the captured image IM1acquired from the camera102, perform image processing, such as adjustment of the horizontal position of the captured image IM1, and output the captured image IM1that is image-processed to the image adjustment device110. By mounting the image display device104on the head, the user US can view the captured image IM1that is image-processed by the image adjustment device110or the server106.

The image display device104generates posture information PN1on the basis of a state of the user US, such as a direction in which the user US faces or a posture of the user US, while being mounted on the head of the user US. The image processing unit111acquires the posture information PN1from the image display device104. That is, the image processing unit111acquires the posture information PN1on the basis of the posture of the image display device104. The image processing unit111displays on the image display device104an image of an area corresponding to the state of the user US, such as a direction in which the user US faces or a posture of the user US, from the captured image IM1on the basis of the posture information PN1.

The controller105generates instruction information NN1on the basis of a state of the user US, such as a movement of a hand or a posture of the user US or a movement of a finger or a posture of the user US, while being attached to the hand of the user US. Hereinafter, the hand or finger is simply abbreviated as the hand. The image processing unit111acquires the instruction information NN1from the controller105. The image processing unit111can change or adjust the captured image IM1displayed on the image display device104on the basis of the instruction information NN1.

The image generating unit112generates a spherical surface image VSS1that is a virtual image made up by a spherical surface, which is CG (Computer Graphics), and stores the spherical surface image VSS1in an internal memory or an external memory.

The image processing unit111acquires the spherical surface image VSS1from the image generating unit112on the basis of the instruction information NN1and displays the spherical surface image VSS1on the image display device104.FIG.2schematically illustrates an image of the user US with the image display device104mounted on the head of the user US when the user US views the spherical surface image VSS1displayed on the image display device104.

When the user US views the spherical surface image VSS1displayed on the image display device104with the image display device104mounted on the head of the user US, the spherical surface image VSS1is arranged around the user US and the image display device104and is set to be displayed within reach of the hand of the user US on the spherical surface image VSS1. The user US feels as if the hand of the user US is in contact with the spherical surface image VSS1by moving the hand to which the controller105is attached to a position corresponding to the spherical surface image VSS1displayed on the image display device104.

The controller105may include an actuator arranged at a part to be in contact with the hand of the user US. The image processing unit111operates the actuator when determining that the hand of the user US has moved to a position corresponding to the spherical surface image VSS1on the basis of the instruction information NN1. When the actuator applies pressure to the hand of the user US, the user US can actually obtain a sense of the hand in contact with the spherical surface image VSS1.

When the spherical surface image VSS1is displayed on the image display device104and the user US moves the hand to which the controller105is attached in an arbitrary direction, the image processing unit111performs image processing on the basis of the instruction information NN1so that the spherical surface image VSS1and the captured image IM1displayed on the image display device104move in accordance with a moving direction, a moving speed, and a destination position of the hand of the user US.

The user US can rotate the spherical surface image VSS1in any direction, at any speed, to any position by moving the hand in any direction, at any speed, to any position. That is, the user US can rotate the spherical surface image VSS1by the movement of the hand. The image processing unit111moves the captured image IM1in accordance with the rotation of the spherical surface image VSS1.

With reference to flowcharts inFIGS.3A to3D, an example of an image adjustment method according to the first embodiment is described. Specifically, an example of a method of adjusting misalignment between the right-eye image IMR1and the left-eye image IML1is described. The image display device104is mounted on the head of the user US, and the controller105is attached to the hand of the user US. The image display device104displays the captured image IM1(right-eye image IMR1and left-eye image IML1).

When the user US views the captured image IM1displayed on the image display device104and feels uncomfortable or experiences VR sickness, the user US operates the controller105, so that the image processing unit111causes the image display device104to display a setting screen in step S101ofFIG.3A.

The setting screen displays adjustment items, such as an item of up-down correction for correcting a misalignment in the up-down direction (vertical direction) between the right-eye image IMR1and the left-eye image IML1, an item of parallax correction for correcting a parallax between the right-eye image IMR1and the left-eye image IML1, and an item of tilt correction for correcting a difference in tilt between the right-eye image IMR1and the left-eye image IML1.

As illustrated inFIG.4, when the right-eye image IMR1and the left-eye image IML1are misaligned in the up-down direction, the user US operates the controller105to select the item of up-down correction displayed on the setting screen. As illustrated inFIG.5, when the parallax between the right-eye image IMR1and the left-eye image IML1is large, the user US operates the controller105to select the item of parallax correction displayed on the setting screen. As illustrated inFIG.6, when the difference in tilt between the right-eye image IMR1and the left-eye image IML1is large, the user US operates the controller105to select the item of tilt correction displayed on the setting screen.

When the user US selects either one of the items of up-down correction, parallax correction, and tilt correction displayed on the setting screen, the controller105outputs the instruction information NN1including the selected item to the image processing unit111in step S102.

When the item of up-down correction is selected in step S101, the controller105outputs the instruction information NN1indicating the selection of the item of up-down correction to the image processing unit111. In step S111, the image processing unit111shifts processing to a processing mode (up-down correction mode) for correcting a misalignment in the up-down direction between the right-eye image IMR1and the left-eye image IML1on the basis of the instruction information NN1output from the controller105.

In step S112, the image processing unit111causes the image display device104to display an item for selecting whether the image to be corrected is the right-eye image IMR1or the left-eye image IML1. When the user US selects the right-eye image IMR1, the controller105outputs the instruction information NN1indicating the selection of the right-eye image IMR1to the image processing unit111. In step S113, the image processing unit111shifts the processing to a processing mode (right-eye up-down correction mode) for correcting the up-down direction of the right-eye image IMR1on the basis of the instruction information NN1output from the controller105.

In step S114, the image processing unit111acquires the spherical surface image VSS1from the image generating unit112and displays the spherical surface image VSS1on the image display device104. Displayed on the image display device104is a blend of the captured image IM1(right-eye image IMR1and left-eye image IML1) and the spherical surface image VSS1.

The user US rotates the spherical surface image VSS1upward or downward, which is in the vertical direction, to reduce the misalignment in the up-down direction between the right-eye image IMR1and the left-eye image IML1. In step S115, the image processing unit111moves the right-eye image IMR1displayed on the image display device104upward or downward in accordance with the rotation of the spherical surface image VSS1. The user US may rotate the spherical surface image VSS1multiple times until no misalignment is in the up-down direction between the right-eye image IMR1and the left-eye image IML1.

In step S112, when the user US selects the left-eye image IML1, the controller105outputs the instruction information NN1indicating the selection of the left-eye image IML1to the image processing unit111. In step S116, the image processing unit111shifts the processing to a processing mode (left-eye up-down correction mode) for correcting the up-down direction of the left-eye image IML1on the basis of the instruction information NN1output from the controller105.

In step S117, the image processing unit111acquires the spherical surface image VSS1from the image generating unit112and displays the spherical surface image VSS1on the image display device104. Displayed on the image display device104is a blend of the captured image IM1(right-eye image IMR1and left-eye image IML1) and the spherical surface image VSS1.

The user US rotates the spherical surface image VSS1upward or downward to reduce the misalignment in the up-down direction between the right-eye image IMR1and the left-eye image IML1. In step S118, the image processing unit111moves the left-eye image IML1displayed on the image display device104upward or downward in accordance with the rotation of the spherical surface image VSS1. The user US may rotate the spherical surface image VSS1multiple times until no misalignment is in the up-down direction between the right-eye image IMR1and the left-eye image IML1.

The image processing unit111may limit the direction of rotation of the spherical surface image VSS1so that the spherical surface image VSS1rotates only in the up-down direction, when shifting the processing to the up-down correction mode in step S111, the right-eye up-down correction mode in step S113, or the left-eye up-down correction mode in step S116. The right-eye image IMR1or the left-eye image IML1may be moved upward or downward by moving the spherical surface image VSS1upward or downward without rotating the spherical surface image VSS1.

When the item of parallax correction is selected in step S101, the controller105outputs the instruction information NN1indicating the selection of the item of parallax correction to the image processing unit111. InFIG.3B, in step S121, the image processing unit111shifts the processing to a processing mode (parallax correction mode) for correcting a parallax between the right-eye image IMR1and the left-eye image IML1on the basis of the instruction information NN1output from the controller105.

In step S122, the image processing unit111causes the image display device104to display an item for selecting whether the image to be corrected is the right-eye image IMR1or the left-eye image IML1. When the user US selects the right-eye image IMR1, the controller105outputs the instruction information NN1indicating the selection of the right-eye image IMR1to the image processing unit111. In step S123, the image processing unit111shifts the processing to a processing mode (right-eye parallax correction mode) for correcting the parallax of the right-eye image IMR1on the basis of the instruction information NN1output from the controller105.

In step S124, the image processing unit111acquires the spherical surface image VSS1from the image generating unit112and displays the spherical surface image VSS1on the image display device104. Displayed on the image display device104is a blend of the captured image IM1(right-eye image IMR1and left-eye image IML1) and the spherical surface image VSS1.

The user US rotates the spherical surface image VSS1rightward or leftward, which is in the horizontal direction, so that the parallax between the right-eye image IMR1and the left-eye image IML1becomes a target parallax. In step S125, the image processing unit111corrects the parallax of the right-eye image IMR1displayed on the image display device104in accordance with the rotation of the spherical surface image VSS1. The user US may rotate the spherical surface image VSS1multiple times until the parallax between the right-eye image IMR1and the left-eye image IML1becomes the target parallax.

When the user US selects the left-eye image IML1in step S122, the controller105outputs the instruction information NN1indicating the selection of the left-eye image IML1to the image processing unit111. In step S126, the image processing unit111shifts the processing to a processing mode (left-eye parallax correction mode) for correcting the parallax of the left-eye image IML1on the basis of the instruction information NN1output from the controller105.

In step S127, the image processing unit111acquires the spherical surface image VSS1from the image generating unit112and displays the spherical surface image VSS1on the image display device104. Displayed on the image display device104is a blend of the captured image IM1(right-eye image IMR1and left-eye image IML1) and the spherical surface image VSS1.

The user US rotates the spherical surface image VSS1rightward or leftward so that the parallax between the right-eye image IMR1and the left-eye image IML1becomes a target parallax. In step S128, the image processing unit111corrects the parallax of the left-eye image IML1displayed on the image display device104in accordance with the rotation of the spherical surface image VSS1. The user US may rotate the spherical surface image VSS1multiple times until the parallax between the right-eye image IMR1and the left-eye image IML1becomes the target parallax.

The image processing unit111may limit the direction of rotation of the spherical surface image VSS1so that the spherical surface image VSS1rotates only in the horizontal direction, when shifting the processing to the parallax correction mode in step S121, the right-eye parallax correction mode in step S123, or the left-eye parallax correction mode in step S126.

When the item of tilt correction is selected in step S101, the controller105outputs the instruction information NN1indicating the selection of the item of tilt correction to the image processing unit111. InFIG.3C, in step S131, the image processing unit111shifts the processing to a processing mode (tilt correction mode) for correcting a difference in tilt between the right-eye image IMR1and the left-eye image IML1on the basis of the instruction information NN1output from the controller105.

In step S132, the image processing unit111causes the image display device104to display an item for selecting whether the image to be corrected is the right-eye image IMR1or the left-eye image IML1. When the user US selects the right-eye image IMR1, the controller105outputs the instruction information NN1indicating the selection of the right-eye image IMR1to the image processing unit111. In step S133, the image processing unit111shifts the processing to a processing mode (right-eye tilt correction mode) for correcting the tilt of the right-eye image IMR1on the basis of the instruction information NN1output from the controller105.

In step S134, the image processing unit111acquires the spherical surface image VSS1from the image generating unit112and displays the spherical surface image VSS1on the image display device104. Displayed on the image display device104is a blend of the captured image IM1(right-eye image IMR1and left-eye image IML1) and the spherical surface image VSS1.

The user US rotates the spherical surface image VSS1in an arbitrary direction so that a difference in tilt between the right-eye image IMR1and the left-eye image IML1becomes smaller. In step S135, the image processing unit111corrects the tilt of the right-eye image IMR1displayed on the image display device104in accordance with the rotation of the spherical surface image VSS1. The user US may rotate the spherical surface image VSS1multiple times until no difference is in tilt between the right-eye image IMR1and the left-eye image IML1.

In step S132, when the user US selects the left-eye image IML1, the controller105outputs the instruction information NN1indicating the selection of the left-eye image IML1to the image processing unit111. In step S136, the image processing unit111shifts the processing to a processing mode (left-eye tilt correction mode) for correcting the tilt of the left-eye image IML1on the basis of the instruction information NN1output from the controller105.

In step S137, the image processing unit111acquires the spherical surface image VSS1from the image generating unit112and displays the spherical surface image VSS1on the image display device104. Displayed on the image display device104is a blend of the captured image IM1(right-eye image IMR1and left-eye image IML1) and the spherical surface image VSS1.

The user US rotates the spherical surface image VSS1in an arbitrary direction so that a difference in tilt between the right-eye image IMR1and the left-eye image IML1becomes smaller. In step S138, the image processing unit111corrects the tilt of the left-eye image IML1displayed on the image display device104in accordance with the rotation of the spherical surface image VSS1. The user US may rotate the spherical surface image VSS1multiple times until no difference is in tilt between the right-eye image IMR1and the left-eye image IML1.

The image processing unit111does not limit the direction of rotation of the spherical surface image VSS1when shifting the processing to the tilt correction mode in step S131, the right-eye tilt correction mode in step S133, or the left-eye tilt correction mode in step S136. The user US thus can rotate the spherical surface image VSS1in any direction.

The image processing unit111can determine a position on the coordinate of the spherical surface image VSS1to which the zenith ZE before rotation of the spherical surface image VSS1by the user US has moved by the rotation of the spherical surface image VSS1by the user US. The image processing unit111calculates the amount of change of the spherical surface image VSS1before and after the rotation of the spherical surface image VSS1by the user US on the basis of the direction of movement and the position of the destination of the zenith ZE on the coordinate of the spherical surface image VSS1.

The amount of change of the spherical surface image VSS1corresponds to the amount of rotation (rotation angle) of the spherical surface image VSS1acquired by combining the amount of rotation about the X axis (rotation angle), the amount of rotation about the Y axis (rotation angle), and the amount of rotation about the Z axis (rotation angle) in the spherical surface image VSS1. The image processing unit111stores the amount of change of the spherical surface image VSS1as a correction value CV1in association with the captured image IM1(right-eye image IMR1or left-eye image IML1) that is image-processed.

That is, the correction value CV1is calculated on the basis of the rotation direction of the spherical surface image VSS1and the moving amount or the moving angle (rotation angle of spherical surface image VSS1) of the zenith ZE. The image processing unit111may store the coordinates of the zenith ZE on the spherical surface image VSS1after the rotation of the spherical surface image VSS1by the user US as the correction value CV1.

The image processing unit111may store the correction value CV1in an internal memory or an external memory. The server106may acquire the correction value CV1in association with the captured image IM1(right-eye image IMR1or left-eye image IML1) that is image-processed from the image adjustment device110through the network and the communication unit113. The server106stores the correction value CV1in an internal memory or an external memory.

When the user US determines that the misalignment between the right-eye image IMR1and the left-eye image IML1has been corrected and operates the controller105, the image processing unit111ends the display of the spherical surface image VSS1and causes the image display device104to display the setting screen in step S141inFIG.3D.

When the user US operates the controller105to select a predetermined item (for example, item of end) displayed on the setting screen, the controller105outputs the instruction information NN1indicating the selection of the item of end to the image processing unit111in step S142. The image processing unit111shifts the processing to a predetermined processing mode corresponding to the selected item in step S143. When the item of end is selected, the image processing unit111shifts the processing to a processing mode (end mode) for ending the image adjustment between the right-eye image IMR1and the left-eye image IML1.

In step S144, the image processing unit111acquires the amount of rotation (rotation angle) before and after the rotation of the spherical surface image VSS1as the amount of change of the spherical surface image VSS1. In step S145, the image processing unit111stores the amount of change of the spherical surface image VSS1as the correction value CV1in association with the captured image IM1(right-eye image IMR1or left-eye image IML1) that is image-processed and ends the processing.

The image adjustment system101, the image adjustment device110, and the image adjustment method according to the first embodiment cause the image display device104to display the spherical surface image VSS1. In the image adjustment system101, the image adjustment device110, and the image adjustment method according to the first embodiment, when a misalignment occurs between the right-eye image IMR1and the left-eye image IML1, the user US operates the controller105and rotates the spherical surface image VSS1, thereby simply and easily adjusting the misalignment between the right-eye image IMR1and the left-eye image IML1displayed on the image display device104.

In the image adjustment system101, the image adjustment device110, and the image adjustment method according to the first embodiment, when the correction value CV1is stored in the image adjustment device110or the server106, the image processing unit111can read the correction value CV1from the image adjustment device110or the server106, adjust the captured image IM1(right-eye image IMR1or left-eye image IML1) captured by the camera102in accordance with the correction value CV1, and display the adjusted image on the image display device104.

Second Embodiment

An example of the configuration of an image adjustment system according to a second embodiment is described with reference toFIG.7. An image adjustment system201includes a camera202, a posture control device220, a communication unit203, an image display device204, a controller205, an image adjustment device210, and a server206. The image adjustment device210includes an image processing unit (image processor)211, an image generating unit (image generator)212, and a communication unit213.

The camera202, the communication unit203, the image display device204, the controller205, the image adjustment device210, and the server206correspond to the camera102, the communication unit103, the image display device104, the controller105, the image adjustment device110, and the server106according to the first embodiment, respectively. The image processing unit211, the image generating unit212, and the communication unit213correspond to the image processing unit111, the image generating unit112, and the communication unit113according to the first embodiment, respectively.

As illustrated inFIG.8, the camera202has a first surface PL1and a second surface PL2that is a surface opposite to the first surface PL1. The camera202includes a right-eye fisheye lens LR1and a left-eye fisheye lens LL1arranged on the first surface PL1, and a right-eye fisheye lens LR2and a left-eye fisheye lens LL2arranged on the second surface PL2. Hereinafter, the right-eye fisheye lens LR1is referred to as a first right-eye lens LR1, the left-eye fisheye lens LL1is referred to as a first left-eye lens LL1, the right-eye fisheye lens LR2is referred to as a second right-eye lens LR2, and the left-eye fisheye lens LL2is referred to as a second left-eye lens LL2.

The camera202includes a first camera2021for capturing the range of 180 degrees on the first surface PL1by using the first right-eye lens LR1and the first left-eye lens LL1, and a second camera2022for capturing the range of 180 degrees on the second surface PL2by using the second right-eye lens LR2and the second left-eye lens LL2. That is, the camera202is an omnidirectional camera (360-degree camera) capable of capturing the range of 360 degrees and is a stereo camera capable of capturing a right-eye image and a left-eye image.

The camera202captures the range of 360 degrees by using the first right-eye lens LR1and the second right-eye lens LR2that are photographic lenses and generates a right-eye image IMR2. The camera202captures the range of 360 degrees by using the first left-eye lens LL1and the second left-eye lens LL2that are photographic lenses and generates a left-eye image IML2.

The posture control device220controls the posture of the camera202. As illustrated inFIG.9, the posture control device220controls a tilt of the camera202in the left-right direction. For example, the posture control device220rotates the camera202about a rotation axis CLa (first rotation axis) passing through the middle position of the first right-eye lens LR1and the first left-eye lens LL1and the middle position of the second right-eye lens LR2and the second left-eye lens LL2and parallel to the optical axes of the lenses LR1, LL1, LR2, and LL2, thereby controlling the tilt of the camera202in the left-right direction. That is, the tilt of the camera202in the left-right direction is a tilt of the camera202about the rotation axis CLa.

As illustrated inFIG.10, the posture control device220controls a tilt of the camera202in the front-rear direction. For example, the posture control device220rotates the camera202about a rotation axis CLb (second rotation axis) passing through the middle position of the first right-eye lens LR1and the second left-eye lens LL2and the middle position of the first left-eye lens LL1and the second right-eye lens LR2and orthogonal to the optical axes of the lenses LR1, LL1, LR2, and LL2, thereby controlling the tilt of the camera202in the front-rear direction. That is, the tilt of the camera202in the front-rear direction is a tilt of the camera202about the rotation axis CLb.

The first rotation axis CLa and the second rotation axis CLb are orthogonal to each other. With the camera202under horizontal posture control, the first rotation axis CLa and the second rotation axis CLb are parallel to the horizontal plane.

As illustrated inFIG.11, the posture control device220includes a three-axis acceleration sensor221, a three-axis magnetic sensor222, a three-axis gyro sensor223, a drive control unit224, and drive units225,226. The three-axis acceleration sensor221is referred to as a first sensor221, the three-axis magnetic sensor222is referred to as a second sensor222, and the three-axis gyro sensor223is referred to as a third sensor223. The drive unit225is referred to as a first drive unit, and the drive unit226is referred to as a second drive unit.

The first sensor221acquires the tilt of the camera202in the left-right direction and the front-rear direction by detecting the direction of gravity. The second sensor222detects the magnitude and direction of the magnetic field (magnetic field), and the third sensor223detects the angular velocity.

The drive control unit224controls at least one of the drive unit225or the drive unit226on the basis of information detected by the first sensor221. When the camera202is moving with acceleration or deceleration, the drive control unit224controls at least one of the drive unit225or the drive unit226on the basis of information detected by the first sensor221and information detected by at least one of the second sensor222or the third sensor223. As illustrated inFIG.9, the drive unit225rotates the camera202in the left-right direction. As illustrated inFIG.10, the drive unit226rotates the camera202in the front-rear direction.

The image adjustment device210can acquire the right-eye image IMR2and the left-eye image IML2captured by the camera202in the range of 360 degrees as a captured image IM2through the communication units203,213and a network.

The server206is connected to the camera202through the network and the communication unit203and connected to the image adjustment device210through the network and the communication unit213. The server206may acquire the captured image IM2from the camera202through the communication unit203and the network, and the image adjustment device210may acquire the captured image IM2from the server206through the network and the communication unit213.FIG.7illustrates that the image adjustment device210or the server206acquires the captured image IM2through the posture control device220, but the image adjustment device210or the server206may acquire the captured image IM2without the posture control device220.

The captured image IM2acquired by the image adjustment device210is input to the image processing unit211. The image adjustment device210performs image processing, such as correction of distortion of the captured image IM2, and outputs the captured image IM2that is image-processed to the image display device204.

With the image display device204mounted on the head of the user US, the image adjustment device210displays the right-eye image IMR2in an area corresponding to the right eye of the user US and the left-eye image IML2in an area corresponding to the left eye of the user US, so that the user US can view the captured image IM2as a three-dimensional image.

The server206may correct distortion of the captured image IM2acquired from the camera202, perform image processing, such as adjusting the horizontal position of the captured image IM2, and output the captured image IM2that is image-processed to the image adjustment device210. By mounting the image display device204on the head, the user US can view the captured image IM2image-processed by the image adjustment device210or the server206.

The image display device204generates posture information PN2on the basis of a state of the user US, such as a direction in which the user US faces or a posture of the user US, while being mounted on the head of the user US. The image processing unit211acquires the posture information PN2from the image display device204. That is, the image processing unit211acquires the posture information PN2on the basis of the posture of the image display device204. The image processing unit211displays on the image display device204an image of an area corresponding to the state of the user US, such as a direction in which the user US faces or a posture of the user US, from the captured image IM2on the basis of the posture information PN2.

The controller205generates instruction information NN2on the basis of a state of the user US, such as a movement of the hand or a posture of the user US, while being attached to the hand of the user US. The image processing unit211acquires the instruction information NN2from the controller205. The image processing unit211can change or adjust the captured image IM2displayed on the image display device204on the basis of the instruction information NN2.

The image generation unit212generates a spherical surface image VSS2that is a virtual image made up by a spherical surface, which is CG, and stores the spherical surface image VSS2in an internal memory or an external memory. The image processing unit211acquires the spherical surface image VSS2from the image generating unit212on the basis of the instruction information NN2and displays the spherical surface image VSS2on the image display device204. The spherical surface image VSS2corresponds to the spherical surface image VSS1according to the first embodiment. The user US feels as if the hand of the user US is in contact with the spherical surface image VSS2by moving the hand to which the controller205is attached to a position corresponding to the spherical surface image VSS2displayed on the image display device204.

The controller205may include an actuator arranged at a part to be in contact with the hand of the user US. The image processing unit211operates the actuator when determining that the hand of the user US has moved to a position corresponding to the spherical surface image VSS2on the basis of the instruction information NN2. When the actuator applies pressure to the hand of the user US, the user US can actually obtain a sense of the hand in contact with the spherical surface image VSS2.

When the spherical surface image VSS2is displayed on the image display device204and the user US moves the hand to which the controller205is attached in an arbitrary direction, the image processing unit211performs image processing on the basis of the instruction information NN2so that the spherical surface image VSS2and the captured image IM2displayed on the image display device204move in accordance with a moving direction, a moving speed, and a destination position of the hand of the user US.

The user US can rotate the spherical surface image VSS2in any direction, at any speed, to any position by moving the hand in any direction, at any speed, to any position. That is, the user US can rotate the spherical surface image VSS2by the movement of the hand. The image processing unit211moves the captured image IM2in accordance with the rotation of the spherical surface image VSS2.

In the posture control device220, the drive control unit224acquires the horizontal plane on the basis of information detected by at least one of the first sensor221, the second sensor222, or the third sensor223and acquires a tilt angle θa (first tilt angle) of the camera202in the left-right direction with respect to the horizontal plane and a tilt angle θb (second tilt angle) of the camera202in the front-rear direction with respect to the horizontal plane.

The drive control unit224controls the drive unit225so that the tilt angle θa is equal to or less than a predetermined angle α (first angle) (θa≤α), thereby adjusting the posture of the camera202in the left-right direction. The drive control unit224controls the drive unit225so that the tilt angle θb becomes equal to or less than a predetermined angle β (second angle) (θb≤β), thereby adjusting the posture of the camera202in the front-rear direction.

As illustrated inFIG.9, when the camera202is viewed from the first surface PL1and inclined to the right, for example, the first right-eye lens LR1is positioned higher than the first left-eye lens LL1on the first surface PL1. In contrast, the second right-eye lens LR2is positioned lower than the second left-eye lens LL2on the second surface PL2.

Thus, the parallax between the right-eye image IMR2and the left-eye image IML2on the first surface PL1is reverse to that on the second surface PL2, and the user US feels uncomfortable with the captured image IM2displayed on the image display device204. In contrast, when the camera202is inclined in the front-rear direction, the parallax between the right-eye image IMR2and the left-eye image IML2on the first surface PL1is not reverse to that on the second surface PL2, so that the user US hardly feels uncomfortable with the captured image IM2displayed on the image display device204.

For the above reason, the angle α is set to be smaller than the angle β (α<β). For example, the angle α is set to 1 degree, and the angle β is set to 5 degrees. Accordingly, the drive control unit224controls at least one of the drive unit225or the drive unit226so that the tilt angle θa in the left-right direction is smaller than the tilt angle θb in the front-rear direction (θa<θb), thereby controlling the posture of the camera202.

After the posture control device220controls the posture of the camera202, the image adjustment system201performs similar processing as in step S101, steps S111to S118, steps S121to S128, steps S131to S138, or steps S141to S144according to the first embodiment.

The image adjustment system201, the image adjustment device210, and the image adjustment method according to the second embodiment cause the image display device204to display the spherical surface image VSS2. In the image adjustment system201, the image adjustment device210, and the image adjustment method according to the second embodiment, when a misalignment occurs between the right-eye image IMR2and the left-eye image IML2, the user US operates the controller205to rotate the spherical surface image VSS2, thereby simply and easily adjusting the misalignment between the right-eye image IMR2and the left-eye image IML2displayed on the image display device204.

In the image adjustment system201, the image adjustment device210, and the image adjustment method according to the second embodiment, when the correction value CV1is stored in the image adjustment device210or the server206, the image processing unit211can read the correction value CV1from the image adjustment device210or the server206, adjust the captured image IM2(right-eye image IMR2or left-eye image IML2) captured by the camera202in accordance with the correction value CV1, and display the adjusted image on the image display device204.

In the image adjustment system201, the image adjustment device210, and the image adjustment method according to the second embodiment, the posture of the camera202is controlled to be horizontal or nearly horizontal by the posture control device220. While the posture of the camera202is controlled, the user US adjusts the misalignment between the right-eye image IMR2and the left-eye image IML2, and thus the above-described misalignment is simply and easily adjusted.

Third Embodiment

An example of the configuration of an image adjustment system according to a third embodiment is described with reference toFIG.7. An image adjustment system301includes a camera302, a posture control device320, a communication unit303, an image display device304, a controller305, an image adjustment device310, and a server306. The image adjustment device310includes an image processing unit311, an image generating unit312, and a communication unit313.

The camera302, the posture control device320, the communication unit303, the image display device304, the controller305, the image adjustment device310, and the server306correspond to the camera202, the posture control device220, the communication unit203, the image display device204, the controller205, the image adjustment device210, and the server206according to the second embodiment, respectively. The image processing unit311, the image generating unit312, and the communication unit313correspond to the image processing unit211, the image generating unit212, and the communication unit213according to the second embodiment, respectively. For clarity of explanation, the same components as those according to the second embodiment are denoted by the same reference numerals.

As illustrated inFIG.8, the camera302includes a first camera3021for capturing the range of 180 degrees on the first surface PL1by using the first right-eye lens LR1and the first left-eye lens LL1, and a second camera3022for capturing the range of 180 degrees on the second surface PL2by using the second right-eye lens LR2and the second left-eye lens LL2. The first and second cameras3021,3022correspond to the first and second cameras2021,2022according to the second embodiment. That is, the camera302is an omnidirectional camera (360-degree camera) capable of capturing the range of 360 degrees and is a stereo camera capable of capturing a right-eye image and a left-eye image.

The camera302captures the range of 360 degrees by using the first right-eye lenses LR1and the second right-eye lenses LR2and generates a right-eye image IMR3. The camera302captures the range of 360 degrees by using the first left-eye lenses LL1and the second left-eye lenses LL2and generates a left-eye image IML3.

The posture control device320controls the posture of the camera302. As illustrated inFIG.9, the posture control device320controls the tilt of the camera302in the left-right direction. For example, the posture control device320rotates the camera302about the rotation axis CLa to control the tilt of the camera302in the left-right direction. That is, the tilt of the camera302in the left-right direction is a tilt of the camera302about the rotation axis CLa. As illustrated inFIG.10, the posture control device320may control the tilt of the camera302in the front-rear direction. For example, the posture control device320rotates the camera302about the rotation axis CLb to control the tilt of the camera302in the front-rear direction. That is, the tilt of the camera302in the front-rear direction is a tilt of the camera302about the rotation axis CLb.

As illustrated inFIG.11, the posture control device320includes the three-axis acceleration sensor221(first sensor), the three-axis magnetic sensor222(second sensor), the three-axis gyro sensor223(third sensor), the drive control unit224, the drive unit225, and the drive unit226. When the camera302is moving with acceleration or deceleration, the drive control unit224controls at least one of the drive unit225or the drive unit226on the basis of information detected by the first sensor221and information detected by at least one of the second sensor222or the third sensor223.

The image adjustment device310can acquire the right-eye image IMR3and the left-eye image IML3captured by the camera302in the range of 360 degrees as the captured image IM3through the communication units303,313and a network.

The server306is connected through the network and the communication unit303to the camera302and connected through the network and the communication unit313to the image adjustment device310. The server306may acquire the captured image IM3from the camera302through the communication unit303and the network, and the image adjustment device310may acquire the captured image IM3from the server306through the network and the communication unit313.FIG.7illustrates that the image adjustment device310or the server306acquires the captured image IM3through the posture control device320, but the image adjustment device310or the server306may acquire the captured image IM3without the posture control device320.

The captured image IM3acquired by the image adjustment device310is input to the image processing unit311. The image adjustment device310performs image processing, such as correction of distortion of the captured image IM3, and outputs the captured image IM3that is image-processed to the image display device304.

With the image display device304mounted on the head of the user US, the image adjustment device310displays the right-eye image IMR3in an area corresponding to the right eye of the user US and the left-eye image IML3in an area corresponding to the left eye of the user US, so that the user US can view the captured image IM3as a three-dimensional image.

The server306may correct distortion of the captured image IM3acquired from the camera302, perform image processing, such as adjustment of the horizontal position of the captured image IM3, and output the captured image IM3that is image-processed to the image adjustment device310. By mounting the image display device304on the head, the user US can view the captured image IM3image-processed by the image adjustment device310or the server306.

The image display device304generates posture information PN3on the basis of a state of the user US, such as a direction in which the user US faces or a posture of the user US, while being mounted on the head of the user US. The image processing unit311acquires the posture information PN3from the image display device304. That is, the image processing unit311acquires the posture information PN3on the basis of the posture of the image display device304. The image processing unit311displays on the image display device304an image of an area corresponding to the state of the user US, such as a direction in which the user US faces or a posture of the user US, from the captured image IM3on the basis of the posture information PN3.

The controller305generates instruction information NN3on the basis of a state of the user US, such as a movement of the hand or a posture of the user US, while being attached to the hand of the user US. The image processing unit311acquires the instruction information NN3from the controller305. The image processing unit311can change or adjust the captured image IM3displayed on the image display device304on the basis of the instruction information NN3.

The image generation unit312generates a spherical surface image VSS3that is a virtual image made up by a spherical surface, which is CG, and stores the spherical surface image VSS3in an internal memory or an external memory. The image processing unit311acquires the spherical surface image VSS3from the image generating unit312on the basis of the instruction information NN3and displays the spherical surface image VSS3on the image display device304. The spherical surface image VSS3corresponds to the spherical surface images VSS1, VSS2according to the first and second embodiments. The user US feels as if the hand of the user US is in contact with the spherical surface image VSS3by moving the hand to which the controller305is attached to a position corresponding to the spherical surface image VSS3displayed on the image display device304.

The controller305may include an actuator arranged at a part to be in contact with the hand of the user US. The image processing unit311operates the actuator when determining that the hand of the user US has moved to a position corresponding to the spherical surface image VSS3on the basis of the instruction information NN3. When the actuator applies pressure to the hand of the user US, the user US can actually obtain a sense of the hand in contact with the spherical surface image VSS3.

When the spherical surface image VSS3is displayed on the image display device304and the user US moves the hand to which the controller305is attached in an arbitrary direction, the image processing unit311performs image processing on the basis of the instruction information NN3so that the spherical surface image VSS3and the captured image IM3displayed on the image display device304move in accordance with a moving direction, a moving speed, and a destination position of the hand of the user US.

The user US can rotate the spherical surface image VSS3in any direction, at any speed, to any position by moving the hand in any direction, at any speed, to any position. That is, the user US can rotate the spherical surface image VSS3by the movement of the hand. The image processing unit311moves the captured image IM3in accordance with the rotation of the spherical surface image VSS3.

With reference to flowcharts inFIGS.12A and12B, an example of an image adjustment method according to the third embodiment is described. Specifically, an example of a method of adjusting misalignment in the up-down direction between the right-eye image IMR3and the left-eye image IML3is described. The image display device304is mounted on the head of the user US, and the controller305is attached to the hand of the user US. The image display device304displays the captured image IM3(right-eye image IMR3and left-eye image IML3).

When the user US corrects a misalignment in the up-down direction between the right-eye image IMR3and the left-eye image IML3displayed on the image display device304, the user US operates the controller305, so that the image processing unit311causes the image display device304to display a setting screen in step S301ofFIG.12A.

The setting screen displays, for example, an item of up-down correction for correcting a misalignment in the up-down direction between the right-eye image IMR3and the left-eye image IML3, as an adjustment item. The image processing unit311may display adjustment items on the setting screen, such as an item of parallax correction for correcting a parallax between the right-eye image IMR3and the left-eye image IML3and an item of tilt correction for correcting a difference in tilt between the right-eye image IMR3and the left-eye image IML3.

As illustrated inFIG.4, when the right-eye image IMR3and the left-eye image IML3are misaligned in the up-down direction, the user US selects the item of up-down correction displayed on the setting screen. In step S302, the controller305outputs the instruction information NN3indicating the selection of the item of up-down correction to the image processing unit311. In step S311, the image processing unit311shifts processing to a processing mode (up-down correction mode) for correcting a misalignment in the up-down direction between the right-eye image IMR3and the left-eye image IML3on the basis of the instruction information NN3output from the controller305.

In step S312, the image processing unit311displays an item for selecting whether the image to be corrected is the right-eye image IMR3or the left-eye image IML3on the image display device304. When the user US selects the right-eye image IMR3, the controller305outputs the instruction information NN3indicating the selection of the right-eye image IMR3to the image processing unit311. In step S313, the image processing unit311shifts the processing to a processing mode (right-eye up-down correction mode) for correcting the up-down direction of the right-eye image IMR3on the basis of the instruction information NN3output from the controller305.

In step S314, the image processing unit111acquires the spherical surface image VSS3from the image generating unit312and displays the spherical surface image VSS3on the image display device204. Displayed on the image display device204is a blend of the captured image IM3(right-eye image IMR3and left-eye image IML3) and the spherical surface image VSS3.

The user US rotates the spherical surface image VSS3upward or downward to reduce the misalignment in the up-down direction between the right-eye image IMR3and the left-eye image IML3. In step S315, the image processing unit311moves the right-eye image IMR3displayed on the image display device304upward or downward in accordance with the rotation of the spherical surface image VSS3.

In step S312, when the user US selects the left-eye image IML3, the controller305outputs the instruction information NN3indicating the selection of the left-eye image IML3to the image processing unit311. In step S316, the image processing unit311shifts the processing to a processing mode (left-eye up-down correction mode) for correcting the up-down direction of the left-eye image IML3on the basis of the instruction information NN3output from the controller305.

In step S317, the image processing unit311acquires the spherical surface image VSS3from the image generating unit312and displays the spherical surface image VSS3on the image display device304. Displayed on the image display device304is a blend of the captured image IM3(right-eye image IMR3and left-eye image IML3) and the spherical surface image VSS3.

The user US rotates the spherical surface image VSS3upward or downward to reduce the misalignment in the up-down direction between the right-eye image IMR3and the left-eye image IML3. In step S318, the image processing unit311moves the left-eye image IML3displayed on the image display device304upward or downward in accordance with the rotation of the spherical surface image VSS3.

The image processing unit311may limit the direction of rotation of the spherical surface image VSS3so that the spherical surface image VSS3rotates only in the up-down direction, when shifting the processing to the up-down correction mode in step S311, the right-eye up-down correction mode in step S313, or the left-eye up-down correction mode in step S316. The right-eye image IMR3or the left-eye image IML3may be moved upward or downward by moving the spherical surface image VSS3upward or downward without rotating the spherical surface image VSS3.

The image processing unit311can determine a position on the coordinate of the spherical surface image VSS3to which the zenith ZE before rotation of the spherical surface image VSS3by the user US has moved by the rotation of the spherical surface image VSS3by the user US. The image processing unit311calculates the amount of change of the spherical surface image VSS3before and after the rotation of the spherical surface image VSS3by the user US on the basis of the direction of movement and the position of the destination of the zenith ZE on the coordinates of the spherical surface image VSS3. The amount of change of the spherical surface image VSS3corresponds to that of the spherical surface image VSS1.

As illustrated inFIG.9, when the camera302is inclined in the left-right direction, the right-eye image IMR3and the left-eye image IML3are misaligned in the up-down direction as illustrated inFIG.4. Accordingly, when the user US rotates the spherical surface image VSS3to adjust the captured image IM3, the misalignment in the up-down direction between the right-eye image IMR3and the left-eye image IML3displayed on the image display device304may not be sufficiently corrected.

InFIG.12B, in step S321, the image processing unit311determines whether the captured image IM3displayed on the image display device304is captured by the first camera3021or the second camera3022on the basis of the posture information PN3or the captured image IM3displayed on the image display device304.

In step S322, the image processing unit311estimates a tilt direction and a tilt angle of the camera302on the basis of the determination result in step S321(first camera3021or second camera3022), and the instruction information NN3or the adjustment direction (upward or downward) and the adjustment amount of the captured image IM3displayed on the image display device304. Assuming that the distance between the right-eye image IMR3and the left-eye image IML3displayed on the image display device304is d and the adjustment amount in the upward or downward direction is h, the tilt angle θa of the camera302is calculated by a relational expression θa=tan−1(h/d)

In step S323, the image processing unit311generates correction information CN for correcting the position of the camera302on the basis of the estimation result in step S322(specifically, the tilt direction and tilt angle of the camera302). Further, the image processing unit311outputs the correction information CN to the posture control device320through the communication units313,303and the network. The image processing unit311may output the correction information CN to the server306through the communication unit313and the network, and the server306may output the correction information CN to the posture control device320through the network and the communication unit303.

In step S324, the posture control device320controls the posture of the camera302on the basis of the correction information CN. In step S325, the image processing unit311confirms the end of the posture control of the camera302by the posture control device320. The posture control device320may generate an end signal ES indicating the end of the posture control of the camera302and output the end signal ES to the image processing unit311through the communication units303,313and the network.

In step S326, the image processing unit311ends the display of the spherical surface image VSS3and performs a correction opposite to that in step S315or step S318. For example, when the right-eye image IMR3is moved upward in step S315, the image processing unit311moves the right-eye image IMR3downward and performs image processing so that the right-eye image IMR3is in a state before the processing of step S315is performed. For example, when the left-eye image IML3is moved downward in step S318, the image processing unit311moves the left-eye image IML3upward and performs image processing so that the left-eye image IML3is in a state before the processing of step S318is performed.

That is, by controlling the posture of the camera302by the posture control device320, the image processing unit311performs the image processing to return the processing to the state before the processing of step S315is performed for the right-eye image IMR3or to the state before the processing of step S318is performed for the left-eye image IML3and ends the processing.

The image adjustment system301, the image adjustment device310, and the image adjustment method according to the third embodiment cause the image display device304to display the spherical surface image VSS3. In the image adjustment system301, the image adjustment device310, and the image adjustment method according to the third embodiment, when a misalignment occurs between the right-eye image IMR3and the left-eye image IML3, the user US operates the controller305to rotate the spherical surface image VSS3, thereby simply and easily adjusting the misalignment between the right-eye image IMR3and the left-eye image IML3displayed on the image display device304.

In the image adjustment system301, the image adjustment device310, and the image adjustment method according to the third embodiment, the tilt direction and the tilt angle of the camera302are estimated on the basis of the adjustment result of the captured image IM3by the image adjustment device310, and the posture control device320controls the posture of the camera302on the basis of the estimation result. The posture of the camera302is controlled to be horizontal or nearly horizontal by the posture control device320, and thus when the user US adjusts a misalignment between the right-eye image IMR3and the left-eye image IML3, the above-described misalignment is simply and easily adjusted.

The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

For example, a virtual image VSS generated by CG may be an ellipsoidal surface (ellipsoid) and may be any closed surface (closed surface) within reach of the hand of the user US. That is, it is only necessary to obtain a sense that the user US comes in contact with the closed surface from the inside, so that the horizontal adjustment of the captured image IM is performed. As the user US rotates the virtual image VSS, the virtual image VSS preferably has a spherical surface or a spherical shape close to a spherical surface, such as an ellipsoid.

The image adjustment system, the image adjustment device, and the image adjustment method according to the present embodiments simply and easily correct the misalignment between the right-eye image and the left-eye image.