Patent Description:
The present disclosure relates to a technology that predicts at least either a position or a posture of a head-mounted display in reference to an image captured of the periphery of the head-mounted display (HMD).

In recent years, it has been customary for video game users to wear an HMD on their head and play video games by operating a game controller while viewing game images displayed on the HMD. When the HMD carries out its tracking process to link the video images of a game space displayed on the HMD to the movement of the user's head, the video game users are allowed to experience an increased sense of immersion in the video images that makes the video games more entertaining.

An example of the related art is disclosed in <CIT>.

<CIT> teaches, for use with a mobile device, positional awareness techniques employing visual-inertial sensory data gathering and analysis hardware with reference to implementations of sensors such as cameras with the mobile device, enabling guidance of the latter.

<CIT> discloses a system for tracking a first electronic device, such as a handheld smartphone, in a virtual reality environment generated by a second electronic device, such as a head mounted display, wherein detection by a camera in one of these devices of at least one visual marker included in the respective other device is performed. Features detected within the field of view corresponding to known features of the visual markers are used to locate and track relative movement of the devices, translating the movement of the second device into an interaction in a virtual experience generated by the second electronic device.

Problems with the prior art are solved by the subject-matter of the independent claims.

There has been known in the art "simultaneous localization and mapping" (SLAM) as a technology for predicting the position and/or posture of an HMD. SLAM that simultaneously performs self-position prediction and environmental map generation is capable of predicting the position and/or posture of an HMD by tracking feature points with use of images captured by a camera mounted on the HMD.

If a television (TV) set that is displaying moving images is present near an HMD and is included as a large part in an image captured by a camera on the HMD, then the tracking process performed by the HMD may fail to extract effective feature points, tending to cause a tracking lost error. In particular, a game system in which an HMD displays game images while at the same time a TV set displays the same game images is likely to suffer a reduction in tracking accuracy when a camera mounted on the HMD captures the game images displayed by the TV set.

It is desirable to provide a technology for highly accurately predicting at least either the position or the posture of an HMD in an environment in which a display different from the HMD is present.

There is provided according to one mode of the present disclosure an information processing apparatus according to claim <NUM>.

There is also provided according to another mode of the present disclosure an image generating method according to claim <NUM>.

<FIG> illustrates in perspective a configurational example of an information processing system <NUM> according to an embodiment. As illustrated in <FIG>, the information processing system <NUM> includes an information processing apparatus <NUM>, a recording apparatus <NUM>, an HMD <NUM>, an input device <NUM> operable by the user with his/her hands and fingers, and an output apparatus <NUM> for outputting images and sounds. The output apparatus <NUM> is a display apparatus different from the HMD <NUM>, and may be a stationary TV set or a projector for projecting video images onto a screen, a wall, or the like. The information processing apparatus <NUM> is electrically connected to an external network <NUM> such as the Internet via an access point (AP) <NUM>. The AP <NUM> has the functions of a wireless access point and a router. The information processing apparatus <NUM> may be electrically connected to the AP <NUM> by a cable or according to a known wireless protocol.

The recording apparatus <NUM> records therein applications such as system software and game software. The information processing apparatus <NUM> may download to the recording apparatus <NUM> game software from a content server via the external network <NUM>. The information processing apparatus <NUM> executes a game program and supplies image data and sound data of the game to the HMD <NUM>. The information processing apparatus <NUM> and the HMD <NUM> may be electrically connected to each other according to a known wireless protocol or by a cable.

The HMD <NUM> is a display apparatus that, when worn on the user's head, displays images on respective display panels positioned in front of the user's eyes. The HMD <NUM> separately displays a left-eye image on the display panel for the left eye and a right-eye image on the display panel for the right eye. These images constitute parallax images as viewed from the left and right viewpoints, realizing a stereoscopic image. Since the user sees the display panels through optical lenses, the information processing apparatus <NUM> supplies the HMD <NUM> with parallax image data where the optical distortions caused by the lenses have been corrected.

Though the user who is wearing the HMD <NUM> does not need the output apparatus <NUM>, the output apparatus <NUM> allows other users to view images displayed on the output apparatus <NUM>. The information processing apparatus <NUM> may display images that are the same as those images viewed by the user wearing the HMD <NUM> on the output apparatus <NUM> or may display images that are different from those images viewed by the user wearing the HMD <NUM> on the output apparatus <NUM>.

The information processing apparatus <NUM> and the input device <NUM> may be electrically connected to each other according to a known wireless protocol or by a cable. The input device <NUM> includes a plurality of operating members such as operating buttons. The user operates the operating members with his/her hands and fingers while gripping the input device <NUM>. When the information processing apparatus <NUM> executes a game program, the input device <NUM> is used as a game controller.

The HMD <NUM> incorporates a plurality of image capturing devices <NUM> that are mounted in respective different positions on a front surface thereof. The image capturing devices <NUM> may have visible-light sensors for use in general digital video cameras, such as charge-coupled devices (CCDs) or complementary-metal-oxide-semiconductor (CMOS) sensors. The image capturing devices <NUM> capture images of a front side of the user in predetermined periodic cycles of <NUM> frames/second, for example, at synchronized timings, and transmit the captured images to the information processing apparatus <NUM>.

The information processing apparatus <NUM> has a function to predict at least either the position or the posture of the HMD <NUM> in reference to an image captured of the periphery of the HMD <NUM>. Though the information processing apparatus <NUM> will be described hereinbelow as having a function to predict both the position and the posture of the HMD <NUM>, it suffices to have a function to predict at least either the position or the posture of the HMD <NUM>.

The information processing apparatus <NUM> predicts, using images captured respectively at a time (t-<NUM>) and a time (t) that are consecutive image capturing times, the amount of movement of the HMD <NUM> between the time (t-<NUM>) and the time (t), and predicts the position and the posture of the HMD <NUM> at the time (t) by using the position and the posture of the HMD <NUM> at the time (t-<NUM>) that precedes the time (t) and the amount of movement. The information processing apparatus <NUM> may derive the position of the HMD <NUM> as positional information in a coordinate system defined in actual space and may derive the posture of the HMD <NUM> as posture information in a coordinate system defined in actual space. The information processing apparatus <NUM> may also derive positional information and posture information of the HMD <NUM> highly accurately by further using sensor data acquired between the time (t-<NUM>) and the time (t) by a posture sensor incorporated in the HMD <NUM>.

<FIG> illustrates in perspective an example of an appearance and a shape of the HMD <NUM>. As illustrated in <FIG>, the HMD <NUM> includes an output mechanism <NUM> and a mount mechanism <NUM>. The mount mechanism <NUM> includes a mounting band <NUM> that, when worn by the user, encircles the user's head to secure the HMD <NUM> to the user's head. The mounting band <NUM> is made of a material or has a structure capable of adjusting its length to match the circumference of the user's head.

The output mechanism <NUM> includes a housing <NUM> shaped to cover the left and right eyes of the user who is wearing the HMD <NUM>. The housing <NUM> houses therein display panels that confront the respective eyes when the user wears the HMD <NUM>. The display panels may be liquid crystal panels, organic electroluminescence (EL) panels, or the like. The housing <NUM> also houses therein a pair of left and right optical lenses to be positioned between the display panels and the user's eyes for enlarging viewing angles of the user. Furthermore, the HMD <NUM> may include speakers and earphones located at positions corresponding to the user's ears, and may be arranged to connect to external headphones.

The image capturing devices <NUM> include a plurality of image capturing devices 14a, 14b, 14c, and 14d disposed on a front outer surface of the housing <NUM>. In terms of the forward direction of the housing <NUM>, the image capturing device 14a is mounted on the upper right corner of the front outer surface of the housing <NUM> such that the optical axis of its sensor is oriented obliquely upwardly to the right, and the image capturing device 14b is mounted on the upper left corner of the front outer surface of the housing <NUM> such that the optical axis of its sensor is oriented obliquely upwardly to the left. The image capturing device 14c is mounted on the lower right corner of the front outer surface of the housing <NUM> such that the optical axis of its sensor is oriented in the forward direction, and the image capturing device 14d is mounted on the lower left corner of the front outer surface of the housing <NUM> such that the optical axis of its sensor is oriented in the forward direction. The image capturing device 14c and the image capturing device 14d constitute a stereo camera.

The HMD <NUM> transmits images captured by the image capturing devices <NUM> and sensor data acquired by the posture sensor to the information processing apparatus <NUM>, and receives game image data and game sound data generated by the information processing apparatus <NUM>.

<FIG> illustrates the HMD <NUM> in functional block form. As illustrated in <FIG>, a controller <NUM> is a main processor for processing and outputting various kinds of data such as image data, sound data, and sensor data, and commands. A storage unit <NUM> temporarily stores data and commands processed by the controller <NUM>. A posture sensor <NUM> acquires sensor data related to the movement of the HMD <NUM>. The posture sensor <NUM>, which may be an inertial measurement unit (IMU), includes at least a triaxial acceleration sensor and a triaxial gyrosensor, and detects triaxial component values (sensor data) in predetermined periodic cycles (<NUM>, for example).

A communication controller <NUM> transmits data output from the controller <NUM>, to the information processing apparatus <NUM> that is external to the HMD <NUM>, by way of wired or wireless communication via a network adapter or an antenna. The communication controller <NUM> also receives data from the information processing apparatus <NUM> and outputs the received data to the controller <NUM>.

When the controller <NUM> receives game image data and game sound data from the information processing apparatus <NUM>, the controller <NUM> supplies the game image data to display panels <NUM> to enable the display panels <NUM> to display game images, and also supplies the game sound data to a sound output unit <NUM> to enable the sound output unit <NUM> to output sounds. The display panels <NUM> include a display panel 130a for the left eye and a display panel 130b for the right eye, and display a pair of parallax images respectively on these display panels. The controller <NUM> transmits the sensor data acquired by the posture sensor <NUM>, sound data acquired by a microphone <NUM>, and the captured images acquired by the image capturing devices <NUM>, from the communication controller <NUM> to the information processing apparatus <NUM>.

<FIG> illustrates the information processing apparatus <NUM> in functional block form. As illustrated in <FIG>, the information processing apparatus <NUM> includes a processor <NUM> and a communication unit <NUM>. The processor <NUM> has an acquiring section <NUM>, a setting section <NUM>, a prediction processing section <NUM>, a game executing section <NUM>, a pattern controlling section <NUM>, and an image generating section <NUM>. The acquiring section <NUM> has a captured image acquiring section <NUM>, a sensor data acquiring section <NUM>, and an operating information acquiring section <NUM>. The image generating section <NUM> has an HMD image generating section <NUM> for generating display images to be displayed on the HMD <NUM> and a TV image generating section <NUM> for generating display images to be displayed on the output apparatus <NUM>.

The communication unit <NUM> receives operating information transmitted from the input device <NUM> and supplies the received operating information to the acquiring section <NUM>. The communication unit <NUM> also receives captured images and sensor data transmitted from the HMD <NUM> and supplies the received captured images and sensor data to the acquiring section <NUM>.

The information processing apparatus <NUM> includes a computer, and the various functions illustrated in <FIG> are performed when the computer runs programs. The computer includes a memory for loading the programs, one or more processors for executing the loaded programs, an auxiliary storage device, and other large scale integration (LSI) circuits as hardware. The processor or processors include a plurality of electronic circuits including semiconductor integrated circuits and LSI circuits. The electronic circuits may be incorporated in one or more chips. It will be understood by those skilled in the art that the functions illustrated in <FIG> are implemented by cooperation between hardware and software and hence by either hardware only, software only, or a combination of hardware and software.

The captured image acquiring section <NUM> acquires an image captured of the periphery of the HMD <NUM> and supplies the acquired image to the prediction processing section <NUM>. The prediction processing section <NUM> carries out a process of predicting the position and the posture of the HMD <NUM> in reference to the captured image, and supplies positional information and posture information representing the predicted result of position and posture to the game executing section <NUM>. The sensor data acquiring section <NUM> acquires sensor data detected by the posture sensor <NUM> of the HMD <NUM> and supplies the acquired sensor data to the prediction processing section <NUM>. It is preferable that the prediction processing section <NUM> increase the accuracy with which to predict the positional information and posture information of the HMD <NUM>, by using the supplied sensor data.

Prior to starting to play a game, the user wearing the HMD <NUM> performs an initial setting process for capturing an image of the user's periphery with the image capturing devices <NUM> and registering the captured image. In the initial setting process, the user defines a zone where the user plays the game (a zone where the user can move), in order to secure his/her own safety while playing the game. If the user is about to leave the playing zone while playing the game, the information processing apparatus <NUM> warns the user of the imminent risk of going out of the playing zone. During the game being played, the image of the peripheral environment that has been registered in the initial setting process may periodically be updated to generate the latest environmental map.

The prediction processing section <NUM> acquires a chronological sequence of images captured by the image capturing devices <NUM> of the HMD <NUM> and detects feature points by dividing each of the images into a grid. The prediction processing section <NUM> associates feature points with each other between an image captured at a time (t-<NUM>) (hereinafter referred to as the "first time") and an image captured at a time (t) (hereinafter referred to as the "second time"), and predicts amounts of movement of the feature points between the images captured at the different times. A process of predicting an amount of movement according to the present embodiment will be described below.

The prediction processing section <NUM> derives the three-dimensional coordinates of a plurality of feature points from two images captured respectively by the image capturing device 14c and the image capturing device 14d at the first time. According to the present embodiment, the image capturing device 14c and the image capturing device 14d constitute a stereo camera, as described above, and the prediction processing section <NUM> derives the three-dimensional coordinates of a plurality of (N) feature points by performing stereographic three-dimensional measurements. The derived three-dimensional coordinates are stored in the memory (not illustrated).

The prediction processing section <NUM> derives the two-dimensional coordinates of a plurality of feature points from a single image captured by the image capturing device 14c at the second time. Alternatively, the prediction processing section <NUM> may derive the two-dimensional coordinates of a plurality of feature points from a single image captured by the image capturing device 14d, rather than the image capturing device 14c.

There is known a method of solving the Perspective n-Point (PnP) problem as a process for predicting the position and the posture of an image capturing device from an association between N (N is an integer of <NUM> or larger) three-dimensional points and two-dimensional coordinates of the N as projected onto the plane of an image captured by the image capturing device. The prediction processing section <NUM> predicts an amount Tv of movement of the image capturing device 14c (that is, an amount of movement in six degrees of freedom (<NUM> DoF) of the HMD <NUM>) from the three-dimensional coordinates of the N feature points derived at the first time and the two-dimensional coordinates of the feature points derived at the second time. Specifically, the prediction processing section <NUM> predicts the amount Tv of movement of the image capturing device 14c according to the following equation (<NUM>).

Here, (u, v) represents the two-dimensional coordinates of the feature points derived from the image captured at the second time, and (X, Y, Z) represents the three-dimensional coordinates of the feature points derived from the image captured by the stereo camera at the first time. The prediction processing section <NUM> reads the three-dimensional coordinates of the feature points from the memory (not illustrated) and acquires (X, Y, Z).

Further, (fx, fy) represents the focal length of the image capturing device 14c, and (cx, cy) represents the principal point of the image, both being internal parameters of the image capturing device 14c. The matrix having the elements r<NUM> through r<NUM> and t<NUM> through t<NUM> represents a rotation/translation matrix. In the equation (<NUM>), (u, v), (fx, fy), (cx, cy), and (X, Y, Z) are known. The prediction processing section <NUM> determines a rotation/translation matrix common to the N feature points by solving the equation with respect to the N feature points.

According to the present embodiment, the prediction processing section <NUM> solves a minimization problem where the elements of a rotation/translation matrix are parameters, by using an objective function (a cost function) indicative of an error. If the Gaussian Newton's method is used, then by finding the partial derivative of an object function with parameters, it can be found how the parameters should be adjusted to reduce the cost. The resultant partial derivative is referred to as Jacobian J (Jacobian matrix). According to the Gaussian Newton's method, Jacobian J is calculated in the course of optimization of the parameters. The prediction processing section <NUM> recursively updates the parameters in order to reduce the objective function, and decides on the parameters at the time a predetermined finishing condition is reached. For example, the prediction processing section <NUM> may determine that the finishing condition is satisfied and may decide on the parameters when the number of recurrences has reached a predetermined count or Jacobian J has become small enough. The prediction processing section <NUM> decides on the elements of the rotation/translation matrix, thereby predicting the amount Tv of movement of the HMD <NUM> between the first time and the second time. At this time, the prediction processing section <NUM> derives a covariance matrix Σv representing the degree of reliability (the degree of confidence) of the predicted amount Tv of movement. The covariance matrix Σv can be determined using Jacobian J according to the following equation (<NUM>).

The covariance matrix Σv corresponds to the degree of reliability of the predicted amount Tv of movement such that the higher the degree of reliability is, the smaller the covariance matrix Σv is, and the lower the degree of reliability is, the larger the covariance matrix Σv is. Specifically, if the feature points in the image captured at the first time and the feature points in the image captured at the second time match each other, then the covariance matrix Σv is derived as small, and if they do not match each other, then the covariance matrix Σv is derived as large.

The prediction processing section <NUM> predicts the amount Tv of movement of the HMD <NUM> between the first time and the second time, and predicts the position and the posture of the HMD <NUM> at the second time by adding the predicted amount Tv of movement to the position and the posture of the HMD <NUM> at the first time. The positional information and the posture information of the HMD <NUM> are supplied to the game executing section <NUM>, in which they are used as inputs to the game.

The operating information acquiring section <NUM> acquires operating information transmitted from the input device <NUM> and supplies the acquired operating information to the game executing section <NUM>. The game executing section <NUM> executes a game program in reference to the operating information from the input device <NUM> and the positional information and the posture information of the HMD <NUM>, and performs a calculating process for moving game characters in a virtual space. The image generating section <NUM> includes a graphics processing unit (GPU) for performing a rendering process, etc., and generates game images.

In the image generating section <NUM>, the HMD image generating section <NUM> generates display images to be displayed on the display panels <NUM> of the HMD <NUM>, whereas the TV image generating section <NUM> generates display images to be displayed on the output apparatus <NUM>. Though not illustrated, the information processing apparatus <NUM> includes a sound generating section for generating game sounds.

<FIG> illustrates an example of a display image displayed on the display panels <NUM>. The HMD image generating section <NUM> generates game images and supplies the generated game images to the display panels <NUM> of the HMD <NUM>. As described above, the display panels <NUM> include the display panel 130a for the left eye and the display panel 130b for the right eye. The HMD image generating section <NUM> generates a game image for the left eye and a game image for the right eye, and supplies the generated game images respectively to the display panel 130a for the left eye and the display panel 130b for the right eye. The display panel 130a for the left eye and the display panel 130b for the right eye display the respective supplied game images. According to the present embodiment, the TV image generating section <NUM> generates the same game images as the game images generated by the HMD image generating section <NUM> and supplies the generated game images to the output apparatus <NUM>.

If the user wearing the HMD <NUM> on the head is playing the game while facing in the direction of the output apparatus <NUM>, then providing the output apparatus <NUM> has a large screen size and the user is close to the output apparatus <NUM>, the output apparatus <NUM> occupies a large zone in the images captured by the image capturing devices <NUM>. At this time, the prediction processing section <NUM> extracts many feature points from the game image displayed on the output apparatus <NUM>. In the event that the prediction processing section <NUM> fails to associate those extracted feature points with the feature points extracted in the past, it is highly possible for the prediction processing section <NUM> to suffer a tracking lost error.

<FIG> illustrates an example of an image captured by the image capturing device 14c, and <FIG> illustrates an example of an image captured by the image capturing device 14d. Since the image capturing device 14c and the image capturing device 14d constitute a stereo camera, the illustrated captured images slightly deviate from each other horizontally. In the examples illustrated in <FIG>, the game image displayed on the output apparatus <NUM> covers a large proportion of the captured images.

In fact, the ratio of the game image displayed on the output apparatus <NUM> to the captured images illustrated in <FIG> is very large. As the game image changes every moment and the prediction processing section <NUM> finds it difficult to extract effective feature points from the captured images, the prediction processing section <NUM> may fail in its process of predicting the position and the posture of the HMD <NUM>.

In the information processing apparatus <NUM> according to the present embodiment, the TV image generating section <NUM> has a function to generate a display image that includes a pattern representing a still image in at least a portion thereof. When the TV image generating section <NUM> generates a display image including a pattern, the prediction processing section <NUM> is able to extract feature points effective for tracking.

<FIG> illustrates an example of an image displayed on the output apparatus <NUM>. The TV image generating section <NUM> generates a display image that includes a pattern <NUM> representing a still image in a portion thereof. In the illustrated example, specifically, the TV image generating section <NUM> generates a display image including the pattern <NUM> representing a rectangular frame and a moving game image <NUM> within the frame, and displays the generated display image on the output apparatus <NUM>. Since the pattern <NUM> representing a rectangular frame is displayed as a still image on the output apparatus <NUM>, the prediction processing section <NUM> is capable of extracting feature points effective for tracking from the pattern <NUM> included in the image captured of the output apparatus <NUM>.

The TV image generating section <NUM> may generate and display on the output apparatus <NUM> a moving game image <NUM> having a reduced image size within the rectangular frame represented by the pattern <NUM>, though it may generate a display image where the rectangular frame represented by the pattern <NUM> is superposed on a moving game image <NUM> that is not scaled down. In order to ensure visibility of the moving game image <NUM>, it is preferable that the image zone occupied by the frame be <NUM>% or less of the screen size of the output apparatus <NUM>. The TV image generating section <NUM> may display a pattern <NUM> as blocks in the respective four corners of the screen of the output apparatus <NUM>, for example, rather than as a rectangular frame. In other words, the TV image generating section <NUM> may generate a display image including a pattern <NUM> as block images in its corners. Alternatively, the TV image generating section <NUM> may generate a display image including a pattern <NUM> on either one or more of upper, lower, left, and right sides thereof, so that the pattern <NUM> will be displayed on either one or more of upper, lower, left, and right sides of the screen of the output apparatus <NUM>. At any rate, the pattern <NUM> is displayed as a still image in order to support the tracking process carried out by the prediction processing section <NUM>.

Inasmuch as the prediction processing section <NUM> extracts corners included in the captured images as feature points, the pattern <NUM> should preferably be of a design containing many corners. In order to increase the accuracy of tracking, the pattern <NUM> should preferably be free of a repetition of identical forms in close positions.

The setting section <NUM> sets ON or OFF of a pattern display function of the TV image generating section <NUM>. The user may set ON or OFF of the pattern display function of the TV image generating section <NUM> in the initial setting process.

<FIG> illustrates an example of a screen for setting ON or OFF of the pattern display function. The screen for setting ON or OFF of the pattern display function is displayed on the display panels <NUM> in the initial setting process. When the user sets "TRACKING ASSISTANCE DISPLAY" to ON, the setting section <NUM> makes the pattern display function active. When the pattern display function is active, if a predetermined condition is fulfilled, the TV image generating section <NUM> generates a display image including the pattern <NUM>. When the user sets "TRACKING ASSISTANCE DISPLAY" to OFF, the setting section <NUM> makes the pattern display function inactive.

When the pattern display function is inactive, the prediction processing section <NUM> may fail in its process of predicting the position and the posture of the HMD <NUM>, by performing the tracking process with use of images captured of the game image displayed on the output apparatus <NUM>. If the prediction processing section <NUM> may fail in the predicting process, then the pattern display function of the TV image generating section <NUM> may automatically be made active, so that the pattern display function can be performed.

The accuracy with which the prediction processing section <NUM> predicts the amount Tv of movement of the HMD <NUM> is increased by displaying the pattern <NUM>. However, the displayed pattern <NUM> has a disadvantage on its own in that it reduces the zone where the moving game image <NUM> is displayed, as illustrated in <FIG>. Accordingly, the pattern <NUM> should preferably not be displayed at all times, and should preferably be displayed only in situations where the accuracy with which to predict the amount Tv of movement of the HMD <NUM> tends to be lowered.

As a result of a test conducted on the tracking process in various environments, the present discloser has specified factors in the reduction in the accuracy with which to predict the amount of movement, as follows.

In an environment with few fixed feature points (a textureless environment), the output apparatus <NUM> displays a game image linked with the movement of the HMD <NUM>, and the image capturing devices <NUM> of the HMD <NUM> capture the game image displayed on the output apparatus <NUM> (see <FIG>).

When the user wearing the HMD <NUM> turns his/her head laterally, the images displayed on the display panels <NUM> change depending on the angle through which the user's head turns. Since the same game image is displayed on the output apparatus <NUM>, the feature points of the game image are also moved on the screen of the output apparatus <NUM>. At this time, if the proportion of the game image in the image captured by the image capturing device 14c is large as illustrated in <FIG>, then the prediction processing section <NUM> extracts many feature points from the game image and predicts the amount Tv of movement of the HMD <NUM> from the extracted feature points. However, as the feature points extracted from the game image are moved in the screen in linked relation to the movement of the HMD <NUM>, the prediction processing section <NUM> is unable to accurately predict the amount Tv of movement of the HMD <NUM> according to the moved feature points. In such a case, the pattern <NUM> should preferably be displayed to support the prediction processing section <NUM> in its process of tracking the feature points.

In such a situation, since the image captured at the first time and the image captured at the second time match each other (no contradiction occurs between the captured images as the game image is simply moved laterally on the screen of the output apparatus <NUM>), the prediction processing section <NUM> can predict the amount Tv of movement of the HMD <NUM> without fail, and the covariance matrix Σv representing the degree of reliability of the predicted amount Tv of movement has a small value (that is, the covariance matrix Σv represents a high degree of reliability). According to the present embodiment, under these circumstances, the pattern controlling section <NUM> determines to control the TV image generating section <NUM> to generate a display image including the pattern <NUM>.

As described above, the prediction processing section <NUM> predicts the amount Tv of movement of the HMD <NUM> between the first time and the second time by using the images captured of the periphery of the HMD <NUM> respectively at the first time and the second time, and derives the covariance matrix Σv representing the degree of reliability of the predicted amount Tv of movement. As described above, the higher the degree of reliability of the predicted amount Tv of movement is, the smaller the covariance matrix Σv is, and the lower the degree of reliability of the predicted amount Tv of movement is, the larger the covariance matrix Σv is. The prediction processing section <NUM> provides the predicted amount Tv of movement and the derived covariance matrix Σv to the pattern controlling section <NUM>.

The prediction processing section <NUM> according to the present embodiment predicts an amount Ti of movement of the HMD <NUM> between the first time and the second time by using the sensor data acquired between the first time and the second time by the posture sensor <NUM> of the HMD <NUM>. Specifically, the prediction processing section <NUM> integrates an acceleration to calculate a change in speed, integrates a speed predicted using the change in speed to calculate a change in position, and integrates an angular velocity to calculate a change in posture, thereby predicting an amount Ti of movement in six degrees of freedom of the HMD <NUM>. Though the prediction processing section <NUM> predicts the amount Ti of movement of the HMD <NUM> between the first time and the second time by using the posture and the speed of the HMD <NUM> at the first time, the posture and the moving speed of the HMD <NUM> at the first time may be the posture and the moving speed predicted at the first time by the prediction processing section <NUM>. The prediction processing section <NUM> provides the amount Ti of movement predicted in reference to the sensor data to the pattern controlling section <NUM>.

<FIG> illustrates the pattern controlling section <NUM> in functional block form. As illustrated in <FIG>, the pattern controlling section <NUM> has a receiving section <NUM>, a difference deriving section <NUM>, a Mahalanobis distance calculating section <NUM>, and a determining section <NUM>. The pattern controlling section <NUM> has a function to determine whether or not to control the TV image generating section <NUM> to generate a display image including the pattern <NUM> representing a still image, according to the difference between the amount Ti of movement and the amount Tv of movement and the degree of reliability of the amount Tv of movement.

The receiving section <NUM> receives the amount Tv of movement derived at the second time, the covariance matrix Σv, and the amount Ti of movement from the prediction processing section <NUM>. The difference deriving section <NUM> calculates the difference ΔT between the amount Ti of movement and the amount Tv of movement according to the following equation (<NUM>).

The Mahalanobis distance calculating section <NUM> calculates a Mahalanobis distance D according to the following equation (<NUM>).

The determining section <NUM> determines whether or not the TV image generating section <NUM> is to generate a display image including the pattern <NUM>, according to the difference ΔT between the amount Ti of movement and the amount Tv of movement and the covariance matrix Σv representing the degree of reliability of the amount Tv of movement. Specifically, the determining section <NUM> determines whether or not the TV image generating section <NUM> is to generate a display image including the pattern <NUM>, from the Mahalanobis distance D calculated using the difference ΔT and the covariance matrix Σv.

According to the present embodiment, a large Mahalanobis distance D means a large difference ΔT between the amounts of movement and a small covariance matrix Σv (a high degree of reliability of the amount Tv of movement). A large difference ΔT between the amounts of movement means that the amount Tv of movement predicted from the captured images and the amount Ti of movement predicted from the sensor data are largely different from each other, i.e., the amount Tv of movement predicted from the captured images is not correct. The foregoing is premised on the assumption that the amount Ti of movement predicted from the sensor data is regarded as accurate in a short period of time, e.g., <NUM>/<NUM> second. Consequently, a large Mahalanobis distance D means a high degree of reliability of the amount Tv of movement despite the fact that the amount Tv of movement is not correct, and corresponds to an environment where the factor <NUM> and the factor <NUM> are fulfilled. Hence, the determining section <NUM> determines whether or not the TV image generating section <NUM> is to generate a display image including the pattern <NUM>, from the Mahalanobis distance D, and determines to have the TV image generating section <NUM> generate a display image including the pattern <NUM> if the Mahalanobis distance D satisfies a predetermined condition. The determining section <NUM> may determine to have the TV image generating section <NUM> generate a display image including the pattern <NUM> if the Mahalanobis distance D is equal to or larger than a predetermined threshold value Th.

While the user wearing the HMD <NUM> faces a white ceiling with few feature points (the output apparatus <NUM> is not present on a ceiling), for example, the image capturing devices <NUM> capture images of the white ceiling, and the prediction processing section <NUM> is unable to extract effective feature points from the captured images. Thus, the predicted amount Tv of movement is not accurate, and the covariance matrix Σv representing the degree of reliability of the predicted amount Tv of movement is large. At this time, though the difference ΔT between the amount Ti of movement calculated from the sensor data of the posture sensor <NUM> and the amount Tv of movement calculated from the captured images is large, the Mahalanobis distance D is not so large because the covariance matrix Σv is of a large value. Hence, the determining section <NUM> may determine not to have the TV image generating section <NUM> generate a display image including the pattern <NUM>. In this case, since the output apparatus <NUM> is not included in the images captured by the image capturing devices <NUM>, it is not necessary to include the pattern <NUM> in the image displayed on the output apparatus <NUM>. The determining section <NUM> according to the present embodiment thus does not determine to include the pattern <NUM> in the displayed image simply when the difference ΔT is large, but determines whether or not to include the pattern <NUM> in the displayed image in reference to the magnitude of the Mahalanobis distance D. Consequently, it is possible to include the pattern <NUM> in the displayed image only when it is required for the tracking process.

The determining section <NUM> may determine whether or not to generate a display image including the pattern <NUM>, according to the result of a digital filtering process performed on the Mahalanobis distance D calculated at every time, with use of an infinite-impulse-response (IIR) filter. For example, if the result of the digital filtering process performed on the Mahalanobis distance D with use of the IIR filter is equal to or larger than a threshold value or if the result of the digital filtering process is equal to or larger than a threshold value continuously for a predetermined period of time, then the determining section <NUM> may determine that the captured images are not suitable for deriving the amount Tv of movement and may determine not to include the pattern <NUM> in a display image. If the calculated Mahalanobis distance D exceeds a predetermined value prior to the digital filtering process, then a predetermined value rather than the Mahalanobis distance D may be input to the IIR filter to avoid a situation where the result of the digital filtering process simply becomes equal to or larger than the threshold value.

The present disclosure has been described above according to the embodiment. The above embodiment is merely an illustrative example, and it will be understood by those skilled in the art that various changes and modifications may be made in the components and processing processes of the embodiment and may also fall within the scope of the disclosure. According to the embodiment, the information processing apparatus <NUM> carries out the predicting process. However, the functions of the information processing apparatus <NUM> may in part or in whole be provided in the HMD <NUM>, so that the HMD <NUM> may carry out the predicting process while functioning as an information processing apparatus.

According to the embodiment, the image capturing devices <NUM> are incorporated in the HMD <NUM>. However, insofar as the image capturing devices <NUM> are able to capture images of a space in the periphery of the HMD <NUM>, the image capturing devices <NUM> may be located in positions other than the HMD <NUM>.

Claim 1:
An information processing apparatus (<NUM>) comprising:
at least one processor (<NUM>) including hardware, wherein the at least one processor (<NUM>) is configured to predict
a first amount of movement of a head-mounted display (<NUM>) between a first time and a second time, the amount of movement defined in six degrees of freedom of the head-mounted display (<NUM>), by using, for predicting, images captured of a periphery of the head-mounted display (<NUM>) respectively at the first time and the second time, derive
a covariance matrix representing a degree of reliability of the predicted first amount of movement such that the higher the degree of reliability is, the smaller the covariance matrix, predict
a second amount of movement of the head-mounted display (<NUM>) between the first time and the second time, by using sensor data acquired by a posture sensor (<NUM>) incorporated in the head-mounted (<NUM>) display between the first time and the second time, generate
a display image to be displayed on a display different from the head-mounted display (<NUM>), and determine
whether or not to generate a display image including a pattern (<NUM>) providing a still image, according to a difference between the first amount of movement and the second amount of movement and the degree of reliability of the first amount of movement as represented by the covariance matrix, the pattern enabling to extract feature points effective for tracking.