Patent Description:
PTL <NUM> discloses a game device that acquires a frame image obtained by imaging a space in front of the game device, that estimates, from the position of a light emitting diode (LED) image of a game controller in the frame image, position information and posture information regarding the game controller in a real space, and that reflects the estimated position information and/or posture information on the processing of a game application.

<CIT> discloses a game device having different kinds of marker lights that emit light independently from the marker lights of another kind.

<CIT> teaches a game system having multiple controllers each equipped with LEDs. To identify the controllers, those LEDs of one controller that are selected to be lit differ from those LEDs of another controller that are selected to be lit.

<CIT> is concerned with a head-mounted display that bears LEDs.

A head-mounted display (HMD) is fitted to the head of a user to provide the user with a virtual reality (VR) video space. The user who wears the HMD operates operation buttons in an input device and can thereby produce various inputs to the video space.

In recent years, techniques for tracking the position and posture of a device to reflect information obtained, on a three-dimensional (3D) model in a VR space, have widely been used. The movement of a player character or a game object in a game space is synchronized with changes in the position and posture of a device to be tracked, and thus an intuitive operation by a user is realized. A plurality of markers that are lit are used for the tracking of the device, an image obtained by imaging the plurality of markers is analyzed, and the positions of marker images within the image are thus identified, with the result that the position and posture of the device in a real space are estimated.

In order to estimate the position and posture of the device with a high degree of accuracy, it is necessary to accurately identify the positions of the marker images within the image obtained by imaging the device. When an image other than the markers is included in a captured image, this may adversely affect the accuracy of estimation of the position and posture of the device, and thus such an image is preferably prevented from being included in the captured image.

Hence, an object of the present invention is to prevent a lit portion other than markers in a device including a plurality of markers from being included in a captured image. Note that, while the device may be an input device that includes operation buttons, the device may be a device that is simply a target to be tracked without including operation members.

The above problem is solved by the subject-matter of the independent claim.

In order to solve the problem described above, a device according to an aspect of the present invention includes a case body and a plurality of markers configured to emit light to the outside of the case body. The device includes a control portion for lighting the plurality of markers at predetermined intervals, and the control portion lights an indicator indicating the state of the device, in a period during which the plurality of markers are unlit.

<FIG> depicts an example of the configuration of an information processing system <NUM> in an embodiment. The information processing system <NUM> includes an information processing device <NUM>, a recording device <NUM>, an HMD <NUM>, an input device <NUM> that is operated with fingers of a user, and an output device <NUM> that outputs images and sounds. The output device <NUM> may be a television set. The information processing device <NUM> is connected through an access point (AP) <NUM> to an external network <NUM> such as the Internet. The AP <NUM> has the functions of a wireless access point and a router, and the information processing device <NUM> may be connected to the AP <NUM> with a cable or may be connected thereto with a known wireless communication protocol.

The recording device <NUM> records system software and applications such as game software. The information processing device <NUM> may download the game software to the recording device <NUM> from a content server through the network <NUM>. The information processing device <NUM> executes the game software and supplies image data and sound data of the game to the HMD <NUM>. The information processing device <NUM> and the HMD <NUM> may be connected to each other with a known wireless communication protocol or may be connected to each other with a cable.

The HMD <NUM> is a display device that is fitted to the head by the user, to display images on display panels located in front of the eyes. The HMD <NUM> individually displays an image for the left eye on a display panel for the left eye and an image for the right eye on a display panel for the right eye. These images form parallax images seen from left and right eyepoints to realize a stereoscopic view. In order for the user to look at the display panels through optical lenses, the information processing device <NUM> supplies, to the HMD <NUM>, parallax image data obtained by correcting optical distortion caused by the lenses.

Although the output device <NUM> is not needed for the user who wears the HMD <NUM>, preparing the output device <NUM> enables another user to see the display image of the output device <NUM>. While the information processing device <NUM> may display, on the output device <NUM>, the same image as the image seen by the user who is wearing the HMD <NUM>, another image may be displayed. For example, when the user who is wearing the HMD <NUM> plays a game together with another user, a game image from the eyepoint of a character of the other user may be displayed from the output device <NUM>.

The information processing device <NUM> and the input device <NUM> may be connected to each other with a known wireless communication protocol or may be connected to each other with a cable. The input device <NUM> includes a plurality of operation members such as operation buttons, and the user operates the operation members with the fingers while grasping the input device <NUM>. When the information processing device <NUM> executes a game, the input device <NUM> is used as a game controller. The input device <NUM> is provided with a posture sensor including a three-axis acceleration sensor and a three-axis gyro sensor, and transmits sensor data to the information processing device <NUM> at a predetermined cycle (for example, <NUM>).

The game of the embodiment handles not only the operation information of the operation members in the input device <NUM> but also the operation information of the position, the posture, the movement, and the like of the input device <NUM>, and reflects the operation information on the movement of the player character within a virtual three-dimensional space. For example, the operation information of the operation members may be used as information for moving the player character, and the operation information of the position, the posture, the movement and the like of the input device <NUM> may be used as information for moving an arm of the player character. In a battle scene within the game, the movement of the input device <NUM> is reflected on the movement of the player character holding a weapon, and thus an intuitive operation by the user is realized, with the result that the immersion of the user in the game is enhanced.

In order to track the position and posture of the input device <NUM>, a plurality of markers (light emitting portions) that can be imaged with image sensing devices <NUM> mounted on the HMD <NUM> are provided on the input device <NUM>. The information processing device <NUM> analyzes an image obtained by imaging the input device <NUM>, to estimate the position information and posture information of the input device <NUM> in a real space, and provides the estimated position information and posture information to the game.

On the HMD <NUM>, a plurality of image sensing devices <NUM> are mounted. The plurality of image sensing devices <NUM> are attached to different positions of the front surface of the HMD <NUM> with different postures such that the overall imaging range obtained by adding up the imaging ranges of the image sensing devices <NUM> includes the entire field of view of the user. The image sensing device <NUM> is preferably an image sensor that can acquire the images of the plurality of markers in the input device <NUM>. For example, when the marker emits visible light, the image sensing device <NUM> includes a visible light sensor, such as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor, that is used in a general digital video camera. When the marker emits invisible light, the image sensing device <NUM> includes an invisible light sensor. The plurality of image sensing devices <NUM> image a space in front of the user at a synchronized timing and predetermined intervals (for example, <NUM> frames/second), and transmit image data obtained by imaging the input device <NUM> to the information processing device <NUM>.

The information processing device <NUM> identifies the positions of a plurality of marker images of the input device <NUM> included in a captured image. Note that, while the one input device <NUM> may be imaged with a plurality of image sensing devices <NUM> at the same timing, since the attachment positions and attachment postures of the image sensing devices <NUM> are known, the information processing device <NUM> combines a plurality of captured images to identify the positions of the marker images.

The three-dimensional shape of the input device <NUM> and the position coordinates of the plurality of markers arranged on the surface thereof are known, and thus, the information processing device <NUM> estimates the position coordinates and the posture of the input device <NUM>, based on the distribution of the marker images within the captured image. The position coordinates of the input device <NUM> may be position coordinates in a three-dimensional space with a reference position being an origin, and the reference position may be position coordinates (longitude and latitude) set before the start of the game.

Note that the information processing device <NUM> can also estimate the position coordinates and the posture of the input device <NUM> by using the sensor data detected with the posture sensor in the input device <NUM>. Hence, the information processing device <NUM> of the embodiment uses the result of the estimation based on the captured image obtained with the image sensing devices <NUM> and the result of the estimation based on the sensor data, so as to perform tracking processing on the input device <NUM> with a high degree of accuracy.

<FIG> depicts an example of the external shape of the HMD <NUM>. The HMD <NUM> includes an output mechanism portion <NUM> and a fitting mechanism portion <NUM>. The fitting mechanism portion <NUM> includes a fitting band <NUM> that is put on by the user to be worn around the head so as to fix the HMD <NUM> to the head. The fitting band <NUM> has a material or a structure in which the length thereof can be adjusted according to the head circumference of the user.

The output mechanism portion <NUM> includes a housing <NUM> having a shape that covers the left and right eyes in a state where the user is wearing the HMD <NUM>, and also includes, therewithin, the display panels directly opposite the eyes when the HMD <NUM> is worn. The display panel may be a liquid crystal panel, an organic electroluminescence (EL) panel, or the like. Within the housing <NUM>, a pair of left and right optical lenses that are located between the display panels and the eyes of the user to enlarge the viewing angle of the user are further included. The HMD <NUM> may further include speakers and earphones in positions corresponding to the ears of the user or may be configured such that external headphones are connected thereto.

A plurality of image sensing devices 14a, 14b, 14c, and 14d are provided on an outer surface on the forward side of the housing <NUM>. With reference to the direction of the line of sight of the user, the image sensing device 14a is attached to an upper right corner of the outer surface on the forward side such that a camera optical axis is directed diagonally upward to the right, the image sensing device 14b is attached to an upper left corner of the outer surface on the forward side such that the camera optical axis is directed diagonally upward to the left, the image sensing device 14c is attached to a lower right corner of the outer surface on the forward side such that the camera optical axis is directed diagonally downward to the right, and the image sensing device 14d is attached to a lower left corner of the outer surface on the forward side such that the camera optical axis is directed diagonally downward to the left. A plurality of image sensing devices <NUM> are installed in this way, and thus, the overall imaging range obtained by adding up the imaging ranges of the image sensing devices <NUM> includes the entire field of view of the user. The field of view of the user described above may be the field of view of the user in a virtual three-dimensional space.

The HMD <NUM> transmits, to the information processing device <NUM>, the sensor data detected with the posture sensor and the image data obtained by imaging by the image sensing devices <NUM>, and receives game image data and game sound data generated in the information processing device <NUM>.

<FIG> depicts a functional block of the HMD <NUM>. A control portion <NUM> is a main processor that processes and outputs various types of data such as the image data, the sound data, and the sensor data and commands. A storage portion <NUM> temporarily stores the data, the commands, and the like processed by the control portion <NUM>. The posture sensor <NUM> acquires the sensor data regarding the movement of the HMD <NUM>. The posture sensor <NUM> includes at least a three-axis acceleration sensor and a three-axis gyro sensor.

A communication control portion <NUM> transmits data output from the control portion <NUM>, to the external information processing device <NUM> through a network adapter or an antenna by wired or wireless communication. In addition, the communication control portion <NUM> receives data from the information processing device <NUM>, and outputs it to the control portion <NUM>.

When the control portion <NUM> receives the game image data or the game sound data from the information processing device <NUM>, the control portion <NUM> supplies the game image data to the display panel <NUM> and makes the display panel <NUM> display the image or supplies the game sound data to a sound output portion <NUM> and makes the sound output portion <NUM> output the sound. The display panel <NUM> includes the display panel for the left eye 130a and the display panel for the right eye 130b, and a pair of parallax images are displayed on the display panels. Further, the control portion <NUM> causes the sensor data from the posture sensor <NUM>, the sound data from a microphone <NUM>, and captured image data from the image sensing devices <NUM> to be transmitted from the communication control portion <NUM> to the information processing device <NUM>.

<FIG> depicts the external shape of the input device <NUM>. <FIG> depicts the front shape of the input device <NUM>, and <FIG> depicts the back shape of the input device <NUM>. The input device <NUM> includes a case body <NUM>, a plurality of operation members 22a, 22b, 22c, and 22d (hereinafter referred to as the "operation members <NUM>" when they are not particularly distinguished) operated by the user, and a plurality of markers 30a to 30t (hereinafter referred to as the "marker <NUM>" or "markers <NUM>" when they are not particularly distinguished) for emitting light to the outside of the case body <NUM>. The operation members <NUM> are arranged on the head of the case body <NUM>, and include an analog stick for performing a tilt operation, a press button, a trigger button for inputting a pulled amount, and the like. An indicator <NUM> indicating the state of the input device <NUM> is arranged on the head of the case body <NUM>. The indicator <NUM> may include an LED device that displays the charged state of the battery of the input device <NUM>.

The case body <NUM> includes a grasping portion <NUM> and a curved portion <NUM> that couples a case body head portion and a case body bottom portion, and the user passes fingers ranging from the index finger to the little finger between the grasping portion <NUM> and the curved portion <NUM> so as to grasp the grasping portion <NUM>. In a state where the user grasps the grasping portion <NUM>, the user operates the operation members 22a, 22b, and 22c with the thumb and operates the operation member 22d with the index finger. Although the markers <NUM>, 30i, and 30j are provided on the grasping portion <NUM>, they are arranged in such positions that they are not hidden by the hand even in a state where the user grasps the grasping portion <NUM>. At least one or more markers <NUM> are provided on the grasping portion <NUM>, thus enhancing the accuracy of tracking.

The marker <NUM> is a light emitting portion that emits light to the outside of the case body <NUM>, and includes a resin portion that diffusely emits light from a light source such as an LED element to the outside on the surface of the case body <NUM>. The marker <NUM> is imaged with the image sensing devices <NUM> so as to be used for estimation processing of position information and posture information regarding the input device <NUM>. Preferably, since the image sensing devices <NUM> image the input device <NUM> at predetermined intervals (for example, <NUM> frames/second), the marker <NUM> emits light in synchronization with periodical imaging timing of the image sensing devices <NUM> and is unlit in the non-exposure period of the image sensing devices <NUM> so as to reduce unnecessary power consumption. The image sensing devices <NUM> and the input device <NUM> are operated based on respective clocks, and in the embodiment, synchronous processing on the exposure period of the image sensing devices <NUM> and a period during which the marker <NUM> is lit is performed as follows.

<FIG> depicts an example of a light emission pattern for the synchronous processing that is used to identify timing of imaging by the image sensing devices <NUM>. A length in a horizontal direction represents an imaging interval (<NUM> msec) corresponding to one frame, and lighting control on the marker <NUM> is performed in units of time grids obtained by dividing the imaging interval. In this example, the imaging interval is divided into <NUM> parts, and one time grid is <NUM>µsec. In <FIG>, the time grid that is colored indicates a lighting period at a first brightness, and the time grid that is not colored indicates a lighting period at a second brightness. Note that, the first brightness is different from the second brightness, and the first brightness may be higher than the second brightness. In a case where the marker <NUM> is imaged when light is emitted at the first brightness, a high-brightness marker image is included in the captured image, whereas, in a case where the marker <NUM> is imaged when light is emitted at the second brightness, a low-brightness marker image is included in the captured image. The light emission pattern is determined such that when the marker <NUM> is continuously imaged at the imaging intervals corresponding to one frame so as to obtain six captured images, the order in which light is emitted at the first brightness and light is emitted at the second brightness differs depending on the time grid.

In the synchronous processing, one or more markers <NUM> imaged with the image sensing devices <NUM> are subjected to lighting control using the light emission pattern illustrated in <FIG>. It is assumed that the marker <NUM> is imaged under such lighting control that, after the start of the synchronous processing, the marker <NUM> is lit at the "first brightness" for the first captured image (Frame <NUM>), the marker <NUM> is lit at the "second brightness" for the second captured image (Frame <NUM>), the marker <NUM> is lit at the "first brightness" for the third captured image (Frame <NUM>), the marker <NUM> is lit at the "second brightness" for the fourth captured image (Frame <NUM>), the marker <NUM> is lit at the "second brightness" for the fifth captured image (Frame <NUM>), and the marker <NUM> is lit at the "first brightness" for the sixth captured image (Frame <NUM>). A time grid corresponding to a combination of the first brightness and the second brightness in the six continuous captured images is the time grid of grid number <NUM>. Hence, thereafter, the input device <NUM> periodically lights the marker <NUM> with timing of grid number <NUM>, and can thereby perform such control that the period during which the marker <NUM> is lit is synchronized with the exposure period of the image sensing devices <NUM> to prevent the marker <NUM> from being lit in the non-exposure period of the image sensing devices <NUM>.

<FIG> depicts a functional block of the input device <NUM>. A control portion <NUM> receives operation information that is input to the operation members <NUM>, and also receives sensor data acquired with a posture sensor <NUM>. The posture sensor <NUM> acquires the sensor data regarding the movement of the input device <NUM>, and includes at least a three-axis acceleration sensor and a three-axis gyro sensor. The control portion <NUM> supplies, to a communication control portion <NUM>, the operation information and the sensor data that are received. The communication control portion <NUM> transmits the operation information and the sensor data output from the control portion <NUM>, to the information processing device <NUM> through a network adapter or an antenna by wired or wireless communication. Further, the communication control portion <NUM> acquires the light emission pattern for the synchronous processing and/or an instruction to emit light, from the information processing device <NUM>.

The input device <NUM> includes a light source <NUM> for lighting the indicator <NUM> and light sources <NUM> for lighting the markers <NUM>. Each of the light source <NUM> and the light sources <NUM> may be an LED element. The marker <NUM> includes the resin portion that diffusely emits light to the outside on the surface of the case body <NUM>, and the resin portion of the marker <NUM> lit by the light source <NUM> may be a resin for sealing an LED element. Here, the marker <NUM> and the light source <NUM> may have a form of one LED device.

The indicator <NUM> assumes the role of notifying the user of the charged state of the battery in the input device <NUM>. The light source <NUM> can emit light of a plurality of colors, and the indicator <NUM> may represent the charged state by the lighting color. For example, when the indicator <NUM> is lit in green, this indicates that the charged state is satisfactory, whereas, when the indicator <NUM> is lit in red, this indicates that the remaining amount in the battery is small. The user can recognize the charged state of the battery from the lighting color of the indicator <NUM>.

<FIG> depicts an example of part of an image obtained by imaging the input device <NUM>. As illustrated in the figure, the captured image includes the images of the markers <NUM> that emit light. In the HMD <NUM>, the communication control portion <NUM> transmits the image data obtained by imaging by the image sensing devices <NUM> to the information processing device <NUM>, and the information processing device <NUM> extracts the images of the markers <NUM> from the image data. In the synchronous processing, the information processing device <NUM> can distinguish the case where the marker <NUM> emits light at the first brightness and the case where the marker <NUM> emits light at the second brightness.

While the synchronous processing on the exposure period of the image sensing devices <NUM> and the period during which the marker <NUM> is lit is performed before the start of the game, the synchronous processing may be performed while the game is being played. Since the image sensing devices <NUM> cannot image the image of the marker <NUM> when synchronization is lost, it is necessary to immediately perform the synchronous processing in a case where the synchronization is lost.

In the synchronous processing, the control portion <NUM> makes one or more light sources <NUM> emit light with the light emission pattern for the synchronous processing (see <FIG>) provided from the information processing device <NUM>. As depicted in <FIG>, in the light emission pattern, the period during which the marker <NUM> is lit at the first brightness and the period during which the marker <NUM> is lit at the second brightness are determined in a plurality of frame periods. The information processing device <NUM> identifies a pattern of changes in the brightness value of the marker <NUM> included in a plurality of continuous captured images, so as to identify a time grid number included in the exposure period of the image sensing devices <NUM>. For example, the exposure period may be set to a length approximately twice the time grid.

<FIG> depicts the period during which the marker <NUM> is lit that is set within the exposure period of the image sensing devices <NUM>. When the information processing device <NUM> identifies the time grid number, the information processing device <NUM> generates an instruction to emit light at the timing of the time grid number and transmits it to the input device <NUM>. In the input device <NUM>, based on the instruction to emit light, the control portion <NUM> periodically makes all the markers <NUM> emit light in the time position of grid number <NUM>. After the establishment of the synchronization, the control portion <NUM> turns on the light source for only a length of one time grid (<NUM> microseconds) in one frame period, and turns off the light source in a period other than the one time grid, with the result that it is possible to reduce wasted power consumption.

Although, in the embodiment, the indicator <NUM> uses the lighting color to notify the user of the charged state of the battery, since the indicator <NUM> is not the marker <NUM>, it is undesirable that the image of the indicator <NUM> is included in the captured image. Hence, after the establishment of the synchronization, while lighting a plurality of markers <NUM> at predetermined intervals, the control portion <NUM> lights the indicator <NUM> in a period during which the markers <NUM> are unlit.

<FIG> illustrates a relation between a period during which the markers are lit and a period during which the indicator is lit. As also illustrated in <FIG>, the control portion <NUM> sets, within the exposure period of the image sensing devices <NUM>, the period during which the markers <NUM> are to be lit. On the other hand, the control portion <NUM> controls the light emission of the light source <NUM> such that the period during the indicator <NUM> is lit is prevented from overlapping the period during which the markers <NUM> are lit. Specifically, the control portion <NUM> makes the light source <NUM> emit light in the period during which the markers <NUM> are unlit so as to light the indicator <NUM>. In other words, the control portion <NUM> does not light the indicator <NUM> in the period during which the markers <NUM> are lit.

With reference to the exposure period of the image sensing devices <NUM>, the control portion <NUM> does not light the indicator <NUM> in the exposure period during which the image sensing devices <NUM> image the input device <NUM>. In this way, the image sensing devices <NUM> do not image the indicator <NUM> that is lit. Note that, since the image sensing devices <NUM> and the input device <NUM> are operated based on respective clocks, the accurate timing at which the image sensing devices <NUM> start the exposure remains unclear for the control portion <NUM>. However, since the control portion <NUM> is in possession of the length of the light exposure period, the control portion <NUM> can establish an additional predetermined time period around the period during which the markers <NUM> are lit, and thereby set an indicator lighting disabled period that certainly includes the exposure period. The control portion <NUM> may determine the period during which the indicator <NUM> is to be lit to fall within a period other than the indicator lighting disabled period. The predetermined time period may be determined with reference to the time grids, and the control portion <NUM> may establish an additional time period corresponding to two or more time grids around the period during which the markers <NUM> are lit so as to set the indicator lighting disabled period, and may thereby determine the period during which the indicator <NUM> is to be lit.

When, in the HMD <NUM>, the communication control portion <NUM> transmits the image data obtained by imaging by the image sensing devices <NUM> to the information processing device <NUM>, the information processing device <NUM> extracts the images of the markers <NUM> from the image data. Since the three-dimensional shape of the input device <NUM> and the position coordinates of the markers <NUM> arranged on the surface thereof are known, the information processing device <NUM> solves a Perspective n-Point (PnP) problem from the distribution of the images of the markers <NUM> within the captured image so as to estimate the position and posture of the input device <NUM> with respect to the image sensing devices <NUM>.

The present invention has been described above based on the embodiment. The embodiment described above is illustrative, and a person skilled in the art would understand that various variations of the constituent elements thereof and the combination of processing processes are possible and that the variations are also within the scope of the present invention.

Although, in the embodiment, the arrangement of a plurality of markers <NUM> in the input device <NUM> including the operation members <NUM> is described, the device to be tracked does not necessarily need to include the operation members <NUM>. In addition, although, in the embodiment, the image sensing devices <NUM> are attached to the HMD <NUM>, the image sensing devices <NUM> may be attached to positions other than the HMD <NUM> as long as the image sensing devices <NUM> can image marker images.

Claim 1:
A device, wherein the device is an input device (<NUM>) including an operation member (<NUM>) operated by a user, said input device comprising:
a case body (<NUM>); and
a plurality of markers (<NUM>) configured to emit light to an outside of the case body,
wherein the device includes
an indicator (<NUM>) configured to indicate a state of the device, and
a control portion (<NUM>) configured to light the plurality of markers (<NUM>) at predetermined intervals,
a light source (<NUM>) of the indicator is different from any light source (<NUM>) of the plurality of markers (<NUM>), and
the control portion (<NUM>) lights the indicator (<NUM>) in a period during which the plurality of markers are unlit, and does not light the indicator (<NUM>) in an exposure period during which an image sensing device (<NUM>) images the device (<NUM>).