Patent ID: 12260014

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Throughout the drawings, the same elements and processes are provided with the same reference signs, and repetitive explanation therefor will be omitted.FIG.1is a schematic diagram illustrating an appearance of a Head Mounted Display (HMD) system1according to the present embodiment.FIG.2is a hardware configuration diagram of the HMD100.

As illustrated inFIG.1, the HMD system1includes an HMD100to be mounted on a user's head, and an input controller400. The input controller400is connected to the HMD100through a near field communication transmitter-receiver to transmit and receive information therebetween. In this connection, wired communication may be carried out to transmit and receive information therebetween.

The input controller400is provided with an operation member including a keyboard and key buttons, and a near field communication transmitter-receiver.

A server200generates a virtual object and transmits the generated virtual object to the HMD100through an external network300. It may be configured that the HMD100itself generates and displays the virtual object.

As illustrated inFIG.2, the HMD100includes a camera111, an in-camera112, and a distance sensor113. In addition, the HMD100includes an acceleration sensor115as an inertial sensor for measuring motion, and a gyro sensor116(the acceleration sensor115and the gyro sensor116correspond to a motion measurement sensor). Furthermore, the HMD100includes a geomagnetic sensor117, a GPS receiver118, a display119, a network communication transmitter-receiver120, a CPU125, a memory128, and a near field communication transmitter-receiver129. The components above are connected to each other via a bus140. The network communication transmitter-receiver120is connected to an antenna123that receives wireless communication waves.

The network communication transmitter-receiver120is a communication interface for establishing communication between the HMD100and the external server200by wireless LAN, wired LAN, or base station communication. During wireless communication, the network communication transmitter-receiver120is connected to the external network300via the antenna123to transmit and receive information. The network communication transmitter-receiver120receives the virtual object generated in the server200via the external network300, in other words, can transmit and receive operation control signals to and from the server200. The network communication transmitter-receiver120is configured by a communication device corresponding to a long-distance communication standard such as Wideband Code Division Multiple Access (W-CDMA) or Global System for Mobile communications (GSM, registered mark).

The near field communication transmitter-receiver129is a communication interface for carrying out near field communication between the input controller400and the HMD100. The near field communication transmitter-receiver129uses, for example, an electronic tag, but is not limited thereto. As the near field communication transmitter-receiver129, a communication device corresponding to Bluetooth (registered mark), Infrared Data Association (IrDA), Zigbee (registered mark), Home Radio Frequency (HomeRF, registered mark), or wireless LAN (IEEE802.11a, IEEE802.11b, IEEE802.11g) may be used.

The HMD100is connected to the input controller400via the near field communication transmitter-receiver129. The HMD100receives operation information which has been accepted by the input controller400through the near field communication line. The CPU125loads a program126stored in the memory128and executes it. When executing the program126, the CPU125reads information data127and uses the data as necessary. The CPU125causes the display119to display an image130and a virtual object131thereon. Therefore, the CPU125, the memory128, the program126, and the information data127are collectively referred to as a display control device1200.

The display119is provided in front of both eyes of the user who is wearing the HMD100. The CPU125causes the display119to display thereon the image130of real space information captured by the camera111, and the virtual object131received from the server200.

The acceleration sensor115is a sensor configured to detect acceleration of the HMD100, which is used to obtain the perpendicular direction based on change in the position of the HMD100and a gravitational acceleration direction of the HMD100.

The gyro sensor116is a sensor configured to detect angular velocity of three-axis rotational direction of the HMD100. The gyro sensor116is used to detect, based on the detected angular velocity, posture of the HMD100, in other words, Euler angles (pitch angle, yaw angle, roll angle) representing the orientation of the local coordinate system with respect to the world coordinate system, or normalized quaternion. By using the acceleration sensor115and the gyro sensor116mounted on the HMD100, it is possible to detect the movement of the head of the user wearing the HMD100.

The geomagnetic sensor117is a sensor configured to detect magnetic force of the Earth, which is used to detect a direction to which the HMD100is directed. By using the three-axis type geomagnetic sensor117which detects not only the geomagnetism of the longitudinal direction and the lateral direction but also that of the vertical direction, it is possible to obtain geomagnetic change to the movement of the head, thereby making it possible to detect the movement of the head. With these sensors, the variation in the user's head movement can be detected in detail while he or she is wearing the HMD100.

The CPU125serves as a controller of the HMD100and executes the program126such as an OS (Operating System) and an application software for operation control stored in the memory128.

The memory128is, for example, a flash memory, and stores the various kinds of the program126used by the CPU125. In addition, the memory128stores the virtual object received from the server200as the information data127.

FIG.3is a functional block diagram of the functions of the CPU125. The CPU125includes a coordinate system calculation unit1251, an outside recognition unit1254, and a display control unit1255. The memory128is partially provided with a virtual object storage unit1281configured to store a virtual object, and a coordinate system information storage unit1282configured to store information of the position and orientation of the coordinate system and the front direction of the inertial coordinate system. Details of the processing executed by each component will be described later.

Here, the front direction of the inertial coordinate system means a reference direction of the inertial coordinate system which remains in the front direction of an average user even when the user temporarily changes the direction of his or her head. When the virtual object that the user wants to frequently check or operate is arranged around an arrangement center direction set in the inertial coordinate system and the arrangement center direction is set in the front direction of the inertial coordinate system, the user can check or operate the frequently used virtual object without rotating his or her head so much.

Furthermore, in the case where the virtual objects that the user wants to frequently check or operate are divided into groups depending on the purpose of use, the arrangement center direction may be set to correspond to each group. For example, an icon group of application software related to the work A may be arranged near the X-axis positive direction of the inertial coordinate system while an icon group of application software related to the work B may be arranged near the Y-axis positive direction. Furthermore, since working target objects are often located immediately near the arrangement center direction, the virtual object may be arranged by avoiding a position immediately near the arrangement center direction.

In the present invention, it is important to detect and control the orientation of each coordinate system. Using the world coordinate system as a reference, the orientation of each coordinate system is expressed by the rotation operation at the time of rotating the world coordinate system so as to be matched with the orientation of each coordinate system with respect to the world coordinate system. Specifically, the rotation operation is expressed by Euler angles (roll angle, pitch angle, yaw angle) or a normalized quaternion.

The coordinate systems used to arrange the virtual object on the HMD100are a conventional world coordinate system, a conventional local coordinate system, and an inertial coordinate system as a new coordinate system for complementing the defects in each of the world coordinate system and the local coordinate system. Hereinafter, the features of each coordinate system will be described with reference toFIG.4toFIG.8.FIG.4explains the local coordinate system.FIG.5explains the inertial coordinate system.FIG.6explains a case where the inertial coordinate system moves within the world coordinate system.

The world coordinate system is a coordinate system having three coordinate axis directions that constitute a three-axis orthogonal coordinate system fixed to the real world, with one point thereof being fixed to the real world as a coordinate origin. Accordingly, even when the HMD100changes its space position in the real world, the arrangement position of the virtual object defined by the world coordinate system does not change. In the present embodiment, the world coordinate system is set based on the posture and a position of the HMD100at the time of initialization of the HMD100. After the initialization, change in the position and the posture of the HMD100in the real world is detected mainly from the measured values of the acceleration sensor115and the gyro sensor116to calculate the position and the orientation of the local coordinate system with respect to the set world coordinate system. In the case where GPS signals can be received, latitude, longitude, and altitude information calculated from the GPS positioning radio waves received by the GPS receiver118may be supplementally used for the calculation of the positions of the local coordinate system and world coordinate system. Furthermore, the direction of the geomagnetism measured by the geomagnetic sensor117may be supplementally used for the calculation of the orientations of the coordinate systems.

Hereinafter, an example of a method of setting the world coordinate system will be described. Since the local coordinate system that is also a three-axis orthogonal coordinate system is used for setting the world coordinate system, in the following, the local coordinate system will be defined in detail, firstly. The origin of the local coordinate system is set to be located near the center of the head which is deeper than the eyeball of the user wearing the HMD100. The origin of the local coordinate system is set near the position that the user feels is his or her own viewpoint position, thereby only requiring rotation of the local coordinate system when the user rotates his or her head but not requiring special processing for eliminating a sense of discomfort. Then, each coordinate axis of the local coordinate system is defined such that the front direction of the user wearing the HMD100is the positive direction of the X-axis, the left-hand direction is the positive direction of the Y-axis, and the upper direction is the positive direction of the Z-axis. Meanwhile, the definition of the local coordinate system is not limited to the example above as long as the origin is located near the center of the HMD100.

The world coordinate system is set at the time of initialization of the HMD100. Firstly, the HMD100is initialized while being in a stationary state. At the time of initialization of the HMD100, the origin of the world coordinate system is made to be matched with the origin of the local coordinate system. When the HMD100is in the stationary state, the acceleration to be detected by the three-axis acceleration sensor is only the gravitational acceleration, from which the perpendicular direction in the local coordinate system can be obtained. Then, the vertically upward direction is defined as the positive direction of the Z-axis of the world coordinate system, and the direction in which the positive direction of the X-axis of the local coordinate system is projected onto the horizontal plane is defined as the positive direction of the X-axis of the world coordinate system. The positive direction of the Y-axis of the world coordinate system is the direction that is the left-hand direction when the positive direction of the X-axis is defined as the front.

In this way, the world coordinate system has been set. On the other hand, since there is no guarantee that the HMD100is held horizontally at the time of its initialization, the orientation of the local coordinate system does not necessarily match with the world coordinate system. However, the world coordinate system is set based on the local coordinate system, and thus the orientation of the local coordinate system based on the world coordinate system has been obtained. The orientation of the local coordinate system in its initial state is expressed as a result of the rotation operation from the world coordinate system in the world coordinate system. In the following, the rotation operation is expressed by a normalized quaternion qLW0although it also can be expressed by Euler angles. The normalized quaternion qLW0is an expression in the world coordinate system.

The normalized quaternion is a quaternion of which a norm is 1, and able to represent rotation about a certain axis. A normalized quaternion q representing rotation of an angle η when a unit vector (nX, nY, nZ) serves as the rotation axis is as follows.

q=cos⁢(η/2)+nX⁢sin⁢(η/2)⁢i+nY⁢sin⁢(η/2)⁢j+nZ⁢sin⁢(η/2)⁢k(1)

In the equation above, each of i, j, k is a unit of a quaternion. The clockwise rotation with respect to the vector (nX, nY, nZ) indicates a rotation direction in which η is positive. Since the rotation of an arbitrary coordinate system is expressed by the normalized quaternion above, the orientation of the local coordinate system and that of the inertial coordinate system are expressed by the normalized quaternions representing the rotation from the world coordinate system. The normalized quaternion representing the orientation of the world coordinate system is 1.

Here, the usage of the symbols is summarized as below. A real number of the normalized quaternion q is represented as Sc(q), and q* is a conjugate quaternion of the normalized quaternion q. An operator that normalizes the norm of the quaternion to 1 is defined by “[ ]”. When q is an arbitrary quaternion, the following equation defines “[ ]”.

[q]=q/(qq⋆)1/2(2)

A denominator on the right side of the equation (2) is a norm of the normalized quaternion q. Next, a quaternion expressing a coordinate point or vector p (pX, pY, pZ) is defined by the following equation.

p=pX⁢i+pY⁢j+pZ⁢k(3)

A projection operator of a vector onto a plane orthogonal to a unit vector n is expressed as P(n). The projection of the vector p is expressed by the following equation.

P⁡(n)⁢p=p+nSc⁡(np)(4)

In the present specification, unless otherwise noted, symbols representing coordinate points and vectors that are not indication of components are assumed to be quaternions.

If a coordinate point or direction vector p1is converted to a coordinate point or direction vector p2by a rotation operation about the origin which is represented as q, p2is calculated by the following equation.

p2=q⁢p1⁢q⋆(5)

Since the normalized quaternion representing the orientation of the coordinate system is obtained as above, it is possible to convert position coordinates of a virtual object between the coordinate systems. A conversion equation of a coordinate point when the coordinate origins differ with each other will be described later. Here, an equation of a normalized quaternion R (n1, n2) for rotation about an axis perpendicular to a plane including n1and n2so as to overlap the unit vector n1on the unit vector n2is described below since it will be used in the later explanation.

R⁡(n1,n2)=[1-n2⁢n1](6)

The orientation qLW0of the local coordinate system at the time of initialization is expressed by the normalized quaternion above. Firstly, at the time of initialization, a gravitational acceleration vector (gX, gY, gZ) is obtained in the local coordinate system, and the quaternion is expressed as gL. Then, a quaternion of a projection vector (0, gY, gZ) of the gravitational acceleration vector, which is projected onto the YZ-plane of the local coordinate system, is defined as hL, and expressed by the following equation using a projection operator.

hL=P⁡(i)⁢gL(7)

The rotation from the local coordinate system to the world coordinate system will be considered by employing a procedure of rotating the X-axis of the local coordinate system in the direction of projection onto the horizontal plane of the world coordinate system in the initial state so as to superimpose it on the horizontal plane, and thereafter, superimposing the Z-axis of the local coordinate system on the Z-axis of the world coordinate system by the rotation about the X-axis of the local coordinate system. When noted that the rotation for superimposing the X-axis of the local coordinate system on the horizontal plane of the world coordinate system is equal to the rotation for superimposing hLon gL, and also noted that qLW0is defined as the rotation from the world coordinate system to the local coordinate system, qLW0is obtained by the following equation.

qL⁢W⁢0=q1*⁢q2*(8)

In the equation above, q1and q2are defined by the following equation.

q1=R⁡([hL],[gL])(9)q2=R⁡(q1⁢kq1*,-q1[hL]⁢q1*)(10)

The world coordinate system is defined in the equations above, meanwhile, the present invention is not limited to the example above as long as the coordinate system is the one on which the position of the external field can be described.

Furthermore, the world coordinate system is set at the initialization in the above, meanwhile, there are cases requiring the world coordinate system that was previously set. In this case, the positional relationship between the world coordinate system set at the time of initialization and the previously set world coordinate system is obtained based on the coordinate values of feature points of the external field, and the latitude, longitude, and altitude information calculated from the GPS positioning radio waves received by the GPS receiver118so as to change the coordinate origins and the orientation of the local coordinate system and the inertial coordinate system based on the previously set world coordinate system.

At the time of initialization, the origin of the inertial coordinate system is set to be the same as that of the local coordinate system, and the orientation of the inertial coordinate system is set to be the same as that of the world coordinate system. The origin and the orientation of the inertial coordinate system are controlled by the method which will be described later.

Next, a method of calculating the change, which occurs due to the motion of the HMD100, in the normalized quaternion qLWrepresenting the orientation of the local coordinate system will be described. When an angular velocity vector detected by the gyro sensor116is (ωX, ωY, ωZ), a quaternion of this angular velocity vector ωLis expressed by the following equation.

ωL=ωX⁢i+ωY⁢j+ωZ⁢k(11)

It should be noted that, in the equation above, the angular velocity vector ωLis an expression of the local coordinate system. An angular velocity vector ωWin the world coordinate system is given by the following equation.

ωW=qL⁢W⁢ωL⁢qLW*(12)

Furthermore, when noted that qLWis an expression of the world coordinate system, a difference equation for determining time development of qLWis as follows.

Δ⁢qL⁢W/Δ⁢t=(1/2)⁢ωW⁢qL⁢W=(1/2)⁢qL⁢W⁢ωL(13)

By applying a measurement interval of the gyro sensor116to Δt, qLWis sequentially updated by the equation (13). At the time of calculation by the equation (13), a technique for increasing approximation accuracy may be used therewith, or correction may be added to keep a norm of qLWat 1. Furthermore, in order to correct accumulation of errors, the measurement result in the geomagnetic direction by the geomagnetic sensor117may be used, or the position information of feature points of the external field detected by the camera111and the distance sensor113may be used.

The acceleration values measured by the acceleration sensor115are used to update a position of the origin of the local coordinate system in the world coordinate system. When an acceleration vector detected by the acceleration sensor115is (aX, aY, aZ), a quaternion of the acceleration vector aLis expressed by the following equation.

aL=aX⁢i+aY⁢j+aZ⁢k(14)

It should be noted that, in the equation above, the acceleration vector aLis an expression of the local coordinate system. An acceleration vector aw in the world coordinate system is given by the following equation.

aW=qL⁢W⁢aL⁢qLW*(15)

It is assumed that position coordinates of the origin of the local coordinate system in the world coordinate system is OLW, a velocity vector of the origin of the local coordinate system is vW, and a gravitational acceleration vector is gW. The gravitational acceleration vector gWwas measured at the time of initialization of the HMD100. A difference equation for determining temporal development of the velocity vector vWis as follows.

Δ⁢vW/Δ⁢t=aW-gW(16)

A difference equation for determining temporal development of the position coordinates OLWis as follows.

Δ⁢OL⁢W/Δ⁢t=vW(17)

By applying a measurement interval of the acceleration sensor115to Δt, vWand OLWare sequentially updated by the equations (16) and (17). At the time of calculation by the equations (16) and (17), a technique for increasing approximation accuracy may be used therewith. Furthermore, latitude, longitude, and altitude information calculated from the GPS positioning radio waves received by the GPS receiver118may be used to update the position of the origin of the local coordinate system in the world coordinate system, or position information of feature points of the external field detected by the camera111and the distance sensor113may be used. Here, in order to simplify the processing, the measurement interval of the acceleration sensor115may be made equal to the measurement interval of the gyro sensor116.

The procedure described above for updating the local coordinate system may be applicable even if the definition of the initial state of the world coordinate system is changed, as long as the world coordinate system is fixed to the external field.

Since the display119for displaying the virtual object is fixed to the local coordinate system, position information relating to the virtual objects arranged in the world coordinate system and the inertial coordinate system is converted for display control so as to be expressed in the local coordinate system. When a coordinate value or vector in the world coordinate system is pW, pWLwhich is an expression of pWin the local coordinate system is calculated by the following equation.

pW⁢L=qLW*(pW-OL⁢W)⁢qL⁢W(18)

Next, it is assumed that position coordinates of the origin of the inertial coordinate system in the world coordinate system are defined as OIW. The origin of the inertial coordinate system is usually matched with the origin of the local coordinate system, meanwhile, it may be different therefrom. When the orientation of the inertial coordinate system is qIWand a coordinate value or vector in the inertial coordinate system is pI, pILwhich is an expression of pIin the local coordinate system is calculated by the following equation.

pI⁢L=qLW*(qI⁢W⁢pI⁢qIW*+OI⁢W-OL⁢W)⁢qL⁢W(19)

Based on the conversion equation described above, conversion of the display position and orientation of the virtual object between the coordinate systems can be calculated.

Hereinafter, how the virtual objects look different depending on the movement of the HMD100and an operation of the coordinate system will be described. The local coordinate system is a coordinate system having three coordinate axis directions constituting the three-axis orthogonal coordinate system fixed to the HMD100with one point thereof being fixed to the HMD100as the coordinate origin when viewed from the HMD100. In accordance with change in the orientation of the head of the user wearing the HMD100, the local coordinate system is also rotated by the same angle as that of the change.

The display119is also fixed to the HMD100and displays a virtual object when the user's line of sight toward the virtual object enters a display range (display surface119b) of the display119. Within the coordinate system, a range which makes the virtual object visible is defined as an effective field of view (FOV)119a. Within the local coordinate system, the FOV119ais a region of the direction range that looks toward the display surface119bof the display as viewed from the user's viewpoint. As illustrated inFIG.4A, a local coordinate system virtual object420is within the FOV119aand thus is displayed on the display119while a local coordinate system virtual object410is not within the FOV119aand thus is not displayed on the display119.

Hereinafter, an example of appearances of virtual objects when the user rotates his or her head and the local coordinate system is rotated accordingly will be described.

FIG.4AandFIG.4Billustrate an example of the appearances of the virtual objects arranged in the local coordinate system. When the user's head is rotated by the yaw angle ψ1from a state illustrated inFIG.4A, the two axes (XL-axis and YL-axis) of the local coordinate system are also rotated by the same angle as the yaw angle ψ1about the ZL-axis as illustrated inFIG.4B. In accordance with the rotation of the local coordinate system, the FOV119a, the local coordinate system virtual object410, and the local coordinate system virtual object420which are fixed to the local coordinate system are also rotated. As a result, the user can continue to view the local coordinate system virtual object420on the same display position, however, cannot view the local coordinate system virtual object410no matter how much the user moves his or her head. In the above, the rotation only by the yaw angle has been described for convenience of explanation. Meanwhile, in the case where the HMD100is rotated by the roll angle φ and the pitch angle θ, the XL-axis, the YL-axis, and the ZL-axis are also rotated by the same angle as the roll angle φ and the pitch angle θ.

The inertial coordinate system is a coordinate system in which the coordinate origin follows the movement of the HMD100(or the user) in the real space while the orientation of the coordinate does not follow the rotation of the HMD100in the real space. When viewed based on the inertial coordinate system, the FOV119ais rotated in accordance with the rotation of the HMD100. The virtual object is arranged in this inertial coordinate system and the virtual object follows the movement of the user, and accordingly, when rotating his or her head, the user can visually recognize all directions of the inertial coordinate system. As a result, it is possible to hold many virtual objects near the user in a state where they can be visually recognized even when the user moves. Here, the meaning of “the coordinate origin follows the HMD100” is that the coordinate origin always stays within a certain distance from the HMD100. In order to simplify the control, the coordinate origin may be set on the same position as that of the local coordinate system, that is, on the center of the HMD100. Furthermore, when the user rotates his or her head in order to visually recognize various directions of the inertial coordinate system, the orientation of the inertial coordinate system may be fixed with respect to the real world, in other words, the world coordinate system in order to make the virtual object appear naturally while the user is rotating his or her head.

FIG.5AandFIG.5Billustrate an inertial coordinate system in which the coordinate origin is fixed to the center of the HMD100in the same manner as the local coordinate system, and the orientation of the coordinate is fixed with respect to the world coordinate system. It is assumed that, in an initial state, an inertial coordinate system virtual object510inFIG.5Ais arranged outside the FOV119aand an inertial coordinate system virtual object520is arranged inside the FOV119a. When the user's head is rotated by the yaw angle ψ1 from the initial state, as illustrated inFIG.5B, the two axes (XL-axis and YL-axis) of the local coordinate system are also rotated by the same angle as the yaw angle ψ1about the ZL-axis. In accordance with the rotation of the local coordinate system above, the FOV119afixed to the local coordinate system is rotated to the FOV119aillustrated inFIG.5B.

On the other hand, since the orientation of the inertial coordinate system is selected to be fixed with respect to the world coordinate system, even when the user's head is rotated by the yaw angle ψ1from the initial state, the inertial coordinate system virtual objects510,520do not change their positions based on the world coordinate system. As a result, the inertial coordinate system virtual object510is also included in the FOV119aafter the rotation, that is, the user can visually recognize the inertial coordinate system virtual object510. In the above, the rotation only by the yaw angle has been described for convenience of explanation. Meanwhile, in the case where the HMD100is rotated by the roll angle φ and the pitch angle θ, the XL-axis, the YL-axis, and the ZL-axis are also rotated by the same angle as the roll angle φ and the pitch angle θ.

In the above, the orientation of the inertial coordinate system is fixed with respect to the world coordinate system. Meanwhile, the orientation of the inertial coordinate system may be changed by a changing operation. Furthermore, the origin of the inertial coordinate system also moves in the world coordinate system in accordance with the movement of the user. As illustrated inFIG.6, it is assumed that the user wearing the HMD100is at (x1, y1) in the world coordinate system and the coordinate origin of the inertial coordinate system in this case is (x1, y1). When the user moves from (x1, y1) to (x2, y2), the coordinate origin of the inertial coordinate system moves in parallel by the same movement amount as the amount of the user's movement above, and changes its position from (x1, y1) to (x2, y2). On the other hand, since the coordinate axis direction is relatively fixed with respect to the world coordinate system, the direction of the XI-YIaxis of the inertial coordinate system of which the coordinate origin is at (x1, y1) is the same as the direction of the XI-YIaxis of the inertial coordinate system of which the coordinate origin is at (x2, y2).

With reference toFIG.7toFIG.9, appearances of real objects and virtual objects arranged in the world coordinate system, the local coordinate system, and the inertial coordinate system will be described. InFIG.7toFIG.9, a rectangular mark represents a real object, and an arrangement position thereof is defined in the world coordinate system. A rhombus object is a world coordinate system virtual object of which an arrangement position is defined in the world coordinate system. A triangle mark represents a local coordinate system virtual object of which an arrangement position is defined in the local coordinate system. A circular mark represents an inertial coordinate system virtual object of which an arrangement position is defined in the inertial coordinate system. A solid line represents that an object is visible, and a dotted line represents that an object is not visible.

FIG.7illustrates change in appearances of objects when the user rotates only his or her head without changing his or her position in the world coordinate system. In an initial state illustrated in (the upper part of)FIG.7, the FOV119aincludes a local coordinate system virtual object711, an inertial coordinate system virtual object731, a world coordinate system virtual object741, and a part of the real object721. When the user wearing the HMD100rotates his or her head from the initial state above, the FOV119ais rotated accordingly. Since the local coordinate system virtual object711is rotated and moves by the same rotation amount as that of the FOV119a, it is also included in the FOV119aafter the rotational movement and can be visually recognized. On the other hand, an inertial coordinate system virtual object732and a world coordinate system virtual object742which were located outside the FOV119abecome visible because they are included in the FOV119aafter the rotation while the inertial coordinate system virtual object731and the world coordinate system virtual object741which were located inside the FOV119abecome invisible.

FIG.8illustrates change in appearances of objects when the user performs a changing operation of the coordinate axis direction of the inertial coordinate system without changing his or her position and the orientation of his or her head in the world coordinate system. As a case requiring the changing operation of the coordinate axis direction of the inertial coordinate system, it is assumed that the user is sitting on a chair and working while facing an L-shaped work table including a front work table and a side work table juxtaposed with a side face of the front work table. In the FOV119aat the time when the user is working on the front work table, it is assumed that the initial state (the upper part) ofFIG.8is visually recognized. Here, when the coordinate axis direction of the inertial coordinate system is rotated, as illustrated in the lower part ofFIG.8, only the inertial coordinate system virtual objects731,732, and733move from left to right ofFIG.8while the positions of the local coordinate system virtual object711, the real object721, and the world coordinate system virtual objects741,743in the real world are unchanged.

FIG.9illustrates change in appearances of objects when the user moves without changing the coordinate axis direction of the inertial coordinate system and the local coordinate system. In this case, it is assumed that the initial state illustrated in (the upper part of)FIG.9is visually recognized. When the user moves (for example, by walking) in the world coordinate system, the real objects721,722,723,724and the world coordinate system virtual objects741,742,743look larger as the user approaches the real objects721,722,723,724. On the other hand, the appearances of each of the local coordinate system virtual objects and the inertial coordinate system virtual objects remains the same.

With reference toFIG.10andFIG.11, processing contents of the HMD system1will be described.FIG.10illustrates a flowchart of the processing in the HMD system1.FIG.11illustrates an example of a data structure storing virtual objects.

When a main power source of the HMD100is turned on, each of the acceleration sensor115, the gyro sensor116, the geomagnetic sensor117, and the GPS receiver118starts measurement (step S01). Each of the camera111, the in-camera112, and the distance sensor113is also started and captures images if necessary for the subsequent processing.

Next, as described above, at the time of initialization of the HMD100, the coordinate system calculation unit1251detects the gravitational direction based on a signal from the acceleration sensor115, and sets the world coordinate system based on the gravitational direction and the X-axis direction of the local coordinate system (step S02). At the time of the initialization, the origin of the inertial coordinate system is made to be matched with the origin of the local coordinate system, and the orientation of the inertial coordinate system is made to be matched with the world coordinate system. The coordinate system calculation unit1251causes the coordinate system information storage unit1282to record the position and orientation of the local coordinate system and inertial coordinate system in the initialization, which are based on the world coordinate system, and the gravitational acceleration vector in the initialization.

After the initialization, a value of the acceleration vector detected by the acceleration sensor115, a value of the angular velocity vector detected by the gyro sensor116, and measured values detected by the various sensors are updated (step S03).

Based on the updated acceleration vector and angular velocity vector, the coordinate system calculation unit1251updates the position and orientation of the local coordinate system which are based on the world coordinate system. Information from other sensors may also be used for this updating processing. The coordinate system information storage unit1282records the updated position and orientation information (step S04).

Based on the updated position and orientation of the local coordinate system and a change control method of the inertial coordinate system which will be described later, the coordinate system calculation unit1251updates the position and orientation of the inertial coordinate system which is based on the world coordinate system. Information from other sensors may also be used for this updating processing. The coordinate system information storage unit1282records the updated position and orientation information (step S05).

The display control unit1255refers to the virtual object storage unit1281and reads out the type of each virtual object and which coordinate system to be used to define the arrangement position of each virtual object. The display control unit1255calculates the arrangement position and orientation of each virtual object in each coordinate system. Furthermore, the display control unit1255converts the arrangement position and orientation of each virtual object in each coordinate system to the local coordinate system of the HMD100(step S06).

The display control unit1255arranges the virtual objects. Arranging the virtual objects within the FOV119aof display119means the same as displaying them on the display119(step S07).

Unless the user turns off the main power source (step S08/No), the processing returns to step S03and signals newly output from the sensors are acquired to execute the subsequent calculation.

Hereinafter, an example of the changing operation of the inertial coordinate system in step S05will be described.

(1) Method of Changing Inertial Coordinate System (User's Instruction)

Basically, the method of changing the direction qIWof the inertial coordinate system is performed in response to a user's instruction. For example, when the camera111recognizes the user's hand, swipe motion in the space causes the inertial coordinate system to be rotated in a swipe direction. At this time, the inertial coordinate system may be rotated about the direction axis that is perpendicular to the swipe direction in a plane perpendicular to the line-of-sight direction and passes through the origin of the inertial coordinate system. The swipe motion is converted to a rotation angular velocity vector ωSW, and the direction is updated by the following equation.

Δ⁢qI⁢W/Δ⁢t=(1/2)⁢ωSQ⁢qIW(20)

In the equation above, Δt represents an update interval of the local coordinate system, and the rotational angular velocity vector ωSWhas a finite value while the swipe motion continues. In the case without the swipe motion, the rotational angular velocity vector ωSWis 0 and the orientation of the inertial coordinate system is fixed with respect to the world coordinate system. The origin of the inertial coordinate system is made to be matched with the origin of the local coordinate system. This method can directly control the orientation of the inertial coordinate system.

Furthermore, the input controller400may be provided with a touch panel screen which allows the user to control the orientation of the inertial coordinate system by swipe motion thereon. In this case, for example, the inertial coordinate system may be rotated about a direction axis that is perpendicular to the swipe direction in the touch panel plane and passes through the origin of the inertial coordinate system.

Still further, a control method adapted to determine the front direction of the inertial coordinate system (for example, X-axis direction), match the front direction with a reference direction on the user side (for example, the direction in which the front direction of the HMD100averagely faces or the front direction of the trunk), and cause the user reference direction not to follow, at least perfectly, with respect to the rotation of the user's head may be combined with the control method described above.

(2) Method of Changing Inertial Coordinate System (Control Based on User Reference Direction)

Firstly, a reference direction on the user side (user reference direction) is determined. The user reference direction may be, for example, a direction in which the front direction of the HMD100faces averagely. The front direction of the HMD100is, for example, a direction from the origin of the local coordinate system toward the center of the display surface119bof the display. The unit direction vector indicating the front direction of the user is represented as uLin the local coordinate system. The unit direction vector of the front direction of the inertial coordinate system is represented as fIin the inertial coordinate system. The unit direction vector fIis, for example, one of the arrangement center directions of a virtual object. The origin of the inertial coordinate system is made to be matched with the origin of the local coordinate system. An average direction <uLW> is determined by smoothing the directions of uLin the world coordinate system. The average direction uLWis given by the following equation.

uL⁢W=qL⁢W⁢uL⁢qLW*(21)

As the averaging method, for example, an exponential moving average is used although the method is not limited thereto.

〈uL⁢W(t+Δ⁢t)〉=[ξ⁢uLW(t)+(1-ξ)⁢〈uL⁢W(t)〉](22)

In the equation above, ξ is an averaging coefficient and takes a value between 0 and 1. Although fIis made to be matched with <uLW>, since high visibility can be obtained when the horizontal direction of the virtual object arranged in the inertial coordinate system is maintained, the direction orthogonal to fIis maintained in the horizontal direction. In this case, the direction qIWof the inertial coordinate system which is based on the world coordinate system is expressed by the following equation.

qI⁢W=q2⁢q1(23)

In the equation above, q1and q2are defined by the following equations.

q1=R⁡([P⁡(k)⁢fI],[P⁡(k)⁢〈uL⁢W〉])(24)q2=R⁡(q1⁢fI⁢q1*,〈uL⁢W〉)(25)

Since the front direction of the inertial coordinate system is matched with the average front direction of the HMD100, even if the user changes the orientation of his or her head, the orientation of the inertial coordinate system does not change immediately. The FOV119ais rotated within the inertial coordinate system, so that the user can visually recognize and operate the virtual object within the inertial coordinate system in a direction range wider than a visible range in the local coordinate system. In other words, this method allows the user to use the space of the inertial coordinate system as an extended area of a displayable area in the local coordinate system.

The desirable following speed at which the front direction of the inertial coordinate system follows the front direction of the HMD100differs depending on use modes. For example, when a period of time during which the head direction is being changed from the average front direction of the HMD100is long, the slower following speed is desirable. On the other hand, for example, in the case where the direction to which the user wants to mainly direct his or her head is changed frequently, the high following speed is desirable. Accordingly, the following speed may be arbitrarily changed by the user's setting. In the averaging method described above, the following speed can be increased as the value of the averaging coefficient ξ of the equation (22) is increased.

Furthermore, the direction of the inertial coordinate system is made to follow the smoothed rotation direction obtained by smoothing the rotation directions of the HMD100which have been calculated based on signals output from the gyro sensor116, which is equivalent to that the smoothing process is applied to the directions of the inertial coordinate system. As a result, even when the user finely moves, it is possible to stably display the inertial coordinate system virtual object.

When there is a plurality of arrangement center directions of a virtual object, the front direction of the inertial coordinate system may be selected or switched from among them in accordance with a user's instruction. In addition, an arbitrary direction of the inertial coordinate system may be defined as the front direction by swipe motion.

The user reference direction may be determined as the front direction of the user's trunk. In order to determine the front direction of the user's trunk, an image of the user may be captured by the in-camera112to detect a user's trunk region based on the captured image by the outside recognition unit1254. Here, the user's reference direction is determined within the horizontal plane of the external field to maintain the visibility of the virtual object. Firstly, the in-camera112detects a direction parallel to a surface of the user's chest in the local coordinate system. A unit vector directed toward the front direction of the user in the normal direction of the detected direction above is expressed as sLin the local coordinate system. A projection direction of sLonto the horizontal plane is defined as the user reference direction. An expression of the reference direction uLWin the world coordinate system is given by the following equation.

uL⁢W=[P⁡(k)⁢qL⁢W⁢sL⁢qLW*](26)

The smoothing process for uLWis performed in order to stabilize the user reference direction, and the calculation procedures after the smoothing process are the same as those in the equations (23) to (25). In this method, since the front direction of the trunk serves as the reference direction, the user can match the front direction of the inertial coordinate system by directing his or her trunk in the direction in which he or she mainly works, thereby making it possible to naturally control the front direction of the inertial coordinate system.

While the user is moving, the traveling direction of the user may be set as the user reference direction (in this case, the smoothing process may be combined by a low-pass filter, etc.). The user reference direction expressed as uLWin the world coordinate system is given by the following equation.

uL⁢W=[P⁡(k)⁢vW](27)

In the equation above, vWis a user's moving speed vector in the world coordinate system. The smoothing process for uLWis performed in order to stabilize the user reference direction, and the calculation procedures after the smoothing process are the same as those in the equations (23) to (25). The smoothed moving direction obtained by smoothing the moving directions (traveling directions) of the HMD100which have been calculated based on the outputs of the acceleration sensor115is calculated to reset the front direction of the inertial coordinate system so as to make it follow the smoothed moving direction.

As described above, according to the control method based on the user reference direction, even when the user temporarily changes the direction of his or her head, the front direction of the inertial coordinate system is maintained in a direction close to the direction to which the user's the head is averagely directed. It is convenient to employ this control method when the inertial coordinate system is used as an extended area of the displayable area in the local coordinate system.

The control method based on the user reference direction may be combined with the change control method based on a user's instruction described in (1) above. When the inertial coordinate system is rotated in accordance with the user's instruction, a direction overlapped with the user reference direction after the rotation is determined as the new front direction of the inertial coordinate system. In addition, when the direction of the body is changed suddenly, the control method based on the user instruction may be performed to immediately adjust the front direction of the inertial coordinate system to the user reference direction.

In the case where the user feels difficulty in operating a virtual object if the inertial coordinate system moves during the operation, it may be configured to allow the user to perform an input operation by using, for example, the input controller400to turn on/off a control mode. It may be also configured to turn off the control mode while the user is performing any operation on the virtual object.

(3) Limitation of Inertial Coordinate System

Limitation in which the Z1axis of the inertial coordinate system is to be matched to the ZWaxis of the world coordinate system, in other words, the vertical direction may be provided. In this case, the swipe control by the user in the method (1) above is effective only for a component in the horizontal direction of the swipe motion.

(4) Processing at Power-Off and Power-on

At the time of power-off, the coordinate system information storage unit1282may store the front direction of the inertial coordinate system. In a mode where the direction of the inertial coordinate system is fixed to the world coordinate system unless the user performs the change operation, the direction to which the front direction of the HMD100is directed is determined as the front direction of the inertial coordinate system at the time of power-off.

As a setting of the inertial coordinate system at the time of power-on, in addition to the setting method of the initialization procedure described above, the coordinate system calculation unit1251may read out the previous front direction information when the power is tuned on so as to match the direction of the inertial coordinate system with the read front direction information in step S02.

(5) Method of Setting Coordinate Origin of Inertial Coordinate System

In the control method described above, the origin of the inertial coordinate system is made to be matched with the origin of the local coordinate system. Meanwhile, it may be configured to set and update the origin of the local coordinate system on a smoothed position obtained by the smoothing process so as not to follow the fine movement of the head but to follow only a certain degree of large movement. As the smoothing process, for example, an exponential moving average of the following equation is used.

〈OI⁢W(t+Δ⁢t)〉=ξ⁢OLW(t)+(1-ξ)⁢〈OI⁢W(t)〉(28)

In the equation above, ξ is an averaging coefficient and takes a value between 0 and 1. However, in order to ensure the visibility of the virtual object arranged in the inertial coordinate system, the origin of the inertial coordinate system is controlled so as to remain within a certain range from the origin of the local coordinate system, that is, the center of the HMD100.

(6) Page Control

It may be configured to provide a plurality of inertial coordinate systems, manage each inertial coordinate system as a page, and arrange virtual objects in each of them.FIG.11illustrates an example of virtual object information storing the types of virtual objects.FIG.11defines, in addition to the world coordinate system real object, the world coordinate system virtual object, and the local coordinate system virtual object, inertial coordinate system virtual objects displayed in a plurality of inertial coordinate systems including the inertial coordinate system1, the inertial coordinate system2, and the inertial coordinate system3. Each of the “icon group for right-handed” and the “icon group for left-handed” inFIG.11is a group of icons. Since the “icon group for right-handed” is displayed only for a right-handed user and the “icon group for left-handed” is displayed only for a left-handed user, exclusive display control is performed therein. In accordance with the exclusive display control above, control of turning on/off a virtual object and the orientation thereof in the inertial coordinate system may be performed. It may be configured to display virtual objects to allow the user to recognize which of them is an operation target, for example, by displaying a virtual object arranged in the inertial coordinate system which is the operation target (active) in a color darker than that of a virtual object arranged in the inertial coordinate system which is not the operation target (nonactive). In addition, it may be configured to change the direction in response to the user's swipe motion for each inertial coordinate system, or change the mode of the direction control method.

Furthermore, it may be configured to display marks such as numbers (color codes) for identifying the plurality of pages defined by the inertial coordinate system1, the inertial coordinate system2, and the inertial coordinate system3which are arranged near the virtual object. It may be configured to change each of the distance between the user and each page, in other words, each of the inertial coordinate system1, the inertial coordinate system2, and the inertial coordinate system3. The page that is the operation target is displayed on the front side. At this time, the size may be controlled depending on the distance so that the expected angle of the object from the user becomes constant.

(7) Display of Omnidirectional Image

A display space of the inertial coordinate system may be used to display an omnidirectional image (real object, virtual object) captured in the past. The omnidirectional image can be viewed by rotating the inertial coordinate system.

Not only the past image, but also an omnidirectional image at the current time may be displayed on the display space of the inertial coordinate system so that the user can see an image from the various directions.

(8) Omission of Local Coordinate System

It may be configured to display an important virtual object, which is used to be displayed in the local coordinate system, near a specific direction of the inertial coordinate system while not providing the local coordinate system. In this case, control for adjusting the specific direction of the inertial coordinate system to the user reference direction enables the user to access the important virtual object immediately.

(9) Automatic Arrangement of Related Object

In the case of performing processing on the world coordinate system virtual object, it may be configured to display, within the inertial coordinate system at that point, a related operation menu or the like near the world coordinate system virtual object that is a processing target. The user can access the operation menu by turning his or her head or rotating the inertial coordinate system.

(10) Application to VR Goggle and Smartphone

The present invention may be applied to a VR goggle and a smartphone. In the case of a VR goggle, an image of the external field is arranged in the world coordinate system as a video see-through image. In the case of a smartphone, a position of the user is measured and the measured position is set as the origin of the inertial coordinate system. Since the distance between the smartphone and a virtual object fixed to the inertial coordinate system changes, the display size of the virtual object may be changed in accordance with the change in the distance (zooming up when the smartphone approaches the arrangement position of the virtual object).

According to the embodiment above, it is possible to arrange a virtual object by using an inertial coordinate system as a new coordinate system for complementing defects in the world coordinate system and the local coordinate system. Conventionally, the local coordinate system has had a problem that, since the virtual object is arranged only in the FOV fixed to the coordinate system in its visible state, the number of virtual objects to be arranged thereon is limited and the visibility is reduced. The present invention is configured to define the arrangement position of the virtual object by using the inertial coordinate system, so that the virtual object can be arranged outside the FOV in the local coordinate system. As a result, the user can visually recognize the virtual object by moving his or her head when necessary, thereby making it possible to solve the problem of the limitation in the number of the virtual objects to be arranged thereon without reducing the visibility.

There has been another problem that, in the case where the virtual object is arranged in the world coordinate system, the virtual object is hardly viewed as the user moves. The present invention is configured to define the arrangement position of the virtual object by using the inertial coordinate system so that the virtual object can be moved in accordance with the movement of the user. As a result, the defects in the world coordinate system can be complemented.

The present invention is not limited to the embodiment described above, and various modifications are included therein. For example, the embodiment described above has been explained in detail in order to clarify the present invention, but is not necessarily limited to those having all the configurations described. In addition, a part of the configuration of the present embodiment can be replaced with that of another embodiment, and the configuration of another embodiment can be added to the configuration of the present embodiment. Furthermore, it is possible to add, delete, or replace another configuration with respect to a part of the configuration of the present embodiment.

Some or all the above-mentioned configurations may be configured by hardware, or the functions may be implemented by execution of programs by the processor. The control lines and the information lines which are considered to be necessary for the purpose of explanation are indicated herein, and not all the control lines and the information lines of actual products are necessarily indicated. Practically, almost all the configurations are connected to each other.

For example, in the embodiment above, the virtual object is displayed on the HMD100by using three coordinate systems, namely, the world coordinate system, the local coordinate system, and the inertial coordinate system. Meanwhile, it may be configured to display the virtual object by only using the inertial coordinate system. Furthermore, it may be configured to display the virtual object by combining the inertial coordinate system with at least one of the world coordinate system and the local coordinate system.

REFERENCE SIGNS LIST

1: HMD system100: HMD111: camera112: in-camera113: distance sensor115: acceleration sensor116: gyro sensor117: geomagnetic sensor118: GPS receiver119: display119a: effective field of view120: network communication transmitter-receiver123: antenna125: CPU126: program127: information data128: memory129: near field communication transmitter-receiver130: image131: virtual object140: bus200: server300: external network400: input controller410: virtual object420: virtual object510: virtual object520: virtual object1200: display control device