Patent ID: 12236544

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following, elements having the same functions thereamong are provided with the same reference signs unless otherwise noted, and repetitive explanation therefor will be omitted. The present invention is not limited to the present embodiment which will be described below.

In the present embodiment, as a display terminal, an example of a Head Mounted Display (HMD) which is to be worn on the head of a user (wearer) will be described. The HMD according to the present invention is a transmission type (see-through type) HMD equipped with a transmission type display, which enables the user to visually recognize both the outside world and display images.

Firstly, an outline of the present embodiment will be described with reference toFIG.1AtoFIG.1C. In the present embodiment, as illustrated inFIG.1A, a user201wears an HMD200in a room and operates the HMD200by making a virtual object100displayed on a display of the HMD200.

The HMD200has distance information (depth information) of the real space and the virtual object100. Conventionally, as illustrated inFIG.1B, the display of the HMD200does not display a portion of the virtual object100located behind (deeper than) a structural object300such as a wall in the real space. Accordingly, in the case above, the user201cannot recognize the whole of the virtual object100.

Furthermore, generally, the user201moves, rotates, and transforms the virtual object100by operating operation points (transform controller)110set on or near the virtual object100. When the virtual object100is displayed as illustrated inFIG.1B, the user201cannot operate the virtual object100since the operation points110of the portion of the virtual object100behind the structural object300are not displayed.

In the present embodiment, in order to solve the problem above, as illustrated inFIG.1C, all the operation points110of the virtual object100are displayed even when the structural object300exists. In addition, the whole of the virtual object100itself may be displayed as well. At this time, an additional object400may be displayed so as to let the user201recognize that a portion which is not originally displayed is being displayed.

Hereinafter, the HMD200according to the present embodiment for realizing the display control as described above will be explained.

[Hardware Configuration]

FIG.2Aillustrates a hardware configuration of the HMD200.FIG.2Billustrates an appearance of the HMD200according to the present embodiment.

The HMD200according to the present embodiment basically has the same configuration as that of a general-purpose computer (information processing device). That is, as illustrated inFIG.2A, the HMD200includes a controller210, cameras214, a display215, an audio interface (I/F)216, a communication I/F217, sensors218, a bus219for electrically connecting each part, and a line-of-sight detection device214c. In addition, the HMD200according to the present embodiment includes a frame241for supporting each part of the HMD200and allowing the user201to wear the HMD200.

The controller210is configured to perform various kinds of processing in accordance with predetermined programs. In the present embodiment, for example, the controller210displays the virtual object100at a predetermined position on the display215.

The controller210of the present embodiment includes a CPU211, a RAM212, and a ROM213. The CPU211loads programs stored in advance in the ROM213onto the RAM212and executes them to implement various functions. In this connection, both the RAM212and the ROM213are collectively referred to as a storage device230(seeFIG.3) in the case of not requiring distinguishment between them. The controller210is disposed, for example, on the frame241.

The cameras214include a color image camera214aand a distance image camera214b. The color image camera214ais configured to capture images of a photographing range including a visual field range of the user201to acquire a color image. The distance image camera214bis configured to acquire a distance image of a photographing range which is substantially the same as the photographing range of the color image camera214a. The cameras214(color image camera214aand distance image camera214b) are disposed, for example, at positions on the foremost portions of the frame241(closest to the display215) which enable the cameras214to capture images of the above-mentioned photographing range.

The display215displays images acquired by the cameras214and display data generated in the HMD200. The display215is constituted by, for example, a transmission type liquid crystal device, an organic EL device, or an optical scanning device using a Micro Electro Mechanical Systems (MEMS). Meanwhile, the device for constituting the display215is not limited thereto, and any device may be used as long as it can realize a transmission type display structure which allows the other side of the display215to be seen through while allowing an image to be displayed on the display215.

In the case of the HMD200according to the present embodiment, the transmission type display215is supported in front of one or both eyes of the user201. The display215can be any shape. The display215may be provided with right and left display panels, and may display one or more UI objects of a graphical user I/F.

The audio I/F216is, for example, an audio output device such as a microphone, a speaker, and a buzzer. The audio I/F216is configured to input an external sound and output a sound such as a sound created in the HMD200and a sound or music transmitted through the communication I/F217. Meanwhile, in the present embodiment, the audio I/F216may not be provided.

The communication I/F217includes such as a coding circuit, a decoding circuit, and an antenna to transmit and receive data to and from other devices through a network (data communication). In the present embodiment, the communication I/F217is an I/F for connecting the HMD200to the network via an access point (not illustrated) or via a base station of a mobile telephone communication network (not illustrated). The HMD200transmits and receives data to and from each server connected to the network via the communication I/F217.

The connection between the HMD200and the access point is performed by, for example, a wireless communication system such as Wi-Fi (registered trademark) or by other communication systems. The connection between the HMD200and the base station of the mobile telephone network is performed by, for example, a Wideband Code Division Multiple Access (W-CDMA, registered trademark) method, a Global System for Mobile communications (GSM) method, a Long Term Evolution (LTE) method, or other communication methods. Meanwhile, in the present embodiment, the HMD200may not be provided with the communication I/F217.

The sensors218are configured to detect such as a current position, inclination, velocity, and an operation by the user201of the HMD200. The HMD200includes, for example, a positional information acquisition sensor such as a GPS receiver218a, a gyro sensor218b, an acceleration sensor218c, a geomagnetic sensor218d, and a touch sensor218e, as the sensors218. Meanwhile, the sensors218may not necessarily be provided with all the sensors above.

The line-of-sight detection device214cis configured to detect a line-of-sight direction of the user201. The line-of-sight detection device214cis implemented by, for example, a line-of-sight detection camera for detecting the line-of-sight direction of the user201. The line-of-sight detection camera is attached so as to include such as an iris and a pupil of the eye of the user201in its photographing range.

The frame241supports components of the HMD200such as the display215, the cameras214, and the controller210.

[Functional Block]

Next, the functions related to virtual object display processing, which are implemented by the controller210of the present embodiment will be described.FIG.3is a functional block diagram illustrating the functions related to the virtual object display processing in the HMD200according to the present embodiment. As illustrated inFIG.3, the controller210of the present embodiment implements the functions of an image acquisition unit228, a display control unit220, and an audio output control unit229. The display control unit220is provided with a space recognition unit221, an instruction reception unit222, a display data generation unit223, and a display correction unit224.

The CPU211loads the programs stored in the ROM213onto the RAM212and executes them so that each of the functions are implemented.

The storage device230is configured to store color image data231, distance image data232, space recognition data233, virtual object data (virtual OJT data)234, additional object data (additional OJT data)235, and audio data236.

The color image data231is an image acquired by the color image camera214a. The distance image data232is an image acquired by the distance image camera214b.

The image acquisition unit228is configured to cause the storage device230to store, as the color image data231and the distance image data232, the color image and the distance image acquired by the color image camera214aand the distance image camera214b, respectively. In the present embodiment, the color image and the distance image are acquired substantially synchronously with each other.

The space recognition unit221is configured to recognize the surrounding real space and cause the storage device230to store the result of the recognition above as the space recognition data233. The space recognition unit221recognizes the surrounding real space based on the color image data231and the distance image data232acquired substantially simultaneously.

In accordance with a scanning operation performed by the user201, the space recognition unit221generates the space recognition data233, which is three-dimensional data (three-dimensional map) of the structural object300existing in the photographing range, based on each image data at predetermined time intervals and causes the storage device230to store the generated space recognition data233. The scanning of the surroundings is performed by the user201, for example, immediately after startup as an initial setting.

The space recognition data233is generated, for example, in a world coordinate system that defines the whole three-dimensional space. For example, as the origin and each axis direction of the world coordinate system, the origin and each axis direction in a local coordinate system of the HMD200, which are specified based on the position and orientation (initial posture) of a main body of the HMD200at the time of receiving an instruction for starting space recognition, are used. In this coordinate system, for example, regarding the initial posture of the main body of the HMD200, when a predetermined position on the display215of the HMD200is set as the origin and the plane of the display215is set as the xy-plane, the z-axis direction is perpendicular to the xy-plane (plane of the display215).

In this connection, an amount of displacement and an amount of rotation of the HMD200in the local coordinate system in response to the scanning operation by the user201with respect to the world coordinate system are calculated based on data obtained by the various sensors218.

The space recognition is performed, for example, by using the technique such as the conventional Spatial Mapping. Specifically, the HMD200according to the present embodiment scans the surroundings by means of the color image camera214aand the distance image camera214b. Then, based on the result of the space recognition, the space recognition unit221uses an application software such as the Spatial Mapping to generate three-dimensional data. The space recognition data233is held as, for example, mesh data.

At this time, Spatial Understanding for recognizing, not only the three-dimensional data, but also the type of the structural object300may be performed simultaneously. The space recognition unit221can recognize the material and type of the structural object existing within the photographing range by the Spatial Understanding. That is, the space recognition unit221can recognize whether the structural object300is, for example, a wall, a floor, or a ceiling. The space recognition unit221of the present embodiment causes the storage device230to store the recognition results as attribute data of the space recognition data233.

The instruction reception unit222is configured to accept, from the user201, a display instruction and an operation instruction with respect to the virtual object100to be displayed on the display215. The display instruction and the operation instruction include, for example, instructions by a line-of-sight (gaze) and movement (gesture) of a finger.

The information of the line-of-sight direction used for the gaze is detected by, for example, the line-of-sight detection device214c.

The gesture includes, for example, click-event (air tap), tap-and-hold, and bloom on the operation points110of the virtual object100. The instruction reception unit222detects movement of a finger in a gesture frame provided within the photographing range of the color image camera214aand the distance image camera214bto detect such as the display instruction and the operation instruction.

For example, upon accepting the display instruction, the instruction reception unit222extracts, from the virtual object data234, the data of the virtual object100that is subject to the instruction, and causes the display data generation unit223which will be described later to generate the display data.

Upon accepting the operation instruction, the instruction reception unit222detects the operation and notifies it to the display data generation unit223, which will be described later.

In accordance with the instruction from the user201through the instruction reception unit222, the display data generation unit223generates, based on the virtual object data234, the display data for displaying the virtual object100that is subject to the instruction at a predetermined position on the display215in a predetermined shape. The display data generated in accordance with the instruction from the instruction reception unit222is displayed on the display215, whereby the virtual object100is displayed so as to be moved, rotated, and transformed in accordance with the instruction of the user201.

At this time, the display data generation unit223generates the display data in which, in the line-of-sight direction of the user201(calculated based on the real space coordinate position of the HMD200worn by the user201and the information of the vertical and horizontal orientations thereof), a region of the virtual object100behind the structural object300is specified as a rear region101. The rear region101(FIG.4) is specified based on the three-dimensional map (space recognition data233) generated by the space recognition unit221and the placement position data of the virtual object100in the real space. The placement position data in the real space is stored in the same coordinate system as that used for the space recognition performed by the space recognition unit221.

The placement position data of the virtual object100in the real space is acquired from the virtual object data234of the virtual object100to be displayed. The virtual object data234includes the size, shape, and initial placement position information for each virtual object100. As the initial placement position information, for example, a placement position point and the operation points110which have been set in advance for each virtual object100are stored. The placement position point is, for example, a three-dimensional position of the center of gravity of the virtual object100. As described above, the operation points110are the points that accept an instruction for transforming the display shape of the virtual object100.

The display data generation unit223reflects the instruction from the instruction reception unit222to the current placement position information to acquire the latest placement position information of the virtual object data234corresponding to the virtual object100to be displayed. Then, the display data generation unit223uses the latest placement position information of the placement position point and the information such as the size and shape to acquire the placement position data of the virtual object100in the real space.

The display data generation unit223uses the latest placement position data in the real space to further identify the rear region101of the virtual object100to be displayed. For example, based on the placement position data of the virtual object100in the real space, the display data generation unit223specifies the depth information of the virtual object100. The depth information is specified based on the distance to the virtual object100, which is calculated based on the real space coordinate position of the HMD200worn by the user201and the vertical and horizontal orientations, the shape data of the virtual object100, and the coordinate position of the structural object300.

In the depth information, the region (portion) of the virtual object100and the operation points110which are positioned relatively deeper than the depth information of the structural object300are defined as the rear region101.

Generally, the rear region101is subject to hidden surface removal processing. For example, as illustrated inFIG.4A, when a portion of the virtual object100is placed deeper than the structural object300such as a wall or furniture, conventionally, the display data generation unit223implements the hidden surface removal processing on the rear region101of the virtual object100to generate the display data for hiding the rear region101.

On the other hand, since the present embodiment aims to achieve more a user-friendly display mode, even in the case above, the display correction unit224implements the processing for correcting a display data so as to display the whole of the virtual object100and the associated operation points110as illustrated inFIG.4B.

The display correction unit224is configured to correct the display data of the virtual object data234in the case where the virtual object data234to be displayed includes the rear region101.

Specifically, the display correction unit224cancels the hidden surface removal processing on the rear region101. Then, the display correction unit224corrects the display data so as to display the rear region101as if the structural object300does not exist. That is, the display correction unit224corrects the display data so that the rear region101is displayed even if a partial region of the virtual object data234is placed deeper than the structural object300.

Furthermore, at this time, the display correction unit224displays the additional object400to let the user know that the virtual object100is not being displayed in an original display mode. The additional object400is an object for making the virtual object100appear more naturally with reality, for example, by displaying the virtual object100as if it penetrated the structural object300or a hole was formed in the structural object300.

The data of the additional object400to be displayed is prepared in advance as the additional object data235in the storage device230. The additional object data235is stored in association with the information such as a display mode of the additional object400, initial size, and initial display position with respect to a display position of the virtual object100.

When the display direction of the virtual object100with respect to the line-of-sight direction of the user201is changed in accordance with the operation performed by the user201, the display correction unit224may transform the display shape of the additional object400in response to the change in the line-of-sight direction. Furthermore, when the display direction of the virtual object100is changed in accordance with the movement of the line-of-sight direction of the user201, the display correction unit224may also transform the display shape of the additional object400in the same manner as above. The display correction unit224transforms the additional object400in accordance with, for example, the same program as that used for transforming the display of the virtual object100in response to the change in the line-of-sight direction.

Next, a flow of display processing of the virtual object100to be performed by the controller210of the present embodiment will be described.FIG.5illustrates a flow of the virtual object display processing of the present embodiment. In the following, it is assumed that the space recognition processing has already been performed as the initial processing. Furthermore, it is assumed that the virtual object100is displayed at a position allowing the whole object to be displayed.

The display data generation unit223and the display correction unit224repeat the following processing until receiving a termination instruction from the user201via the instruction reception unit222(step S1101).

Upon accepting an operation for moving the virtual object100from the user201via the instruction reception unit222(step S1102), the display data generation unit223generates display data (step S1103).

In step S1103, firstly, the display data generation unit223calculates a display position of the virtual object100on the display215. Next, the display data generation unit223specifies a line-of-sight direction, and calculates depth information in accordance with the virtual object data234corresponding to the virtual object100. In addition, the display data generation unit223also calculates depth information of the structural object300. Then, the display data generation unit223specifies the rear region101in the display data.

Next, the display data generation unit223determines whether the virtual object100to be displayed includes the rear region101(step S1104).

In the case of absence of the rear region101(step S1104; No), the display data generation unit223notifies the display correction unit224thereof. The display correction unit224displays the display data of the virtual object100at the position on the display215which has been calculated in step S1103without any correction (step S1112). Then, the controller210waits for the next operation instruction.

In the case of presence of the rear region101(step S1104; YES), the display data generation unit223determines whether the whole of the virtual object100is the rear region101(step S1105).

When the whole of the virtual object100is the rear region101(step S1105; YES), the display data generation unit223notifies the display correction unit224thereof. Then, the display correction unit224clears the display of the virtual object100(step S1111), and terminates the processing.

On the other hand, when the whole of the virtual object100is not the rear region101(step S1105; No), in other words, when the rear region101exists in a portion of the virtual object100, the display data generation unit223notifies the display correction unit224thereof. The display correction unit224corrects the display data of the virtual object100by the above-described method (step S1106). Here, the display correction unit224corrects the display data so as to display all the operation points110of the virtual object100and the portion of the virtual object100which is subject of an operation by the operation points110. Then, the display correction unit224displays the corrected display data at the position on the display215which has been calculated in step S1103(step S1107).

Thereafter, the display correction unit224acquires the additional object data235from the storage device230, and superimposes the additional object data235on the rear region101of the virtual object100to display the superimposed data (step S1108). Then, the controller210waits for the next operation instruction.

Hereinafter, the processing described above will be explained by using a specific example. As illustrated inFIG.6A, a frame215adefined by the display215displays the virtual object100and the additional object400. Here, the structural object300is a real object in the real space. In the following, a display mode will be described while omitting the frame of the display215from the drawings. As illustrated inFIG.6B, the subsequent drawings illustrate only the real object which is visible through the display215and the display data of the virtual object100.

For example, it is assumed that the virtual object100is displayed at a position illustrated inFIG.7A. When the user201provides, in the real space, an operation instruction (gesture) to push the virtual object100in the direction of the arrow203with the finger202and repeats this operation, the display position of the virtual object100relative to the structural object300that is the real object in the real space is made to move.

According to the present embodiment, as illustrated inFIG.7B, even when the display position of a portion of the virtual object100is located deeper than the structural object300, the virtual object100is displayed as a whole, together with its operation points110. Furthermore, at this time, the additional object400is displayed to show as if a hole was formed in the wall that is the structural object300.

Furthermore, at this time, as illustrated inFIG.7C, the audio output control unit229may be configured to perform audio output. When displaying the additional object400, the display correction unit224notifies the audio output control unit229thereof. Upon receiving the notification from the display correction unit224, the audio output control unit229extracts a sound from the audio data236to output the sound via the audio I/F216.

As the audio data236, various pieces of data may be stored in association with the materials of the structural object300. The materials of the structural body300are recognized by, for example, Spatial Understanding implemented by the space recognition unit221described above.

In addition, at this time, an onomatopoeic word representing the voice data236to be output by a string, for example, “crack” as illustrated inFIG.7C, may be displayed. In this case, onomatopoeic word data237for representing the audio data236by strings is associated with the voice data236and stored in advance in the storage device230. The display correction unit224generates the display data so as to display, near the additional object400, the onomatopoeia word data237which has been stored in association with the sound data236to be output.

In this connection, a balloon may be further displayed and the onomatopoeic word data237may be displayed in the balloon. In addition, only the onomatopoeia word data237may be displayed without outputting the sound data236.

As the virtual object100sinks into the structural object300deeply, the display correction unit224further transforms the shape of the additional object400in response thereto. For example, as illustrated inFIG.8A, a case where the virtual object100has the shape combining two rectangular parallelepipeds with different widths therebetween in the horizontal direction will be described as an example. Here, it is assumed that the virtual object100is to be pushed into the structural object300from the side of the rectangular parallelepiped having a small width in the horizontal direction in the direction of an arrow203.

In this case, as illustrated inFIG.8B, the display correction unit224processes the size of the additional object400in accordance with the width of the virtual object100in the horizontal direction and display the processed additional object400. Specifically, as illustrated inFIG.8A, when a region of the rectangular parallelepiped having the small width in the horizontal direction virtually sinks through the structural object300, the display correction unit224displays the additional object400in the small size. On the other hand, when a region having the width in the horizontal direction larger than that in the case ofFIG.8Avirtually sinks through the structural object300, the display correction unit224displays the additional object400in the size larger than that shown inFIG.8Aas illustrated inFIG.8B.

Furthermore, the line-of-sight direction changes due to change in standing positions of the user201. For example, as illustrated inFIG.8C, when the user201faces the wall that is the structural object300, even if the display position of the virtual object100in the world coordinate system does not change, the display mode of the virtual object100varies between the case ofFIG.8Cand the cases ofFIG.8AandFIG.8B. Accordingly, the display shape of the virtual object100also changes.

In this case, as illustrated inFIG.8C, the display correction unit224may also transform and display the shape of the additional object400in response to the change in the display shape of the virtual object100.

As described above, the HMD200according to the present embodiment comprises: the color image camera214aconfigured to acquire a color image of a predetermined photographing range; the distance image camera214bconfigured to acquire a distance image of the photographing range; the display215; and the display control unit220configured to display the virtual object100on the display215. The display control unit220includes: the space recognition unit221configured to use the color image and the distance image to generate a three-dimensional map of the structural object300existing within the photographing range; the display data generation unit223configured to generate display data in which a region of the virtual object100behind the structural object300in a line-of-sight direction is specified as the rear region101, based on the three-dimensional map and real space placement position data of the virtual object100to be displayed; and the display correction unit224configured to correct the display data and display the corrected display data on the display215. The display correction unit224is configured to correct the display data so as to display the operation points110of the rear region101. The operation points110are points that accept an operation instruction with respect to the virtual object100via the operation points110.

As described above, the present embodiment is configured to display the operation points110of the rear region101of the virtual object100behind the structural object300, which is supposed not to be displayed in a natural state. With this configuration, even when the virtual object100is moved in accordance with the operation performed by the user201, the user201can operate the virtual object100after the movement.

Furthermore, in the present embodiment, when displaying the operation points110of the rear region101, the display correction unit224also displays the rear region101as well as the additional object400around the virtual object100. With this configuration, regardless of the display position of the virtual object100on the display215, it is possible to realize a high operability without depriving the user of a natural visibility, while maintaining a sense of reality.

First Modification

The display mode of the virtual object100in which the rear region101is partially provided is not limited to the embodiment above. For example, the display correction unit224may display the virtual object100so as to distinguish the rear region101from another region102(hereinafter, referred to as a front region).

For example, as illustrated inFIG.9A, the display correction unit224corrects the display data so as to display a line121between the rear region101and the front region102. Otherwise, as illustrated inFIG.9B, the display correction unit224corrects the display data so as to display the rear region101in a surface pattern122which differentiates the surface of the rear region101from that of the original virtual object100. The correction mode illustrated inFIG.9Aand that inFIG.9Bmay be combined with each other.

Second Modification

The additional object400may be highlighted.FIG.10illustrates an example of the highlighted additional object400. The manner of highlighting for the additional object400may be stored in advance in the additional object data235. Otherwise, the display correction unit224may use various image processing software to process and display the stored additional object400in the highlight display mode.

Highlighting the additional object400enables the user201to easily recognize that the display mode of the virtual object100is different from the original display mode thereof. That is, in the highlight display mode, the user201can grasp more intuitively that the region which is supposed not to be displayed is being displayed.

Third Modification

The additional object data235may include textures which vary depending on the materials and/or types of the structural object300into which the virtual object100sinks. In this case, the display correction unit224refers to the space recognition data233generated by the space recognition unit221to specify the material and/or type of the structural object300. Then, the display correction unit224extracts the additional object data235of the texture corresponding to the specified material and/or type, and displays the extracted additional object data235as the additional object400.

For example, when the structural object300is made of a hard material such as a wall, the additional object400representing a state in which the wall is broken as illustrated inFIG.10is prepared as the additional object data235. On the other hand, when the structural object300is made of a soft material such as a cushion, an additional object402representing a state in which the virtual object100sinks into the structural object300is prepared and displayed as illustrated inFIG.11.

In this connection, at this time, the highlight display mode may be changed depending on the material and/or type of the structural object300or the texture of the additional object400.

Fourth Modification

The display correction unit224may cut out a part of the color image data231and use it as the additional object400. Particularly, as illustrated inFIG.12, an image of the same texture as that of a structural object301, which exists further behind the structural object300into which the virtual object100is pushed, may be used. In this case, the display correction unit224uses the space recognition data233to process the additional object data235in accordance with the placement position of the additional object400. Then, the display correction unit224places the processed additional object data235.

As described above, when the data of the texture which is the same as that of the background image is used as the additional object400, the user201can more naturally accept the display mode of the virtual object100according to the embodiment above.

Fifth Modification

The embodiment described above is configured to clear the display of the virtual object100when the virtual object100is moved to a position where the whole of the virtual object becomes the rear region101. Meanwhile, the present invention is not limited thereto. Even when the whole of the virtual object100becomes the rear region101, the virtual object100and/or the operation points110may be displayed.

For example, as illustrated inFIG.13, the display correction unit224corrects the display data so as to display the virtual object100at the calculated position. In addition, the display correction unit224may display, as the additional object, a miniature virtual object401for assisting a use's operation in front of the virtual object100. In this case, the display correction unit224generates and displays the miniature virtual object401based on the virtual object100.

With this configuration, even when the whole of the virtual object100is placed deeper than the structural object300in the depth direction, the user201can perform an operation while feeling a sense of reality.

Sixth Modification

The display215may be a non-transmission type display. In this case, as illustrated inFIG.14, the display correction unit224superimposes the virtual object100and the additional object400on the color image (through image500) acquired by the color image camera214a.

In the case above, the display terminal is not limited to the HMD200including a transmission type display. For example, the display terminal may be a portable information processing device such as an HMD and a portable terminal which include a non-transmission type display.

Seventh Modification

Means for inputting an operation instruction is not limited to a line of sight and gesture, but may include such as a voice and a motion controller.

Eighth Modification

In the embodiment above, the display correction unit224is configured to correct the display of the virtual object100so as to display all the operation points110of the virtual object100. Meanwhile, the present invention is not limited thereto. The operation points110to be displayed may be a part of the operation points110of the rear region101. In other words, at least one of the operation points110of the rear region101may be displayed.

Similarly, the entire shape of the virtual object100is not necessarily displayed. A part of the rear region101may be displayed.

The present invention is not limited to the embodiment and the modifications described above, and other various modifications are included therein. For example, the embodiment and the modifications described above have been explained in detail in order to clarify the present invention, but are not necessarily limited to those having all the configurations as described. In addition, a part of the configuration of the present embodiment and the modifications can be replaced with that of other embodiments and other modifications, and the configuration of other embodiments and other modifications can be added to the configuration of the present embodiment. Furthermore, it is possible to add, delete, or replace another configuration with respect to a part of the configuration of the present embodiment and the modifications.

Some or all the configurations described above, functions, processing units, and processing means may be implemented by hardware, for example, by designing them with an integrated circuit. In addition, the configurations and functions described above may be implemented by software by interpreting and executing programs in which the processor implements the respective functions. Information such as programs, tables, and files for implementing various functions can be placed in recording devices such as a memory, a hard disk, and an SSD (Solid State Drive), or recording media such as an IC card, an SD card, and a DVD.

Furthermore, the control lines and the information lines which are considered to be necessary for the purpose of explanation are indicated herein, but not all the control lines and the information lines of actual products are necessarily indicated. It may be considered that almost all the configurations are actually connected to each other.

REFERENCE SIGNS LIST

100: virtual object,101: rear region,102: front region,110: operation points,121: line,122: surface mode,200: HMD,201: user,202: finger,203: arrow,210: controller,211: CPU,212: RAM,213: ROM,214: cameras,214a: color image camera,214b: distance image camera,214c: line-of-sight detection device,215: display,215a: frame,216: audio I/F,217: communication I/F,218: sensors,218a: GPS receiver,218b: gyro sensor,218c: acceleration sensor,218d: geomagnetic sensor,218e: touch sensor,219: bus,220: display control unit,221: space recognition unit,222: instruction reception unit,223: display data generation unit,224: display correction unit,228: image acquisition unit,229: audio output control unit,230: storage device,231: color image data,232: distance image data,233: space recognition data,234: virtual object data,235: additional object data,236: audio data,237: onomatopoeic word data,241: frame,300: structural object,301: structural object,400: additional object,401: miniature virtual object,402: additional object,500: through image