MIXED REALITY SYSTEM, PROGRAM, MOBILE TERMINAL DEVICE, AND METHOD

One or more embodiments of the invention is a mixed reality system for displaying a mixed-reality image on a display, the system including a mobile terminal device having the display and a photographing device. The system includes a plurality of feature point sets arranged in the prescribed real space. The mobile terminal device recognizes each of the feature point sets photographed by the photographing device, determines a viewpoint position of a virtual camera, in a virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, based on arrangement position and posture information for each of the feature point sets and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, and generates a mixed-reality image in which an image of the virtual object is superimposed on a photographed image of the real space.

TECHNICAL FIELD

The present invention relates to a mixed reality system, etc., particularly to a mixed reality system, etc. that makes it possible for a user present in a prescribed real space to experience a mixed reality.

BACKGROUND ART

Recently, mixed reality (MR) technology for seamlessly integrating the real world with a virtual world in real time has been known. MR technology makes it possible for a user experiencing MR technology to have an experience as if a virtual object is present in that location. Regarding MR technology, technology for executing MR itself with high accuracy such that an object in a virtual space is displayed without causing discomfort to a user has been developed. For example, NPL 1 discloses a system capable of recognizing a real space environment in a mixed reality environment in real time.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

Conventionally, the target of MR technology has been a real space within a sensor observation range, and it has been difficult to realize MR in a wide space. For example, in an outdoor space, it has been difficult to realize an MR in which a virtual huge robot flies between buildings.

The present invention has been made in order to solve the problem described above, and the chief object thereof is to provide a mixed reality system, etc. that makes it possible for a user to experience, in a relatively wide space such as an outdoor space, MR according to the location.

Solution to Problem

In order to achieve the above object, a system according to an aspect of the present invention is a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system including a mobile terminal device having the display and a photographing device that photographs the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships, and the mobile terminal device stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space, recognizes each of the feature point sets photographed by the photographing device, determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.

Furthermore, in the present invention, preferably, in the case where the first virtual object and the second virtual object overlap each other in a field-of-view range of the virtual camera when the mixed-reality image is generated, if a depth distance of the second virtual object from the viewpoint position is larger than that of the first virtual object, the mobile terminal device generates a mixed-reality image in which the image of the second virtual object with overlapping parts removed is superimposed.

Furthermore, in the present invention, preferably, the mixed reality system includes five or more feature point sets, and the mobile terminal device, at prescribed time intervals, recognizes each of the feature point sets photographed by the photographing device, determines the viewpoint position, and generates the mixed-reality image.

Furthermore, in the present invention, preferably, the mixed reality system includes four feature point sets arranged in the prescribed real space, and the individual feature point sets have positional relationships corresponding to four vertex positions of a virtual rectangle in the real space and are arranged away from each other by a predefined distance in a vertical direction and a predefined distance in a horizontal direction.

Furthermore, in the present invention, preferably, one or a plurality of the feature point sets are securely attached to each of a plurality of rod-like bodies installed in a horizontally-isolated manner.

Furthermore, in the present invention, preferably, each of the feature point sets is securely attached to a transmission member formed of a light transmitting material.

Furthermore, in the present invention, preferably, each of the feature point sets is securely attached by using a wire.

In order to achieve the above object, a program according to an aspect of the present invention is a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system including the mobile terminal device having the display and a photographing device that photographs the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships, and the program causes the mobile terminal device to execute a step of recognizing each of the feature point sets photographed by the photographing device, a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and a step of generating a mixed-reality image in which an image of a second virtual object according to the viewpoint position is superimposed on a photographed image of the real space, on the basis of data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position.

Furthermore, in order to achieve the above object, a mobile terminal device according to an aspect of the present invention is a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system including the mobile terminal device having the display and a photographing device that photographs the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and the mobile terminal device stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space, and recognizes each of the feature point sets photographed by the photographing device, determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.

Furthermore, in order to achieve the above object, a method according to an aspect of the present invention is a method executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system including the mobile terminal device having the display and a photographing device that photographs the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and the method includes a step of recognizing each of the feature point sets photographed by the photographing device, a step of determining a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and a step of generating a mixed-reality image in which an image of a second virtual object according to the viewpoint position is superimposed on the photographed image of the real space, on the basis of data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, data of the second virtual object, in the virtual space, that does not correspond to the real object, and the viewpoint position.

Furthermore, in order to achieve the above object, a system according to an aspect of the present invention is a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system including a server and a mobile terminal device having the display and a photographing device that photographs the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; the server stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space; and the mobile terminal device recognizes each of the feature point sets photographed by the photographing device, determines a viewpoint position of a virtual camera, in the virtual space, corresponding to a position and a photographing direction of the photographing device in the real space, on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and on the basis of the data of the first virtual object, the data of the second virtual object, and the viewpoint position, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.

Furthermore, in order to achieve the above object, a system according to an aspect of the present invention is a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system including a mobile terminal device having the display and a photographing device that photographs the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; the mobile terminal device stores data, obtained in advance, of a first virtual object that corresponds to a real object present in the prescribed real space and that defines the virtual space, and data of a second virtual object, in the virtual space, that does not correspond to the real object, stores arrangement position and posture information in the virtual space for each of the feature point sets arranged in the prescribed real space, recognizes each of the feature point sets photographed by the photographing device, determines position and posture information of the first virtual object on the basis of the arrangement position and posture information for each of the feature point sets, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and on the basis of the data of the first virtual object and the data of the second virtual object, generates a mixed-reality image in which an image of the second virtual object according to the viewpoint position is superimposed on the photographed image of the real space.

Furthermore, in order to achieve the above object, a program according to an aspect of the present invention is a program executed by a mobile terminal device in a mixed reality system for displaying, on a display for displaying a virtual object to a user present in a prescribed real space, a mixed-reality image in which an image of the virtual object arranged in a virtual space corresponding to the real space is superimposed on a photographed image of the real space, the mixed reality system including the mobile terminal device having the display and a photographing device that photographs the real space, characterized in that: the mixed reality system includes a plurality of feature point sets arranged in the prescribed real space, the feature point sets including identifiable information that allows identification of each of the feature point sets, and at least three feature point sets among the plurality of feature point sets being arranged so as to have predefined positional relationships; and the program causes the mobile terminal device to execute a step of recognizing each of the feature point sets photographed by the photographing device, a step of determining position and posture information of a first virtual object, obtained in advance, that corresponds to a real object present in the prescribed real space and that defines the virtual space, on the basis of arrangement position and posture information in the virtual space in which each of the feature point sets is arranged, the arrangement position and posture information being obtained from the identifiable information of some or all of the recognized feature point sets, and relative position and posture information of the mobile terminal device with respect to each of the feature point sets, the relative position and posture information being determined from shapes and sizes of the feature point sets, and a step of generating a mixed-reality image in which an image of a second virtual object according to a viewpoint position is superimposed on the photographed image of the real space, on the basis of data of the first virtual object and data of the second virtual object, in the virtual space, that does not correspond to the real object.

Advantageous Effects of Invention

According to the present invention, it is possible for a user to experience, in a relatively wide space such as an outdoor space, MR according to the location.

DESCRIPTION OF EMBODIMENT

Now, a mixed reality (MR) system that provides a user with a mixed reality space in which a virtual space and a real space are mixed together will be described with reference to the drawings. In this specification, for convenience of description, descriptions that are more detailed than necessary may be omitted. For example, detailed descriptions about well-known matters or repeated descriptions about substantially the same configurations may be omitted. Note that, in each figure, the same signs indicate the same or corresponding parts unless specifically mentioned otherwise.

An MR system according to this embodiment allows a user to experience an MR by rendering, in a superimposed fashion, a virtual object, which is an object arranged in a virtual space corresponding to a real space, on a photographed real space image visually recognized by a user through, for example, the screen of a smartphone.

FIG. 1is a schematic illustration showing part of a mixed reality system (MR system)1according to an embodiment of the present invention. The MR system1according to the embodiment of the present invention is realized in a predefined prescribed real space (real space50). The real space50is a predefined outdoor real space, and in that space, real objects, which are objects in the real world, exist. For example, the real objects are structures such as buildings, bridges, and walls, which are stationary in the real space. However, mobile items may be included in the real objects. Furthermore, the real space50may be an indoor real space instead of an outdoor real space, and may partly include the indoor real space.

As shown inFIG. 1, the MR system1includes a mobile terminal device2, and a feature point set cloud3that includes a plurality of feature point sets4arranged in the real space50.

FIG. 2is a block diagram showing the hardware configuration of the mobile terminal device2according to the embodiment of the present invention. The mobile terminal device2includes a processor11, an input device12, a display device13, a photographing device14, a storage device15, and a communication device16. These constituent devices are connected via a bus17. Note that interfaces are interposed between the bus17and the individual constituent devices as needed. In this embodiment, the mobile terminal device2is a smartphone. However, the mobile terminal device2may be a terminal such as a tablet computer or a computer equipped with a contact-type input device, such as a touchpad, as long as the terminal is a portable terminal having the above-described configuration.

The processor11controls the overall operation of the mobile terminal device2. For example, the processor11is a CPU. Alternatively, an electronic circuit such as an MPU may be used as the processor11. The processor11executes various kinds of processing by loading programs and data stored in the storage device15and executing the programs. In one example, the processor11is constituted of a plurality of processors.

The input device12is a user interface that accepts an input to the mobile terminal device2from a user. The display device (display)13displays an application screen, an image photographed by the photographing device14, or the like to the user of the mobile terminal device2in accordance with the control by the processor11. In this embodiment, the mobile terminal device2, which is a smartphone, includes a touchscreen as the input device12, the touchscreen also functions as the display device13, and the input device12and the display device13are constructed in an integrated form. However, the input device12and the display device13may be disposed at different positions as separate units. In this case, the input device12may be, for example, a touchpad or a button, and the display device13may be a liquid crystal display, a display using an organic EL, a plasma display, or the like.

The photographing device (image capturing device)14photographs (captures) a still picture or a moving picture of a real space and stores the photographed image or moving picture data in the storage device15. The photographing device14is, for example, a camera constituted of an image sensor.

The storage device15is a storage device provided in a general smartphone and including a RAM, which is a volatile memory, and a ROM, which is a non-volatile memory. The storage device15may include an external memory.

In one example, the storage device15includes a main storage device and an auxiliary storage device. The main storage device is a volatile storage medium that allows high-speed reading and writing of information and is used as a storage area and a work area when the processor11processes information. The auxiliary storage device stores various programs and data that is used by the processor11when executing the individual programs. The auxiliary storage device is, for example, a hard disk device, but may be any kind of non-volatile storage or non-volatile memory that is capable of storing information, and may be of the removable type. For example, the auxiliary storage device stores an operating system (OS), middleware, application programs, various kinds of data that may be referred to as these programs are executed, etc.

The communication device16sends data to and receives data from other computers via a network such as the Internet. For example, the communication device16carries out wireless communications such as mobile communications or wireless LAN communications to connect to the network. In one example, the communication device16downloads a program from a server and stores the program in the storage device15. However, the communication device16may carry out wired communications using an Ethernet (registered trademark) cable or the like. In the case where data is not sent to or received from other computers, the mobile terminal device2need not include the communication device16.

Feature point sets4indicate local feature point sets and, in this embodiment, the feature point sets4are ArUco markers (NPL 1). The feature point sets4are arranged and fixed in the real space50in advance. Each of the feature point sets4is associated with unique identification information (ID). However, it suffices that each of the feature point sets4includes identifiable information that makes it possible to identify each of the feature point sets4. Identification information is an example of the identifiable information. Furthermore, the feature point sets4are not limited to the ArUco markers.

FIG. 3is a diagram showing a feature point set4according to the embodiment of the present invention. The feature point set4is constituted of white and black squares and has a square shape as a whole. The black squares are arranged at positions constituting the outer edge of the feature point set4. The white and black squares are arranged at different positions in each feature point set4, and an ID (for example, ID number) is assigned according to the arrangement position. For example, the feature point set4is made by performing printing on a metal plate having a flat surface by using a pigment, or attaching the metal plate after printing to another support, and the metal plate is coated for sunlight reflection prevention.

It suffices that the feature point set4makes it possible to identify each feature point set4and is arranged and fixed in the real space50, and thus is not limited to the above-described example. The feature point set4may be made by printing the pattern of the feature point set4itself, or may be made by including the pattern of the feature point set4itself in what is printed. For example, the feature point set4has a square shape, the side length of which is 10 cm.

As shown inFIG. 1, the feature point set cloud3includes four feature point sets4, and each of the feature point sets4is securely attached, in a vertically-isolated manner, to each of a pair of rod-like bodies41installed in a horizontally-isolated manner. For example, the rod-like bodies41are fixed poles or columns, and are securely attached to the ground. The individual feature point sets4included in the feature point set cloud3have positional relationships corresponding to four vertex positions of virtual rectangles in the real space50, and are arranged away from each other by a predefined distance in the vertical direction and a predefined distance in the horizontal direction. Preferably, two sides constituting a virtual rectangle are parallel to each other in the perpendicular direction (vertical direction) and the horizontal direction. Note that the feature point set cloud3indicates a plurality of feature point sets4. The feature point set cloud3may indicate a plurality of feature point sets4that are locally present, or may indicate a plurality of feature point sets4included in a single photographed image photographed by the photographing device14. Furthermore, the MR system1may include a plurality of feature point set clouds3.

In this embodiment, the real space50is an area with a range of a few kilometers square, such as the whole area around Shibuya station.FIG. 4is a schematic illustration showing part of the real space50according to the embodiment of the present invention. The real space50includes a real object51such as a building.

A virtual space60is a virtual space associated with the real space50, and a position in the real space50and a position in the virtual space60are associated with each other on a one-to-one basis. The MR system1uses, in the virtual space60, three-dimensional space data representing the real object51present in the real space50. Thus, the three-dimensional space data defines the virtual space60associated with the real space50, and represents the three-dimensional shape of the real object51. The three-dimensional space data is obtained in advance, and each item of three-dimensional space data is constituted of a three-dimensional shape element, which is a basic unit having three-dimensional position information in the virtual space60. This basic unit will be referred to as a three-dimensional shape element. In this embodiment, a three-dimensional mesh is used as the three-dimensional shape element. Thus, the three-dimensional space data is constituted of the three-dimensional mesh. The three-dimensional mesh is formed of one or more polygons, and at least one of vertices of polygons constituting each three-dimensional mesh has the three-dimensional position information. However, the three-dimensional shape element may be point cloud data or the like.

The three-dimensional position information possessed by the three-dimensional shape element is position information in the real space50corresponding to position information in the virtual space60. Here, the position information in the real space50is, for example, a 6DoF (6 axes) value including a latitude, a longitude, an altitude, and a direction, and the position information in the virtual space60is also a corresponding 6DoF value.

Recently, three-dimensional mesh data of a relatively wide area, which is the area of interest for the real space50in this embodiment, is provided by various companies, and it is possible to obtain accurate three-dimensional mesh data relatively easily.

For example, three-dimensional mesh data as described above is obtained in advance as follows. First, a highly accurate laser scanner is used to obtain, in advance, point cloud data indicating the three-dimensional shape of the real object51in the real space50.FIG. 5shows an example of point cloud data in the virtual space60associated with the real space50. For example, each item of point cloud data has three-dimensional coordinates (x, y, z) and is arranged in the virtual space60associated with the real space50. Note that each item of point cloud data is color point cloud data including color information. Next, from the color point cloud data obtained as described above, polygons having the individual point clouds as vertices are formed by using a known method. A three-dimensional mesh is formed of the polygons. However, since a three-dimensional shape element constituting three-dimensional space data does not need to have color information, in the above-described example, the point cloud data need not have color information. In another example, a three-dimensional mesh is generated from a plurality of photographed images in the real space50by using a known method.

FIG. 5shows the virtual space60corresponding to the real space50shown inFIG. 4, and also shows a first virtual object61, which is a virtual object corresponding to the real object51present in the real space50. Although all of the objects arranged in the virtual space60are virtual objects, for convenience of description, a virtual object corresponding to the real object51will be referred to as the first virtual object61, and a virtual object that does not correspond to the real object51, i.e., a virtual object that is not present in the real space50will be referred to as the second virtual object62. Thus, three-dimensional space data is data of the first virtual object61, and the virtual space60is defined by the data of the first virtual object61. In this embodiment, for example, the coordinates of three-dimensional space data means the same as the coordinates of the virtual space60.

A position in the virtual space60is determined by using 6DoF including three-dimensional coordinates (x, y, z), as shown inFIG. 5. However, since the three-dimensional coordinates in the virtual space60shown inFIG. 5are an example, the origin of the three-dimensional coordinates may be set to any position, and individual coordinate axes may have any directions as long as the coordinate axes are orthogonal to each other. Alternatively, regarding a position in the virtual space60, a three-dimensional position may be determined by using other kinds of coordinate system.

FIG. 6is the functional block diagram of the mobile terminal device2according to the embodiment of the present invention. The mobile terminal device2includes an application unit21. The application unit21includes a recognition unit22, a position determination unit23, an image generation unit24, and an image display unit25. In this embodiment, these functions are realized by the execution of programs by the processor11. In this embodiment, since various functions are realized by loading programs, a portion of one part (function) may be included in another part. However, these functions may be realized with hardware by configuring an electronic circuit or the like for realizing a portion or the entirety of each of the functions.

On the mobile terminal device2, an OS is installed and an application running on the OS for the purpose of allowing a user to experience MR is also installed. The application unit21corresponds to this application and is stored in the storage device15as an application program.

The storage device15stores data of the first virtual object61, in the virtual space60, corresponding to the real object51present in the real space50, and data of the second virtual object62in the virtual space60. In this embodiment, the data of the second virtual object62is also formed of a three-dimensional mesh, and position information included in the data of the second virtual object is represented by the coordinates of three-dimensional space data or is associated with the coordinates of the three-dimensional space data. In one example, the second virtual object62is a virtual character or a virtual item.FIG. 7is an example of an image of the second virtual object62. In one example, the data of the second virtual object62is the data of a virtual object in which at least one of the shape, size, or orientation changes in accordance with at least one of position and time.

The storage device15stores feature point set position data in which the IDs of individual feature point sets4are associated with arrangement position information in the virtual space60, the arrangement position information corresponding to position information in the real space50in which the individual feature point sets4are arranged. The arrangement position information here is represented by the coordinates of three-dimensional space data or is associated with the coordinates of the three-dimensional space data. Furthermore, to be more precise, the arrangement position information is arrangement position and posture information and is, for example, a 6DoF (six degrees of freedom) value. Thus, for example, the arrangement position information (arrangement position and posture information) is position and posture information of the feature point sets4using a prescribed position of the first virtual object61(origin of the virtual space60) as the origin. Note that, similarly to the case where ArUco is generally used, the storage device15stores a software module for using ArUco, and the application unit21uses this module to associate the individual feature point sets4with the IDs.

In one example, the storage device15stores three-dimensional space data with a program for causing the three-dimensional space data to function as a three-dimensional space database. For example, in this case, the application unit21uses combinations of the IDs of the recognized individual feature point sets4to query a three-dimensional space database, thereby obtaining three-dimensional space data according to the combinations of the IDs.

The application unit21starts the photographing device14when an application is started by a user operation or the like. The recognition unit22recognizes individual feature point sets4photographed by the photographing device14. For example, the recognition unit22recognizes the feature point sets4on the basis of the outer edges of the feature point sets4and the arrangement positions of the white and black squares of the feature point sets4, and recognizes the IDs of the feature point set4. Preferably, the recognition unit22recognizes, at prescribed time intervals such as 0.5 s, the individual feature point sets4photographed by the photographing device14.

The position determination unit23determines a virtual camera viewpoint position in the virtual space60, the virtual camera viewpoint position corresponding to the position and photographing direction of the photographing device14in the real space50. For example, the virtual camera viewpoint position determined by the position determination unit23is represented by 6DoF (six degrees of freedom) in the virtual space60. For example, a virtual camera image angle is determined in accordance with the actual angle of view of the photographing device14.

In one example, the application unit21determines whether or not the number of feature point sets4photographed by the photographing device14is greater than or equal to a prescribed number, and in the case where it is determined that the number of the feature point sets4photographed by the photographing device14is greater than or equal to the prescribed number, the position determination unit23determines a virtual camera viewpoint position. For example, in the case where the application unit21determines that the recognition unit22has recognized three or more feature point sets4, the position determination unit23determines a virtual camera viewpoint position.

The position determination unit23obtains arrangement position information related to the arrangement of the feature point sets4by using the IDs of the recognized feature point sets4and feature point set position data stored in the storage device15. Furthermore, the position determination unit23determines relative position information indicating the positions of the mobile terminal device2relative to the recognized feature point sets4from the shapes and sizes of the feature point sets4. To be more precise, the relative position information is relative position and posture information, an example of which is information including 6DoF as a relative position, and the relative position information includes the three-dimensional position and facing direction of the mobile terminal device2(photographing device14) relative to the feature point sets4. Thus, for example, the relative position information (relative position and posture information) is the position and posture information of the feature point sets4using a virtual camera position as the origin. The position determination unit23obtains arrangement position information for the recognized individual feature point sets4and determines relative position information indicating the position of the mobile terminal device2relative to the feature point sets4. The position determination unit23determines a virtual camera viewpoint position (6DoF) on the basis of the obtained arrangement position information and the determined relative position information.

In one example, the position determination unit23individually calculates tentative virtual camera viewpoint positions (6DoF) by using the arrangement position information and relative position information of the recognized individual feature point sets4. The position determination unit23determines a virtual camera viewpoint position by using the average value of the plurality of calculated 6DoF values. Alternatively, the position determination unit23determines the virtual camera viewpoint position by using the median value of the plurality of calculated 6DoF values. Alternatively, the position determination unit23determines the virtual camera viewpoint position by inputting the plurality of calculated 6DoF values to an estimation model such as a particle filter (NPL 4).

In one example, instead of or in addition to the above-described example, the position determination unit23determines a virtual camera viewpoint position by using arrangement relationships among known feature point sets4. Here, since the individual feature point sets4are arranged away from each other by predefined distances in the vertical direction and the horizontal direction, it is possible to derive the positions of the other feature point sets4from the position of one feature point set4. In one example, the position determination unit23determines relative position information for each of the recognized feature point sets4, and then determines a first virtual rectangle constituted of the recognized feature point sets4. Meanwhile, the position determination unit23obtains arrangement position information for each of the recognized feature point sets4, and then determines a second virtual rectangle constituted of the obtained arrangement position information. The position determination unit23determines a virtual camera viewpoint position by using the arrangement position information of the second virtual rectangle and the shape and size of the first virtual rectangle with respect to the second virtual rectangle.

In another example, in the case where a tentative virtual camera viewpoint position is set as a 6DoF value calculated by using the arrangement position information and relative position information of one feature point set4, the position determination unit23calculates the difference between the positions of the other feature point sets4derived from the one feature point set4and the positions of the other feature point sets4identified from an image photographed by the photographing device14. The position determination unit23extracts, from among the calculated individual 6DoF values, 6DoF values the calculated difference of which is within a prescribed range, and determines a virtual camera viewpoint position by using the average value or median value of the 6DoF values or by inputting the 6DoF values to a known estimation model. Alternatively, the position determination unit23weights each of the calculated 6DoF values in accordance with the calculated difference, and determines the virtual camera viewpoint position by using the average value or median value of the weighted 6DoF values or inputting the 6DoF values to a known estimation model. Alternatively, the position determination unit23determines the virtual camera viewpoint position on the basis of the calculated 6DoF values, and then corrects the virtual camera viewpoint position by using the arrangement relationship of a known feature point set cloud3.

In one example, the position determination unit23determines the position of the mobile terminal device2relative to feature point sets4as follows. The sizes and shapes of the individual feature point sets4are defined in advance. By using a known method, the position determination unit23recognizes the feature point sets4as images, recognizes the outlines of the feature point sets4, and calculates a coordinate transformation matrix for performing transformation from a feature point set coordinate system to a camera coordinate system. By using the calculated coordinate transformation matrix, the position determination unit23calculates the positions and postures, in the camera coordinate system, of the feature point sets4in an image photographed by the photographing device14, and determines the positions (6DoF) of the feature point sets4relative to the photographing device14.

In one example, the position determination unit23determines the position of the mobile terminal device2relative to feature point sets4as follows. The position determination unit23uses a known method (NPLs 2 and 3) in which local features, which are robust features resistant to rotation and movement, are used to detect objects included in a captured image. In this example, the storage device15stores the template images of the individual feature point sets4in association with the IDs of the feature point sets4. The position determination unit23determines a relative position (6DoF) by using local features to derive projections (such as rotation, enlargement/reduction, and deformation) between the template images and an image photographed by the photographing device14. Here, the local features refer to patterns or conspicuous structures seen in an image, such as points, edges, or small image patches. Specifically, the projection is estimated by using four corner points of the template image and a homography matrix obtained by using the local features. The homography matrix refers to a parameter to be used when a plane is projected on another plane by using projection transformation.

Although a virtual camera viewpoint position is calculated and determined by using 6DoF, position information in other formats may also be used as long as equivalent information is included.

On the basis of data of the first virtual object61and data of the second virtual object62stored in the storage device15, as well as the virtual camera viewpoint position determined by the position determination unit23, the image generation unit24generates a mixed-reality image in which an image of the second virtual object62according to the viewpoint position is superimposed on a photographed image of the real space50. The image display unit25displays an application screen including the generated mixed-reality image on the display device13.

In one example, the image generation unit24reads data of the first virtual object61and data of the second virtual object62corresponding to the virtual camera viewpoint position determined by the position determination unit23, and generates a mixed-reality image in which an image of the second virtual object62according to the viewpoint position is superimposed on a photographed image of the real space50. When generating the mixed-reality image, a known method as indicated in NPL 1 or NPL 5 may be used.

FIG. 8is an example of a mixed-reality image in which the image of the second virtual object62shown inFIG. 7is superimposed on a photographed image of the real space50. In the case where the first virtual object61and the second virtual object62overlap each other in the field-of-view range of a virtual camera, if the depth distance of the second virtual object62from the virtual camera viewpoint position is larger than that of the first virtual object61, the image generation unit24generates a mixed-reality image in which the image of the second virtual object62without the overlapping parts is superimposed. For example, the image generation unit24performs rendering processing of the first virtual object61as an invisible mesh, and performs normal rendering processing for the second virtual object62. By doing so, for the above-described overlapping parts, the invisible rendering processing for the first virtual object61, which is closer to the virtual camera viewpoint position than the second virtual object62is, is performed. Note that the field-of-view range of the virtual camera is determined in accordance with the image angle of the photographing device14.

FIG. 8shows the second virtual object62, part of which is not displayed. InFIG. 8, it can be seen that the second virtual object62is displayed such that the second virtual object62exists behind a building, which is the real object51.

In one example, the image generation unit24generates an MR moving picture by generating a mixed-reality image at a prescribed frame rate. In one example, in the case where the photographing device14does not recognize feature point sets4or in the case where a virtual camera viewpoint position is not determined, the image display unit25directly displays a photographed image of the real space50as a mixed-reality image.

In this embodiment, due to the operations of the individual parts of the application unit21, the MR system1needs to include at least three feature point sets4in a single photographed image photographed by the photographing device14. The vertical direction distance and horizontal direction distance between two feature point sets4, among the feature point sets4included in a feature point set cloud3, corresponding to adjacent vertex positions of a virtual rectangle are determined on the basis of the image angle of the photographing device14when away from the feature point set cloud3by a predetermined distance.

FIG. 9is a diagram for explaining positional relationships among the mobile terminal device2in the MR system1and feature point sets4included in a feature point set cloud3according to the embodiment of the present invention. InFIG. 9, the distance h represents a distance between feature point sets4fixed to the rod-like body41, and the distance w represents a distance between feature point sets4fixed to the upper sides of a pair of the rod-like bodies41or a distance between feature point sets4fixed to the lower sides of the rod-like bodies41. The distance d represents a distance from the feature point set cloud3to the photographing device14(mobile terminal device2). The distance between feature point sets4is, for example, a distance between the centers of the feature point sets4, and the distance from the feature point set cloud3is, for example, a distance from a virtual rectangle (plane) constituted of the feature point sets4included in the feature point set cloud3. For example, the distance d is a distance from a user to the feature point set cloud3, which is recommended by the MR system1to the user who wishes to have an MR experience.

In one example, in the case where the vertical image angle of the photographing device14is a degrees and the horizontal image angle of the photographing device14is β degrees, the maximum value hmaxof the distance h and the maximum value wmaxof the distance w are determined by the following formulas, respectively.

By arranging the individual feature point sets4such that h≤hmaxand w≤wmax, when the photographing device14is away from the feature point set cloud3by the distance d, it becomes possible for the photographing device14to capture, in a single photographed image, the feature point sets4included in the feature point set cloud3. In order for the position determination unit23to calculate a position more accurately, distances between the feature point sets4are preferably made larger, and thus, it is preferred to arrange the individual feature point sets4such that h=hmaxand w=wmax. In one example, in the case where d=2 m, α=63 degrees, and β=50 degrees, h=1.2 m and w=0.9 m.

Next, information processing in the MR system1according to the embodiment of the present invention will be described by using the flowchart shown inFIG. 10. The information processing indicated inFIG. 10is realized by causing the mobile terminal device2to execute a program. In this processing, a user having the mobile terminal device2is present in the real space50, and an application including the function of the application unit21is started by the user. In one example, while the application is running, the mobile terminal device2regularly executes the processing in this flowchart.

In step101, the mobile terminal device2determines whether or not it is possible to recognize three or more feature point sets4from a photographed image photographed by the photographing device14of the mobile terminal device2. In the case where three or more feature point sets4are recognized, this processing proceeds to step102, and in the case where less than three feature point sets4are recognized, this processing is terminated.

Next, in step102, the mobile terminal device2obtains arrangement position information related to the arrangement of the individual feature point sets4by using the individual IDs of the recognized feature point sets4and feature point set position data stored in the storage device15. The mobile terminal device2determines relative position information indicating the positions of the mobile terminal device2relative to the individual feature point sets4from the shapes and sizes of the recognized feature point sets4.

Next, in step103, the mobile terminal device2determines a virtual camera viewpoint position on the basis of each item of the arrangement position information and relative position information for the recognized individual feature point sets4.

Next, in step104, the mobile terminal device2determines whether or not the second virtual object62is present within a virtual camera field-of-view range. In the case where the second virtual object62is present, this processing proceeds to step105, and in the case where the second virtual object62is not present, this processing is terminated. In step105, on the basis of data of the first virtual object61, data of the second virtual object62, and the determined virtual camera viewpoint position, the mobile terminal device2generates a mixed-reality image in which an image of the second virtual object62according to the viewpoint position is superimposed on a photographed image of the real space50. In the case where this processing is terminated, the mobile terminal device2directly displays the photographed image of the real space50as the mixed-reality image.

Next, the operation and advantages of the MR system1(mobile terminal device2) according to the embodiment of the present invention will be described. In this embodiment, the MR system1includes a plurality of feature point sets4in the prescribed real space50, and the individual feature point sets4are arranged at four vertex positions of a virtual rectangle in the real space50away from each other by a predefined distance in the vertical direction and a predefined distance in the horizontal direction. Furthermore, the storage device15stores three-dimensional space data representing the real object51present in the real space50and defining the virtual space60, and stores arrangement position information of the individual feature point sets4, the arrangement position information being associated with the three-dimensional space data.

By recognizing the feature point sets4, the mobile terminal device2obtains the arrangement position information of the recognized individual feature point sets4. Meanwhile, the mobile terminal device2determines relative position information (6DoF) indicating the positions of the mobile terminal device2relative to the individual feature point sets4from the shapes and sizes of the recognized individual feature point sets4in a photographed image. The mobile terminal device2determines a camera viewpoint position (6DoF) on the basis of the arrangement position information and relative position information of the recognized individual feature point sets4, and generates a mixed-reality image with respect to the photographed image of the real space50according to the virtual camera viewpoint position.

As described above, in the MR system1, the position of the real object51is associated with the positions of the feature point sets4by using the coordinates of the three-dimensional space data (virtual space60), and thus, by recognizing the feature point sets4, it is possible for the mobile terminal device2to obtain the arrangement position information of the recognized feature point sets4in the virtual space60. Furthermore, in this embodiment, by using the arrangement position information and the relative position information of the mobile terminal device2with respect to the feature point sets4, it is possible to determine a virtual camera viewpoint position (6DoF) corresponding to the photographing device14(self position) being operated by a user.

Furthermore, in this embodiment, after calculating individual tentative virtual camera viewpoint positions (6DoF) from the individual feature point sets4, the mobile terminal device2determines a virtual camera viewpoint position on the basis of the calculated individual 6DoF values. The accuracy of the virtual camera viewpoint position is very important in generating a mixed-reality image. By using such a configuration, in this embodiment, it becomes possible to determine the virtual camera viewpoint position more accurately. Furthermore, in this embodiment, the mobile terminal device2determines the virtual camera viewpoint position by further using the arrangement relationship of a known feature point set cloud3. By doing so, it becomes possible to determine the virtual camera viewpoint position more accurately. In this case, when the mobile terminal device2determines the virtual camera viewpoint position, in order to execute operations more accurately and quickly, it is preferred that the adjacent feature point sets4constituting the feature point set cloud3be substantially parallel in the perpendicular direction (vertical direction) and the horizontal direction.

Furthermore, in this embodiment, when generating a mixed-reality image, the mobile terminal device2performs rendering processing as an invisible mesh for the first virtual object61, and performs normal rendering processing for the second virtual object62. By using such a configuration, in this embodiment, it becomes possible to realize higher geometric consistency (front/rear relationship, occlusion, etc.) with the real object51when displaying the second virtual object62. The geometric consistency in the mixed-reality image is extremely important when reality is applied to the second virtual object62.

The above-described operation and advantages similarly apply to other embodiments or other examples unless explicitly mentioned otherwise.

The present invention, in another embodiment thereof, may also be a program for realizing the functions or information processing shown in the flowchart of the above-described embodiment of the present invention, or a computer-readable storage medium storing the program. The present invention, in yet another embodiment thereof, may also be a method that realizes the functions or information processing shown in the flowchart of the above-described embodiment of the present invention. The present invention, in yet another embodiment thereof, may also be a server that can provide a computer with a program for realizing the functions or information processing shown in the flowchart of the above-described embodiment of the present invention. The present invention, in yet another embodiment thereof, may also be a virtual machine that realizes the functions or information processing shown in the flowchart of the above-described embodiment of the present invention. The present invention, in yet another embodiment thereof, may also be an arrangement structure of feature point sets4in the MR system1.

Modifications of the embodiment of the present invention will be described below. The modifications that will be described below may be combined, as appropriate, and applied to an arbitrary embodiment as long as no inconsistency arises.

In one modification, feature point sets4are two-dimensional codes or include two-dimensional codes. For example, the feature point sets4are QR codes (registered trademark) or data matrices. In yet another modification, two-dimensional codes serving as the feature point sets4include, as identifiable information, identifiers indicating a latitude, a longitude, an angle, etc. In this case, it is possible for the mobile terminal device2that has recognized the two-dimensional codes to obtain arrangement position information of the feature point sets4directly from the obtained identifiable information. In this case, the storage device15need not store feature point set position data.

In one modification, the feature point sets4are known AR markers.

In one modification, posters, paintings, pictures, etc. having spatial distribution of different feature points that are individually identifiable are used as the feature point sets4. The recognition unit22and the position determination unit23use the above-described known method (NPLs 2 and 3) using local features. In this case, the local features serve as the identifiable information. In this modification, the storage device15stores the template images of the individual feature point sets4in association with the IDs of the feature point sets4. The recognition unit22recognizes the IDs of the individual feature point sets4by extracting the local features from the feature point sets4photographed by the photographing device14and comparing the local features with the template images of the individual feature point sets4. The position determination unit23obtains arrangement position information related to the arrangement of the feature point sets4by using the IDs of the recognized feature point sets4and feature point set position data stored in the storage device15. Furthermore, the position determination unit23determines relative position information of the mobile terminal device2with respect to the feature point sets4by using local features to derive projections (such as rotation, enlargement/reduction, and deformation) between the template images and images photographed by the photographing device14. For each of the recognized feature point sets4, the position determination unit23obtains the arrangement position information and determines the relative position information, and determines a virtual camera viewpoint position on the basis of the arrangement position information and the relative position information.

In one modification, a feature point set cloud3includes three feature point sets4arranged to have predefined positional relationships. For example, two of the feature point sets4are securely attached to one rod-like body41of a pair of rod-like bodies41installed in the real space50in a horizontally-isolated manner, and one of the feature point sets4is securely attached to the other rod-like body41. In this embodiment, by including the three feature point sets4having predefined positional relationships in the feature point set cloud3, it becomes possible for the position determination unit23to determine a virtual camera viewpoint position by using the arrangement relationship of the known feature point set cloud3. In this case, the three feature point sets4are preferably arranged to have positional relationships corresponding to the positions of three vertices among four vertices of a virtual rectangle in the real space50.

In one modification, the MR system1includes a plurality of feature point sets4each of which is securely attached to each of three or more rod-like bodies41installed in the real space50in a horizontally-isolated manner. To each of the rod-like bodies41, a plurality of the feature point sets4are securely attached in a vertically-isolated manner. However, only one of the feature point sets4may be attached to some of the rod-like bodies41.

In one modification, each of the feature point sets4is securely attached to a transmission member42formed of a light transmitting material. The transmission member42is, for example, a glass or resin that transmits almost the entire range of visible light and may be part of a structure such as window glass or may be portable, such as a glass plate.FIG. 11is a diagram showing a feature point set cloud3securely attached to window glass.

In one modification, each of the feature point sets4is securely attached by using a wire. However, the feature point sets4may be fixed in the real space50by using members having functions equivalent to wires.

In one modification, the MR system1includes a feature point set cloud3including five or more feature point sets4. For example, the feature point set cloud3is constituted of twenty feature point sets4. This modification assumes the case where the photographing device14cannot capture, in a single photographed image, all of the feature point sets4included in the feature point set cloud3.

The recognition unit22recognizes, at prescribed time intervals, the individual feature point sets4included in an image photographed by the photographing device14. The position determination unit23determines, at prescribed time intervals, a camera viewpoint position (6DoF) on the basis of the arrangement position information and relative position information of the recognized individual feature point sets4. On the basis of data of the first virtual object61and data of the second virtual object62, as well as the virtual camera viewpoint position determined by the position determination unit23at prescribed time intervals, the image generation unit24generates a mixed-reality image with respect to the photographed image of the real space50according to the virtual camera viewpoint position.

As described above, after the photographing device14recognizes the plurality of feature point sets4, when the photographing position of the photographing device14is moved by the movement of a user, the mobile terminal device2recognizes a plurality of other feature point sets4. By using such a configuration, it becomes possible for the mobile terminal device2to perform “roaming processing” in which calibration is performed while sequentially switching among a plurality of feature point sets4as the user moves in the real space50. Thus, in this modification, it is possible for the mobile terminal device2to consecutively calculate a self position (6DoF) even in the case where a plurality of feature point sets4photographed by the photographing device14change in accordance with time. This makes it possible to continuously provide the user moving in the real space50with an MR experience.

In this modification, the position determination unit23may determine a camera viewpoint position (6DoF) on the basis of the arrangement position information and relative position information for each of some feature point sets4of the recognized feature point sets4. For example, the camera viewpoint position (6DoF) may be determined on the basis of the arrangement position information and relative position information for each of the feature point sets4, in a photographed image, having a size larger than or equal to a prescribed size.

In one modification, as shown inFIG. 11, the MR system1includes a plurality of feature point set clouds3including a first feature point set cloud3a, a second feature point set cloud3b, a third feature point set cloud3c, and a fourth feature point set cloud3d. In this modification, although the individual feature point set clouds3include mutually different feature point sets4, a plurality of the feature point set clouds3include the same feature point sets4. For example, the first feature point set cloud3aincludes feature point sets4included in the second feature point set cloud3band feature point sets4not included in the second feature point set cloud3b.

The recognition unit22recognizes, at prescribed time intervals, the individual feature point sets4included in an image photographed by the photographing device14. The application unit21selects one feature point set cloud3of the recognized feature point set clouds3as a reference feature point set cloud3. The position determination unit23determines relative position information for each of the recognized feature point sets4included in the reference feature point set cloud3, and then determines a first virtual rectangle constituted of the recognized feature point sets4. Meanwhile, the position determination unit23obtains arrangement position information for each of the recognized feature point sets4, and then determines a second virtual rectangle constituted of the obtained arrangement position information. The position determination unit23determines a virtual camera viewpoint position by using the arrangement position information of the second virtual rectangle and the shape and size of the first virtual rectangle with respect to the second virtual rectangle.

In one example, each time the recognition unit22recognizes each feature point set4, the application unit21selects a reference feature point set cloud3. In one example, the application unit21selects, as the reference feature point set cloud3, a feature point set cloud3having more feature point sets4included in a photographed image. In one example, instead of or in addition to the above-described selection method, the application unit21selects, as the reference feature point set cloud3, a feature point set cloud3having more feature point sets4included in the center part of the photographed image. In one example, instead of or in addition to the above-described selection method, the application unit21selects, as the reference feature point set cloud3, a feature point set cloud3having a higher ratio of large-sized feature point sets4included in the photographed image.

In this way, after the photographing device14recognizes feature point sets4included in the reference feature point set cloud3, when the photographing position of the photographing device14is moved by the movement of a user, the mobile terminal device2recognizes feature point sets4included in another reference feature point set cloud3. By using such a configuration, it becomes possible for the mobile terminal device2to perform “roaming processing” in which calibration is performed while sequentially switching a feature point set cloud3as the user moves in the real space50. Thus, in this modification, it is possible for the mobile terminal device2to consecutively calculate a self position (6DoF) even in the case where a plurality of feature point sets4photographed by the photographing device14change with time. This makes it possible to continuously provide the user moving in the real space50with an MR experience.

Note that one feature point set cloud3may include six feature point sets4or may include nine feature point sets4.

In one modification, the mobile terminal device2(position determination unit23) fixes a virtual camera viewpoint position to a prescribed position (for example, predefined 6DoF) in the virtual space60and dynamically determines the position of the coordinates of three-dimensional space data. The mobile terminal device2determines three-dimensional position information of the first virtual object61instead of determining a camera viewpoint position. For example, the mobile terminal device2determines the coordinates of the three-dimensional space data. Accordingly, the coordinates of the feature point sets4and the coordinates of the second virtual object62are changed.

In one modification, arrangement position information (arrangement position and posture information) or relative position information (relative position and posture information) uses a quaternion value instead of a 6DoF value.

In one modification, as shown inFIG. 12, the MR system1is provided with a plurality of mobile terminal devices2and a server6, and the mobile terminal devices2and the server6are connected to a network8such as the Internet and are able to carry out communication with each other. The server6is provided with a configuration provided in an ordinary server. For example, the server6is provided with a processor, such as a CPU, that controls individual components, a storage element such as a ROM or a RAM, a storage device such as a hard disk drive, a display device such as a display, an input device such as a mouse, a keyboard, or a touchscreen, and a communication device such as a network board. These constituent elements are connected via a bus.

The storage device provided in the server6stores data of the first virtual object61(three-dimensional space data) and data of the second virtual object62. The storage device15of the mobile terminal device2need not store the data stored in the server6. For example, the server6stores three-dimensional space data with a program for causing the three-dimensional space data to function as a three-dimensional space database, and transmits the three-dimensional space data to the mobile terminal device2in response to a query from the mobile terminal device2. The storage device provided in the server6may store feature point set position data on behalf of the storage device15of the mobile terminal device2.

In yet another modification, the operations of the mobile terminal device2of the MR system1are realized by a Web application provided by the server6. For example, the server6is provided with all of the functions of the application unit21, and stores various kinds of data used in the operation of the application unit21. In this case, the server6provides the mobile terminal device2with a Web application corresponding to the application unit21. When the Web application installed in the mobile terminal device2is started, the mobile terminal device2accesses the Web application on the server6. The mobile terminal device2realizes the functions of the application unit21on the mobile terminal device2by communicating with the server6.

In the processing or operation described above, the processing and operation may be modified freely unless any inconsistency arises in the processing or operation, such as an inconsistency that a step involves the use of data that could not yet be available in that step. Furthermore, the embodiments described above are examples for explaining the present invention, and the present invention is not limited to these embodiments. The present invention can be embodied in various forms not departing from the gist thereof.

REFERENCE SIGNS LIST