Display control device, display control method, and recording medium

There is provided a display control device including a display controller configured to place a virtual object within an augmented reality space corresponding to a real space in accordance with a recognition result of a real object shown in an image captured by an imaging part, and an operation acquisition part configured to acquire a user operation. When the user operation is a first operation, the display controller causes the virtual object to move within the augmented reality space.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority Patent Application JP 2013-068395 filed Mar. 28, 2013, the entire contents of which are incorporated herein by reference.

BACKGROUND

The present disclosure relates to a display control device, a display control method, and a recording medium.

A technology called augmented reality (AR) has recently been drawing attention, which shows a user a real space having additional information superimposed thereover. The information shown to the user in the AR technology may be visualized using various forms of virtual objects such as text, icons, or animation. The placement of annotation over an AR space is generally executed on the basis of recognition in three-dimensional structure in the real space shown in an image.

A structure from motion (SfM) technique and a simultaneous localization and mapping (SLAM) technique are known as techniques for recognizing a three-dimensional structure in the real space. In the SfM technique, multiple images are captured from different viewpoints, and, from those images, a three-dimensional structure in the real space shown in the images is recognized using parallax. The SLAM technique is described in Andrew J. Davison, “Real-Time Simultaneous Localization and Mapping with a Single Camera”, Proceedings of the 9th IEEE International Conference on Computer Vision Volume 2, 2003, pp. 1403-1410. JP 2009-237845A discloses a technique for recognizing three-dimensional positions of feature points, which are selected for initialization in the SLAM technique, by using the SfM technique.

SUMMARY

Once a virtual object is placed in an AR space, the virtual object generally maintains a state that is determined in advance independent of a user's intention. However, there is a case where the user wants to change the state of the virtual object after the placement of the virtual object. For example, there is a case the where the user wants to change the position of the virtual object in the AR space after the placement of the virtual object.

In light of the foregoing, it is desirable in the present disclosure to provide technology capable of changing a position of a virtual object placed in an AR space in accordance with a user's intention.

According to an embodiment of the present disclosure, there is provided a display control device which includes a display controller configured to place a virtual object within an augmented reality space corresponding to a real space in accordance with a recognition result of a real object shown in an image captured by an imaging part, and an operation acquisition part configured to acquire a user operation. When the user operation is a first operation, the display controller causes the virtual object to move within the augmented reality space.

According to another embodiment of the present disclosure, there is provided a display control method which includes placing a virtual object within an augmented reality space corresponding to a real space in accordance with a recognition result of a real object shown in an image captured by an imaging part, acquiring a user operation, and causing the virtual object to move within the augmented reality space when the user operation is a first operation.

According to another embodiment of the present disclosure, there is provided a non-transitory computer-readable recording medium having a program recorded thereon, the program being for causing a computer to function as a display control device including a display controller configured to place a virtual object within an augmented reality space corresponding to a real space in accordance with a recognition result of a real object shown in an image captured by an imaging part, and an operation acquisition part configured to acquire a user operation. When the user operation is a first operation, the display controller causes the virtual object to move within the augmented reality space.

According to one or more of embodiments of the present disclosure, it is possible to change a position of a virtual object placed in an AR space in accordance with a user's intention.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Further, in this specification and the appended drawings, there are some cases where multiple structural elements that have substantially the same function and structure are distinguished from one another by being denoted with different alphabets or numbers after the same reference numeral. Note that, in the case where it is not necessary to distinguish the multiple structural elements that have substantially the same function and structure from one another, the multiple structural elements are denoted with the same reference numeral only.

Further, the “detailed description of the embodiments” will be described in the following order.1. Embodiment1-1. Overview of display control device1-2. Functional configuration example of display control device1-3. Initial display of virtual object1-4. Position/attitude control on virtual object1-5. Size control on virtual object1-6. Hardware configuration example2. Conclusion

1-1. Overview of Display Control Device

First, an overview of a display control device10according to an embodiment of the present disclosure will be described.FIG. 1is a diagram illustrating an overview of the display control device10according to an embodiment of the present disclosure. Referring toFIG. 1, there is shown the display control device10held by a user Ua. The display control device10includes an imaging part120, which is directed towards a real space1, an operation part140, and a display part160. The imaging part120generates an image by capturing the real space1.

In the example shown inFIG. 1, the display part160displays an image Im1captured by the imaging part120. The user Ua is capable of grasping the real space1by placing a viewpoint on the image Im1displayed by the display part160. However, the image Im1may not necessarily be displayed on the display part160. For example, in the case where the display part160is a transmissive head mounted display (HMD), the display part160does not display the image Im1, and the user Ua may place the viewpoint directly on the real space1instead of the image Im1.

Further, a real object A1is shown in the image Im1. For example, when the real object A1is recognized from the image Im1, the display control device10places a virtual object in an AR space corresponding to the real space1on the basis of the recognition result of the real object A1. In this way, the user Ua can view the virtual object placed in the AR space by the display control device10via the display part160. The real object A1may be recognized by the display control device10, or may be recognized by a device (for example, server) that is different from the display control device10.

Here, after the virtual object is placed in the AR space, the virtual object generally maintains a state that is determined in advance independent of the user's intention. However, there is a case where the user Ua wants to change the state of the virtual object after the placement of the virtual object. In light of the foregoing, the present disclosure proposes technology capable of changing a state of a virtual object placed in an AR space in accordance with a user's intention.

Note that, although description below will be made as an example of the case where the display control device10is employed as a camera-equipped smartphone, the display control device10may also be employed as a device other than a smartphone. For example, the display control device10may be employed as a video camera, a digital camera, a personal digital assistant (PDA), a personal computer (PC), a mobile phone, a mobile music playback device, a mobile video processing device, a mobile game console, a telescope, or a binocular.

Heretofore, an overview of a display control device according to an embodiment of the present disclosure has been described.

1-2. Functional Configuration Example of Display Control Device

Subsequently, a functional configuration example of the display control device10according to an embodiment of the present disclosure will be described.FIG. 2is a diagram showing a functional configuration example of the display control device10according to an embodiment of the present disclosure. As shown inFIG. 2, the display control device10includes a controller110, the imaging part120, a sensor part130, the operation part140, a storage150, and the display part160.

The controller110corresponds to, for example, a processor such as a central processing unit (CPU) or a digital signal processor (DSP). The controller110exhibits various functions that the controller110has by executing a program stored in the storage150or another storage medium. The controller110has functional blocks such as an operation acquisition part111, a sensor data acquisition part112, an image recognition part113, an environment recognition part114, and a display controller115. The functions of the respective functional blocks will be described later.

The imaging part120is a camera module that captures an image. The imaging part120captures a real space using an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and generates an image. The image generated by the imaging part120is output to the controller110. Note that, although the imaging part120is provided in an integrated manner with the display control device10in the example shown inFIG. 2, the imaging part120may be provided separately from the display control device10. For example, an imaging device connected to the display control device10via wire or radio may be used as the imaging part120.

The sensor part130acquires sensor data. For example, the sensor part130includes a 3-axis acceleration sensor. The 3-axis acceleration sensor measures gravitational acceleration applied to the imaging part120, and generates sensor data (acceleration data) that shows the size and the direction of the gravitational acceleration in three dimensions. Additionally, the sensor part130may include a geomagnetic sensor. The geomagnetic sensor generates sensor data (geomagnetic data) showing the direction of geomagnetism of the imaging part120in a coordinate system. Further, the sensor part130may also include a positioning sensor (for example, global positioning system (GPS) sensor). The positioning sensor generates sensor data (positioning data) showing the latitude and the longitude of the display control device10in the real space. Note that, although the sensor part130is provided in an integrated manner with the display control device10in the example shown inFIG. 2, the sensor part130may be provided separately from the display control device10.

The operation part140detects an operation performed by a user and outputs the operation to the controller110. In the present specification, since a case is assumed where the operation part140is formed of a touch panel, the operation performed by the user corresponds to an operation of tapping the touch panel. However, the operation part140may also be formed of hardware other than a touch panel (for example, button). Note that, although the operation part140is provided in an integrated manner with the display control device10in the example shown inFIG. 2, the operation part140may be provided separately from the display control device10.

The storage150uses a recording medium such as semiconductor memory or a hard disk to store a program for causing the controller110to operate. Further, for example, the storage150can also store various types of data (for example, various types of sensor data and virtual objects) used by the program. Note that, although the storage150is provided in an integrated manner with the display control device10in the example shown inFIG. 2, the storage150may be provided separately from display control device10.

The display part160displays various types of information in accordance with the control performed by the display controller115. For example, the display part160displays an image of an AR application generated by the display control device10. The display part160is formed of, for example, a liquid crystal display (LCD) or an organic electroluminescence (EL) display device. Note that, although the display part160is provided in an integrated manner with the display control device10in the example shown inFIG. 2, the display part160may be provided separately from the display control device10. For example, a display device connected to the display control device10via wire or radio may be used as the display part160.

Heretofore, a functional configuration example of the display control device10according to an embodiment of the present disclosure has been described.

From the next section onwards, the description of the functions that the display control device10according to an embodiment of the present disclosure has will be continued in the following order: “Initial display of virtual object”; “Position/attitude control on virtual object”; and “Size control on virtual object”. Note that all the functions described in the respective sections of “Initial display of virtual object”, “Position/attitude control on virtual object”, and “Size control on virtual object” may be used in combination, or only some of the functions may be used in combination.

1-3. Initial Display of Virtual Object

First, an initial display of a virtual object will be described. Referring toFIG. 3, in an image Im2captured by the imaging part120, there is shown a real object A1. Further, the real object A1is recognized by the image recognition part113, and a virtual object V1associated with the recognition result is placed in an AR space corresponding to the real space by the display controller115. Accordingly, the display part160displays the virtual object V1placed in the AR space.

In more detail, when the image recognition part113recognizes the position and the attitude of the real object A1, the display controller115identifies the position of the virtual object V1in accordance with the position of the real object A1, also identifies the attitude of the virtual object V1in accordance with the attitude of the real object A1, and places the virtual object V1in accordance with the identified position and attitude. The relationship between the position of the real object A1and the position of the virtual object V1may be determined in advance. Further, the relationship between the attitude of the real object A1and the attitude of the virtual object V1may also be determined in advance.

For example, the image recognition part113checks a partial image included in the image Im2against patches of respective feature points included in feature data, and detects feature points included in the image Im2. In the case where the feature points belonging to the real object A1are detected in high density in a region within the image Im2, the image recognition part113may recognize that the real object A1is shown in the region. The image recognition part113may further recognize the position and the attitude of the recognized real object A1on the basis of positional relationship between the detected feature points and three-dimensional shape data.

In the example shown inFIG. 3, when the user Ua visits an aquarium, there is the real object A1at the back wall surface of a water tank. When the user Ua holds the imaging part120over the real object A1and the image recognition part113recognizes the real object A1, a shark serving as an example of the virtual object V1associated with the recognition result is placed in the AR space corresponding to the real space by the display controller115. However, the virtual object V1may be any virtual object other than the shark.

Further, in the example shown inFIG. 3, the display controller115causes the size of the virtual object V1in the AR space to be displayed as a size M1. The size of the virtual object V1in the AR space may be determined by a technique to be described later. Further, the display controller115causes the ratio of the current size of the virtual object V1in the AR space to the real size of the virtual object V1to be displayed as a ratio N1. For example, the real size of the virtual object V1may also be registered in advance. Note that the real size of the virtual object means the size of the real object corresponding to the virtual object. In the case where a shark is assumed as the virtual object, the real size of the virtual object means the size of the shark as the real object.

Here, for example, in the case where the attitude of the real object A1is not normal, it is assumed that the attitude of the virtual object V1placed in the AR space is not normal. For example, let us assume the case where the relationship between the attitude of the real object A1and the attitude of the virtual object V1is determined such that the attitude of the virtual object V1is rendered normal when the real object A1is placed on the horizontal plane. In this case, as shown inFIG. 3, it can be expected that the attitude of the virtual object V1becomes not normal in the case where the real object A1is present on the wall surface.

Accordingly, this section proposes technology for rendering the initial display of the virtual object V1normal, independently of whether the attitude of the real object A1is normal.

As shown inFIG. 4, let us assume that a real object A0is placed in a normal attitude. In this case, the attitude of a virtual object V0placed in the AR space becomes normal in accordance with the position and the attitude of the real object A0. On the other hand, let us assume that a real object A1is placed in an attitude that is not normal as described above. In this case, the attitude of a virtual object V1placed in the AR space becomes not normal in accordance with the position and the attitude of the real object A1.

In such a case, for example, the display controller115may place a virtual object V2in a manner that the attitude of the virtual object V2becomes an attitude corresponding to a gravity vector G. Regarding the gravity vector G, when the sensor part130detects acceleration data, the acceleration data may be acquired as the gravity vector G by the sensor data acquisition part112. For example, if a relationship that is to be satisfied between the direction indicated by the gravity vector G and the attitude of the virtual object V1is determined in advance, the display controller115may rotate the virtual object V1so as to satisfy the relationship.

Referring toFIG. 4, there is shown, as the virtual object V2, the result obtained by rotating the virtual object V1such that the relationship that is to be satisfied between the direction indicated by the gravity vector G and the attitude of the virtual object V1is satisfied. Further, referring toFIG. 5, an image Im3is captured, and there is shown, as the virtual object V2, the result obtained by rotating the virtual object V1such that the relationship is satisfied. In this way, by changing the attitude of the virtual object V1with the gravity vector G taken into account, it becomes possible to place the virtual object V1in the AR space such that the virtual object V1has a normal attitude.

In more detail, the display controller115may grasp what attitude the real object A1is in based on the relationship between the direction indicated by the gravity vector G and the opposite vector of the normal vector of the real object A1, and may determine a degree of rotation of the virtual object V1in accordance with the attitude of the real object A1.

For example, in the case where the angle between the direction indicated by the gravity vector G and the opposite vector of the normal vector of the real object A1is more than or equal to 0 degree and less than 45 degrees (or less than or equal to 45 degrees), the display controller115may determine that the real object A1is placed on a floor surface. In such a case, when the virtual object V1is placed in the normal vector direction of the real object A1, it is not necessary that the display controller115rotate the virtual object V1.

Further, for example, in the case where the angle between the direction indicated by the gravity vector G and the opposite vector of the normal vector of the real object A1is more than or equal to 45 degrees (or more than 45 degrees) and less than 135 degrees, the display controller115may determine that the real object A1is pasted on a wall surface. In such a case, when the virtual object V1is placed in the normal vector direction of the real object A1, the display controller115may rotate the virtual object V190 degrees in the direction indicated by the gravity vector G.

Further, for example, in the case where the angle between the direction indicated by the gravity vector G and the opposite vector of the normal vector of the real object A1is more than or equal to 135 degrees (or more than 135 degrees) and less than or equal to 180 degrees, the display controller115may determine that the real object A1is pasted on a ceiling. In such a case, when the virtual object V1is placed in the normal vector direction of the real object A1, the display controller115may rotate the virtual object V1180 degrees in the direction indicated by the gravity vector G.

FIG. 6is a flowchart showing an operation example of an initial display of the virtual object V2. First, when the imaging part120captures an image, the image recognition part113recognizes the real object A1from the image captured by the imaging part120(S11). The image recognition part113calculates the position and the attitude of the real object A1(S12). Further, the sensor data acquisition part112acquires sensor data detected by the sensor part130(S13), and identifies a gravity vector on the basis of the sensor data (S14). For example, in the case where acceleration data is acquired as the sensor data, the acceleration data may be identified as the gravity vector.

The display controller115identifies the position of the virtual object V1in accordance with the position of the real object A1(S15). Subsequently, the display controller115identifies the attitude of a virtual object in accordance with the gravity vector (S16). The display controller115places the virtual object V2in the AR space on the basis of the identified position and attitude (S17). In the case where the recognition of real object A1performed by the image recognition part113is not continued (“No” in S18), the controller110may complete the operation, and in the case where the recognition of the real object A1performed by the image recognition part113is being continued (“Yes” in S18), the processing may return to step S11.

Heretofore, “Initial display of virtual object” has been described.

1-4. Position/Attitude Control on Virtual Object

Subsequently, position/attitude control on a virtual object will be described. Referring toFIG. 7, the display controller115places the virtual object V2in an AR space2. Here, as shown inFIG. 7, the position and attitude of the imaging part120are represented by P1and Q1, respectively, and the position and the attitude of the virtual object V2are represented by p1and q1, respectively. Let us assume that, in this state, the position and the attitude of the imaging part120are changed to P2and Q2, respectively. In this case, the virtual object V2is generally fixed within the AR space. However, there is a case where the user Ua wants to change the position of the virtual object V2in the AR space after the placement of the virtual object V2.

For example, assuming a scene where the user Ua is about to take a photograph of the virtual object V2, there may be a case where the user Ua wants to move the virtual object V2and decides a background, and then take a photograph of the virtual object V2. Further, for example, in the case where the user Ua does not want to include the real object A1within an imaging range, a case is assumed where the user Ua wants to take a photograph of the virtual object V2after moving the virtual object V2to a position such that the real object A1is out of the imaging range.

Accordingly, this section proposes technology for making it possible to change the position of the virtual object V2placed in the AR space in accordance with a user's intention.

First, description will be made of the case where the position of the virtual object V2placed in the AR space is not changed. For example, the display controller115may fix the virtual object V2within the AR space while there is no first operation being performed. The first operation may be any operation, and may be an operation of specifying the virtual object V2. Hereinafter, description will be made as an example of the case where the operation of specifying the virtual object V2is used as the first operation. The operation of specifying the virtual object V2may include, for example, an operation of specifying the virtual object V2using one or more operating objects.

The operation of specifying the virtual object V2may be an operation of tapping the virtual object V2. Further, the operation of specifying the virtual object V2may be, in the case where the display part160is a transmissive HMD, an operation of holding the virtual object V2between two operating objects (for example, two fingers). Alternatively, the operation of specifying the virtual object V2may be a gesture of turning a line of sight to the virtual object V2. For example, the operation of specifying the virtual object V2may be performed during the time period from the start to the release of specifying the virtual object V2.

As shown inFIG. 7, let us assume that the position and the attitude of the imaging part120are changed to P2and Q2, respectively, under the state where the virtual object V2is not specified. Under the state where the virtual object V2is not specified, the display controller115may fix the virtual object V2within the AR space. For example, under the state where the virtual object V2is not specified, the display controller115may cause the size of the virtual object V2in the image to be changed in accordance with the distance between the imaging part120and the virtual object V2.

For example, referring toFIG. 8, an image Im4is captured. As shown inFIG. 8, under the state where the virtual object V2is not specified, the display controller115may make the size of the virtual object V2in the image Im4to decrease, as the user Ua moves away from the virtual object V2and the distance between the imaging part120and the virtual object V2increases.

Further, under the state where the virtual object V2is not specified, the display controller115may cause the attitude of the virtual object V2in the image to be changed in accordance with the attitude of the virtual object V2based on the imaging part120. For example, under the state where the virtual object V2is not specified, in the case where the user Ua changes an imaging direction and the attitude of the virtual object V2based on the imaging part120has changed, the display controller115may cause the attitude of the virtual object V2in the image to be changed in accordance with the changed attitude.

On the other hand, in the case where a user operation acquired by the operation acquisition part111is an operation of specifying the virtual object V2, the display controller115may cause the virtual object V2to move within the AR space. While the display controller115causes the virtual object V2to move within the AR space, the display controller115may control any output. For example, while the display controller115causes the virtual object V2to move within the AR space, the display controller115may cause the fact that the virtual object V2is being moved to be displayed.

For example, as shown inFIG. 9, the display controller115may cause a message L1to be displayed, the message L1showing that the virtual object V2is “moving”. However, the fact that the virtual object V2is being moved may be shown without using a message. For example, the display controller115may cause the fact that the virtual object V2is being moved to be displayed by allowing the virtual object V2to have a motion.

In what way to move the virtual object V2is not limited. As an example, as shown inFIG. 10, under the state where the virtual object V2is specified, in the case where the position of the imaging part120is changed from P2to P3and the attitude of the imaging part120is changed from Q2to Q3, the display controller115may cause the virtual object V2to move within the AR space under the state where the relative position relationship between the imaging part120and the virtual object V2is maintained.

In the case where the virtual object V2is moved within the AR space in this way, the virtual object V2may be fixed in the imaging range. Accordingly, the display controller115may cause some kind of virtual object (for example, a stick that pierces the virtual object V2) indicating that virtual object V2is fixed to the imaging range to be displayed. In the example shown inFIG. 10, the position of the virtual object V2is moved from p1to p2, and the attitude of the virtual object V2is changed from q1to q2.

Further, the display controller115may grasp the position and the attitude of the imaging part120in any technique. For example, the display controller115may grasp the position and the attitude of the imaging part120on the basis of a result of environment recognition performed by the environment recognition part114. As the environment recognition performed by the environment recognition part114, calculation based on the SLAM technique can be used. According to the calculation based on the SLAM technique, a three-dimensional structure of a real space shown in an image captured by the imaging part120and a position and an attitude of the imaging part120can be recognized dynamically.

Note thatFIG. 10shows an example in which the display controller115causes one virtual object V2specified by an operation for specifying the virtual object V2to move within the AR space, but the number of virtual objects to be moved may not be one. In the case where a user operation acquired by the operation acquisition part111is an operation for specifying a virtual object, the display controller115may cause one or multiple virtual objects specified by the operation for specifying a virtual object to move within the AR space.

According to such a configuration, the user Ua can cause the virtual object V2to move by changing the position and the attitude of the imaging part120during the time period from the start to the release of the operation of specifying the virtual object V2. Accordingly, a sense as if the virtual object V2is moved using a drag and drop operation can be given to the user Ua, and the user Ua can intuitively move the virtual object V2within the AR space.

Here, it is expected that an environment of the virtual object V2at a destination may be any of various environments. Accordingly, the display controller115may control the virtual object V2on the basis of the environment of the virtual object V2at the destination. For example, as shown inFIG. 11, in the case where the environment of the virtual object V2at the destination is a movement-incapable region R2, the display controller115may cause the position of the virtual object V2to be shifted to a movement-capable region R1. The movement-incapable region R2may be a region in which it is not possible for a real object to enter from outside, such as a pillar and a wall.

Alternatively, the display controller115may cause the virtual object V2to perform a motion associated with the environment of the virtual object V2at the destination. For example, in the case where the environment of the virtual object V2at the destination is the sea, the display controller115may express the virtual object V2in an animation in which the virtual object V2starts swimming. Further, for example, in the case where the environment of the virtual object V2at the destination is the land, the display controller115may express the virtual object V2in an animation in which the virtual object V2stops swimming.

FIG. 12is a diagram showing a display example of after movement of the virtual object V2in a case where the virtual object V2is to be moved within the AR space. Referring toFIG. 12, an image Im5is captured, and the virtual object V2is being moved. In this way, the position of the virtual object V2placed in the AR space can be changed in accordance with a user's intention. In the example shown inFIG. 12, an aquarium is used as the background, and the family of the user Ua and the virtual object V2can be fit into a single imaging range.

FIG. 13is a flowchart showing an operation example of controlling a position/attitude of the virtual object V2. First, the environment recognition part114starts environment recognition (S21), and the operation acquisition part111acquires a user operation. In the case where an operation for specifying the virtual object V2is not performed (“No” in S22), the display controller115fixes the virtual object V2in the AR space (S23), and the controller110proceeds to S27.

On the other hand, in the case where an operation for specifying the virtual object V2is performed (“Yes” in S22), the display controller115causes the virtual object V2to move in the AR space (S24), and controls the virtual object V2on the basis of an environment at a destination (S25). In addition, the display controller115causes the fact that the virtual object V2is being moved to be displayed (S26), and the controller110proceeds to S27. In the case where environment recognition performed by the environment recognition part114is not continued (“No” in S27), the controller110may complete the operation, and in the case where the environment recognition performed by the environment recognition part114is being continued (“Yes” in S27), the processing may return to step S22.

Heretofore, “Position/attitude control on virtual object” has been described.

1-5. Size Control on Virtual Object

Subsequently, size control on a virtual object will be described. Referring toFIG. 14, the display controller115places the virtual object V2in an AR space2. Here, the size of the virtual object V2in an image is generally decided in accordance with the size of the real object A1in the image. However, there is a case where the user Ua wants to change the size of the virtual object V2in the AR space after the placement of the virtual object V2.

For example, assuming a scene where the user Ua is about to take a photograph of the virtual object V2, there may be a case where the user Ua wants to increase the size of the virtual object V2in the AR space up to the real size, and then take a photograph of the virtual object V2. Accordingly, this section proposes technology for making it possible to change the size in the AR space of the virtual object V2placed in the AR space in accordance with a user's intention.

Specifically, in the case where a user operation is a second operation, the display controller115may change the size of the virtual object V2in the AR space on the basis of the second operation. The second operation may be any operation, and may include, as shown inFIG. 14for example, an operation of changing a distance between multiple operating objects. For example, in the case where the user operation is an operation of making the distance between multiple operating objects smaller (for example, pinch-close operation), the size of the virtual object V2in the AR space may be reduced.

Further, for example, in the case where the user operation is an operation of making the distance between multiple operating objects larger (for example, pinch-open operation), the size of the virtual object V2in the AR space may be increased. Referring toFIG. 15, as a result of increasing the size of the virtual object in the AR space, there is shown a state where the virtual object placed in the AR space is changed from the virtual object V2to a virtual object V3.

FIG. 16is a diagram showing a display example of after enlargement of a virtual object in a case where the virtual object is to be enlarged within an augmented reality space. As shown inFIG. 16, in the case where it is possible to enlarge the virtual object V3, a limit on enlargement may be set for the virtual object V3. (Alternatively, in the case where it is possible to reduce the virtual object V3, a limit on reduction may be set for the virtual object V3.) For example,FIG. 16shows an image Im6, and as shown inFIG. 16, the display controller115may stop enlarging the virtual object when the size of the virtual object in the AR space has reached the real size of the virtual object.

Further, in the case where the size of the virtual object in the AR space has reached the real size of the virtual object, the display controller115may cause the fact that the size of the virtual object in the AR space has reached the real size of the virtual object to be displayed. For example, as shown inFIG. 16, the display controller115may cause a message L2to be displayed, which shows that the size of the virtual object in the AR space has reached the real size.

Further, in the example shown inFIG. 16, the display controller115causes a size M2to be displayed, the size M2being the size of the virtual object V3displayed in the AR space. Further, the display controller115causes a ratio N2to be displayed, the ratio N2being the ratio of the current size of the virtual object V3in the AR space to the real size of the virtual object V3.

Here, a technique of determining whether the size of the virtual object in the AR space has reached the real size of the virtual object is not particularly limited. As an example, the display controller115may determine whether the size of the virtual object in the AR space has reached the real size of the virtual object on the basis of known data related to the real size of the real object A1, the size of the real object A1in the image, and known data related to the real size of the virtual object.

More specifically, in the case where the real size of the real object A1is registered as the known data, the display controller115may calculate the ratio of the real size of the real object A1to the size of the real object A1in the image. In this way, when the real object A1and the virtual object are placed at the same distance from the imaging part120, the display controller115can calculate the size of the virtual object in the AR space by multiplying the size of the virtual object in the image by the calculated ratio.

Note that, even when the distance between the real object A1and the imaging part120is different from the distance between the virtual object and the imaging part120, the size of the virtual object in the AR space can be calculated. For example, with the use of the ratio of the real size of the real object A1to the size of the real object A1in the image, the position of the real object A1in the real space based on the position of the imaging part120can be grasped. Further, even if the user Ua moves the imaging part120to a position at which the virtual object is included in the imaging range, the display controller115can grasp, on the basis of a result of environment recognition, the position of the real object A1in the real space and the position of the virtual object in the AR space based on the position of the imaging part120.

Then, using the position of the virtual object in the AR space based on the position of the imaging part120, the display controller115can grasp the size of the virtual object in the AR space from the size of the virtual object in the image. In the case where the real size of the virtual object is registered as the known data, the display controller115can determine whether the size of the virtual object in the AR space has reached the real size of the virtual object by comparing the real size of the virtual object with the size of the virtual object in the AR space.

Since a size of a virtual object in an image is generally determined in accordance with a size of a real object in the image, it was necessary, in order to place the virtual object having a desired size in the AR space, to adjust the size of the real object in the real space or to adjust the size of the real object shown in the image. With the size control on the virtual object according to the present embodiment, it becomes possible to place the virtual object having a desired size in the AR space in accordance with the user operation.

FIG. 17is a flowchart showing an operation example of controlling a size of a virtual object. First, the environment recognition part114starts environment recognition (S31), and the operation acquisition part111acquires a user operation. In the case where there is no size changing operation (“No” in S32), the controller110proceeds to S37.

On the other hand, in the case where there is a size changing operation (“Yes” in S32), the display controller adjusts the size of the virtual object in the AR space (S33), and in the case where the size of the virtual object in the AR space has not reached the real size (“No” in S34), the controller110proceeds to S37. On the other hand, in the case where the size of the virtual object in the AR space has reached the real size (“Yes” in S34), the display controller115stops enlarging the virtual object (S35), causes the fact that the size of the virtual object in the AR space has reached the real size to be displayed (S36), and the controller110proceeds to S37.

In the case where the environment recognition performed by the environment recognition part114is not continued (“No” in S37), the controller110may complete the operation, and in the case where the environment recognition performed by the environment recognition part114is being continued (“Yes” in S37), the processing may return to step S31.

Heretofore, the case where the number of virtual objects is one has been described, but there may be a case where the number of virtual objects may be two or more. In such a case, the sizes of some of the multiple virtual objects may be adjusted, or the sizes of all the multiple virtual objects may be adjusted. For example, the display controller115may select a virtual object the size of which is to be changed in accordance with the distances from the imaging part120to the respective virtual objects.

FIG. 18is a diagram showing a display example of before enlargement of the virtual object V2in the case where the virtual object V2out of virtual objects V11and V2is to be enlarged within the AR space. Referring toFIG. 18, an image Im7is captured.FIG. 19is a diagram showing a display example of after enlargement of the virtual object V2. Referring toFIG. 18, an image Im8is captured. In the examples shown in those figures, the display controller115selects the virtual object that is the nearest from the imaging part120(virtual object placed at the nearest side when seen from the imaging part120) as the target the size of which is to be changed.

In this case, for example, the virtual object that is the nearest from the imaging part120may be displayed in a mode which can be distinguished from the other virtual objects. For example, the virtual object that is the nearest from the imaging part120may be displayed in a manner that the virtual object has a motion and the other virtual objects may be displayed in a manner that they stand still. However, the technique for adjusting sizes of some of the multiple virtual objects is not limited thereto.

For example, in the case where the user Ua conducts a selection operation, the display controller115may select a virtual object the size of which is to be changed on the basis of the selection operation. The selection operation is not limited, and may be a tap operation, for example. Alternatively, in the case where the size changing operation is a pinch-open operation or a pinch-close operation, the display controller115may select the virtual object that is directly specified by the operating object in the course of the pinch-open operation or the pinch-close operation.

Heretofore, “Size control on virtual object” has been described.

1-6. Hardware Configuration Example

Next, a hardware configuration example of the display control device10according to an embodiment of the present disclosure will be described.FIG. 20is a diagram showing a hardware configuration example of the display control device10according to an embodiment of the present disclosure. It should be noted that the hardware configuration example shown inFIG. 20is merely an example of a hardware configuration of the display control device10. Accordingly, the hardware configuration of the display control device10is not limited to the example shown inFIG. 20.

As shown inFIG. 20, the display control device10includes a central processing unit (CPU)801, read only memory (ROM)802, random access memory (RAM)803, a sensor804, an input device808, an output device810, a storage device811, a drive812, an imaging device813, and a communication device815.

The CPU801functions as an arithmetic processing unit and a control unit, and controls entire operation of the display control device10in accordance with various programs. Further, the CPU801may be a microprocessor. The ROM802stores a program, a calculation parameter, and the like used by the CPU801. The RAM803temporarily stores a program used in execution of the CPU801, a parameter varying as appropriate during the execution, and the like. The CPU801, the ROM802, and the RAM803are connected with each other via the host bus configured from a CPU bus or the like.

The sensor804includes various types of detection sensors such as a terminal state detection sensor for detecting states of the display control device10, and a peripheral circuit thereof. Examples of the sensor804include a tilt sensor, an acceleration sensor, an orientation sensor, a temperature sensor, a humidity sensor, and a light intensity sensor. A detection signal obtained by the sensor804is transmitted to the CPU801. In this way, the CPU801can know the states (tilt, acceleration, orientation, temperature, humidity, light intensity, and the like) of the display control device10.

The input device808is configured from, for example, an input part for inputting information by a user, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever, and an input control circuit which generates an input signal based on the input by the user and outputs the generated input signal to the CPU801. The user of the display control device10can input various kinds of data to the display control device10and can instruct the display control device10to perform a processing operation by operating the input device808.

The output device810includes, for example, display devices such as a liquid crystal display (LCD) device, an organic light emitting diode (OLED) device, and a lamp. Further, the output device810includes audio output devices such as a speaker and headphones. For example, the display devices each display a captured image, a generated image, and the like. On the other hand, the audio output devices each convert audio data and the like into audio and output the audio.

The storage device811is a device for storing data, which is configured as an example of a storage of the display control device10. The storage device811may include, for example, a storage medium, a recording device for recording data in the storage medium, a reading device for reading out the data from the storage medium, and a deletion device for deleting the data recorded in the storage medium. The storage device811stores a program and various data executed by the CPU801.

The drive812is a reader/writer for the storage medium and is built in or externally attached to display control device10. The drive812reads out information recorded in a removable storage medium71which is mounted thereto, such as a magnetic disk, an optical disc, a magneto-optical disk, or semiconductor memory, and outputs the information to the RAM803. Further, the drive812can also write information in the removable storage medium71.

The imaging device813includes an imaging optical system, such as a shooting lens and a zoom lens for focusing light, and a signal conversion device, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The imaging optical system focuses light emitted from a subject and forms an image of the subject on a signal conversion part, and the signal conversion device converts the formed image of the subject into an electrical image signal.

The communication device815is a communication interface which is configured from, for example, a communication device for establishing a connection with a network. In addition, the communication device815may be a wireless local area network (LAN) enabled communication device, a long term evolution (LTE) enabled communication device, or a wired communication device for performing wired communication. The communication device815can communicate with other devices via a network30.

Heretofore, a hardware configuration example of the display control device10according to an embodiment of the present disclosure has been described.

As described above, according to the embodiments of the present disclosure, there is provided the display control device10including the display controller115configured to place a virtual object within an AR space corresponding to a real space in accordance with a recognition result of a real object A1shown in an image captured by the imaging part120, and the operation acquisition part111configured to capture a user operation. In the case where the user operation is a first operation, the display controller115moves the virtual object within the AR space. According to such a configuration, it becomes possible to change a position of the virtual object placed in the AR space in accordance with a user's intention.

Further, according to the position/attitude control and the size control on the virtual object of an embodiment of the present disclosure, when the user Ua performs an operation of enlarging a virtual object up to a real size, for example, there may occur a case where it is necessary for the user Ua to move away from the virtual object in order to fit the virtual object within the imaging range in the process of the operation of enlarging the virtual object. By making the user Ua perform such an operation, it becomes possible for allowing the user Ua to grasp the real size of the virtual object more intuitively.

In an embodiment of the present disclosure, as a scene for moving the virtual object within the AR space and adjusting the size of the virtual object, the scene is mainly assumed where a photograph is taken in which a virtual object is fit within the imaging range. However, an embodiment of the present disclosure may be applied to any scene other than such a scene. For example, an embodiment of the present disclosure can be effectively applied also to a scene in which a virtual object which appears in a game application is moved within the AR space and the size of the virtual object is adjusted.

Further, in an embodiment of the present disclosure, the description has been mainly made of the example that the result of controlling the virtual object by the display control device10is reflected in the AR space of the display control device10. However, for example, in the case where a single AR space is shared between the display control device10and other devices, the result of controlling the virtual object by the display control device10may be reflected in the AR space of the other devices which are communicable with the display control device10.

Further, it is also possible to create a program for causing hardware such as a CPU, ROM, and RAM, which are built in a computer, to exhibit substantially the same functions as those of respective structures of the display control device10described above. Further, there is also provided a non-transitory computer-readable recording medium having the program recorded thereon.

(1) A display control device including:

a display controller configured to place a virtual object within an augmented reality space corresponding to a real space in accordance with a recognition result of a real object shown in an image captured by an imaging part; and

an operation acquisition part configured to acquire a user operation,

wherein, when the user operation is a first operation, the display controller causes the virtual object to move within the augmented reality space.

(2) The display control device according to (1),

wherein, when the user operation is the first operation, the display controller causes the virtual object to move within the augmented reality space under a state where a relative position relationship between the imaging part and the virtual object is maintained.

(3) The display control device according to (1) or (2),

wherein, when the user operation is the first operation, the display controller causes one or a plurality of virtual objects specified by the first operation to move within the augmented reality space.

wherein, while the display controller causes the virtual object to move within the augmented reality space, the display controller causes a fact that the virtual object is being moved to be displayed.

wherein, while the first operation is not being performed, the display controller fixes the virtual object within the augmented reality space.

(6) The display control device according to (5),

wherein, while the first operation is not being performed, the display controller causes a size of the virtual object in the image to be changed in accordance with a distance between the imaging part and the virtual object.

(7) The display control device according to (5),

wherein, while the first operation is not being performed, the display controller causes an attitude of the virtual object in the image to be changed in accordance with an attitude of the virtual object based on the imaging part.

(8) The display control device according to any one of (1) to (7),

wherein the display controller controls the virtual object on a basis of an environment of the virtual object at a destination.

(9) The display control device according to (8),

wherein, when the environment of the virtual object at the destination is a movement-incapable region, the display controller causes a position of the virtual object to be shifted to a movement-capable region.

wherein the first operation includes an operation of specifying the virtual object using one or a plurality of operating objects.

(11) The display control device according to (1),

wherein, when the user operation is a second operation, the display controller changes a size of the virtual object in the augmented reality space on a basis of the second operation.

(12) The display control device according to (11),

wherein, when the size of the virtual object in the augmented reality space has reached a real size of the virtual object, the display controller stops enlarging the virtual object.

(13) The display control device according to (11) or (12),

wherein, when the size of the virtual object in the augmented reality space has reached a real size of the virtual object, the display controller causes a fact that the size of the virtual object in the augmented reality space has reached the real size of the virtual object to be displayed.

(14) The display control device according to any one of (11) to (13),

wherein the display controller determines whether the size of the virtual object in the augmented reality space has reached a real size of the virtual object on a basis of known data related to a real size of the real object, a size of the real object in the image, and known data related to the real size of the virtual object.

(15) The display control device according to any one of (11) to (14),

wherein, when a selection operation is performed, the display controller selects a virtual object a size of which is to be changed on a basis of the selection operation.

(16) The display control device according to any one of (11) to (14),

wherein the display controller selects a virtual object a size of which is to be changed in accordance with distances from the imaging part and the respective one or plurality of virtual objects.

(17) The display control device according to any one of (11) to (16),

wherein the second operation includes an operation of making a distance between a plurality of operating objects larger.

(18) The display control device according to (1),

wherein the display controller places the virtual object in a manner that the virtual object has an attitude corresponding to a gravity vector.

(19) A display control method including:

placing a virtual object within an augmented reality space corresponding to a real space in accordance with a recognition result of a real object shown in an image captured by an imaging part;

acquiring a user operation; and

causing the virtual object to move within the augmented reality space when the user operation is a first operation.

(20) A non-transitory computer-readable recording medium having a program recorded thereon, the program being for causing a computer to function as a display control device including

a display controller configured to place a virtual object within an augmented reality space corresponding to a real space in accordance with a recognition result of a real object shown in an image captured by an imaging part, and

an operation acquisition part configured to acquire a user operation,

wherein, when the user operation is a first operation, the display controller causes the virtual object to move within the augmented reality space.