Patent Description:
In recent years, a moving body operated using a steering device or the like, such as a drone, has been used. For example, an image of a landscape captured in the sky using a drone equipped with a camera is used.

For example, Patent Literature <NUM> describes a technology for efficiently transferring an image by switching a mode from an image capturing mode to an image transfer mode when a pre-defined mode switching condition occurs.

As a method of controlling a movement of a moving body such as a drone, other than a method in which the movement of the moving body is controlled by a user using a steering device, a method in which a route for the moving body to move along is set in advance such that the moving body moves along the preset route may be considered. In this case, for example, it may be considered to set a movement route of the moving body on a map.

However, in the method of setting the route of the moving body on the two-dimensional map, in a case where the moving body moves three-dimensionally, it is difficult to intuitively set the movement of the moving body. The technology described in Patent Literature <NUM> is not intended to intuitively generate movement information for controlling a movement of a moving body such as a drone.

Therefore, the present disclosure proposes a new and improved information processing apparatus, information processing method, and program making it possible to intuitively generate information for moving a moving body. Solution to Problem.

Note that, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished from each other by attaching different alphabets after the same reference numeral. For example, the plurality of components having substantially the same functional configuration will be distinguished from each other, like a user terminal 10a and a user terminal 10b, if necessary. However, in a case where it is not necessary to particularly distinguish the plurality of components having substantially the same functional configuration from each other, only the same reference numeral will be attached. For example, in a case where it is not necessary to particularly distinguish the user terminal 10a and the user terminal 10b from each other, they will simply be referred to as user terminal <NUM>.

Note that the description will be given in the following order.

First, a configuration of an information processing system <NUM> according to an embodiment of the present disclosure will be described with reference to <FIG> is a diagram illustrating a configuration of the information processing system <NUM> according to an embodiment of the present disclosure. The information processing system <NUM> includes a user terminal <NUM> and a moving body <NUM>. The user terminal <NUM> and the moving body <NUM> are communicably connected to each other.

The user terminal <NUM> may be, for example, a smartphone, a tablet terminal, or the like. The user terminal <NUM> generates movement information for controlling a movement of the moving body <NUM> according to an operation of a user, and transmits the movement information to the moving body <NUM>. In addition, the user terminal <NUM> can also display a virtual object and the like, which will be described later, according to an operation of the user.

The moving body <NUM> is a device moving on the basis of the movement information generated by the user terminal <NUM>. Here, the moving body <NUM> can be any type of movable device, but it will be assumed in the following description that the moving body <NUM> is a drone. In addition, the moving body <NUM> may be equipped with an imaging device for imaging a landscape.

A configuration of the user terminal <NUM> according to an embodiment of the present disclosure will be described with reference to <FIG> is a functional block diagram illustrating a configuration of the user terminal <NUM> according to an embodiment of the present disclosure.

The user terminal <NUM> functions to acquire image information, sensor information, information based on the operation of the user, and the like, and output results of performing various types of processing on the acquired information. The functions of the user terminal <NUM> are implemented by an information processing apparatus <NUM>, an imaging unit (first imaging device) <NUM>, a sensor unit <NUM>, an input unit <NUM>, and a display unit <NUM> included in the user terminal <NUM> in cooperation with each other.

The imaging unit <NUM> may be any known type of imaging device that captures an image. The imaging unit <NUM> includes any known imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor. In addition to such an imaging element, the imaging unit <NUM> may include various members such as a lens for forming an image of a subject on the imaging element and a light source for irradiating the subject with illumination light. The imaging unit <NUM> transmits the image information obtained by capturing the image to the information processing apparatus <NUM>.

The sensor unit <NUM> includes at least one of various known types of sensors, for example, a distance measuring sensor, an inertial measurement unit (IMU), and the like. The distance measuring sensor may be, for example, a stereo camera, a time of flight (ToF) sensor, or the like. The distance measuring sensor detects distance information, for example, regarding a distance between the user terminal <NUM> and an object or the like existing on the periphery thereof, and transmits the detected distance information to the information processing apparatus <NUM>. In addition, the IMU includes at least one of, for example, an acceleration sensor, a gyro sensor, or a magnetic sensor. The IMU transmits detected information as IMU information to the information processing apparatus <NUM>.

The input unit <NUM> functions to generate input information on the basis of an operation by the user. The input unit <NUM> can be, for example, a touch panel or the like. The input unit <NUM> generates the input information on the basis of any kind of operation by the user, for example, a touch operation, a drag operation, a pinch-out operation, a pinch-in operation, or the like. The input unit <NUM> transmits the generated input information to an acquisition unit <NUM>.

The information processing apparatus <NUM> functions to perform various types of processing on the basis of the acquired information and control a display on the display unit <NUM> on the basis of the results of the processing. The functions of the information processing apparatus <NUM> are implemented by the acquisition unit <NUM>, a processing unit <NUM>, a display control unit <NUM>, a storage unit <NUM>, and a communication control unit <NUM> in cooperation with each other.

The acquisition unit <NUM> acquires information input from at least one of the imaging unit <NUM>, the sensor unit <NUM>, or the input unit <NUM>. The acquisition unit <NUM> transmits the acquired information to the processing unit <NUM>.

The processing unit <NUM> functions to perform various types of processing on the basis of the information transmitted from the acquisition unit <NUM>. For example, the processing unit <NUM> functions to generate information for controlling the moving body <NUM> on the basis of the information transmitted from the acquisition unit <NUM>. In addition, the processing unit <NUM> generates information regarding contents to be displayed on a display screen of the display unit <NUM>. The detailed configuration and functions of the processing unit <NUM> will be described later with reference to <FIG>. The processing unit <NUM> transmits the generated information to the display control unit <NUM>, the storage unit <NUM>, or the communication control unit <NUM>.

The display control unit <NUM> functions to control a display on the display screen of the display unit <NUM>. The display control unit <NUM> controls, for example, a display of a virtual object based on an object existing in a real space on the display screen of the display unit <NUM>, on the basis of the information transmitted from the processing unit <NUM>.

The display unit <NUM> is any known type of display device that functions to display an image. In the present embodiment, the display unit <NUM> is integrated with the above-described input unit <NUM> and configured as a touch panel. As will be described later, the user can cause the information processing apparatus <NUM> to generate movement information for controlling a movement of the moving body <NUM> in the user terminal <NUM> by performing a predetermined operation while referring to the display screen of the display unit <NUM>.

The storage unit <NUM> functions to store various kinds of information such as information generated or acquired by the information processing apparatus <NUM>. For example, the storage unit <NUM> may store information regarding a virtual object generated in advance. Note that a method of generating a virtual object will be described later. Furthermore, the storage unit <NUM> may store movement information for controlling a movement of the moving body <NUM>. More specifically, the storage unit <NUM> may store information (waypoint information) regarding a specific point (also referred to as "waypoint") included in a route of the moving body <NUM>. The route of the moving body <NUM> may be formed by connecting a plurality of waypoints to one another. Furthermore, the storage unit <NUM> may store movement information, imaging information, or the like generated by the processing unit <NUM>. The information stored in the storage unit <NUM> is referred to by the processing unit <NUM>, the display control unit <NUM>, or the communication control unit <NUM> if necessary.

The communication control unit <NUM> functions to control transmission of various kinds of information generated by the processing unit <NUM>. The communication control unit <NUM> controls transmission of movement information or imaging information generated by the processing unit <NUM>. The movement information or the imaging information is transmitted to the moving body <NUM>. The moving body <NUM> can move or capture an image on the basis of the transmitted information.

Next, the processing unit <NUM> included in the information processing apparatus <NUM> will be described in more detail with reference to <FIG> is a functional block diagram illustrating a configuration of the processing unit <NUM>. As illustrated in <FIG>, the processing unit <NUM> includes a detection unit <NUM>, a self-position calculation unit <NUM>, a virtual object calculation unit <NUM>, a generation unit <NUM>, and a prediction unit <NUM>.

The detection unit <NUM> functions to perform various types of detection on the basis of the information transmitted from the acquisition unit <NUM>. The functions of the detection unit <NUM> are implemented by a plane detection unit <NUM> and an object detection unit <NUM>. The plane detection unit <NUM> functions to detect a plane included in an image on the basis of the image information, the distance information, and the like. The object detection unit <NUM> functions to detect a predetermined object included in an image on the basis of the image information, the distance information, and the like. The detection unit <NUM> transmits a detection result to the self-position calculation unit <NUM>.

The self-position calculation unit <NUM> functions to calculate a self-position of the user terminal <NUM>. Here, the self-position of the user terminal <NUM> includes not only a position where the user terminal <NUM> exists but also a posture of the user terminal <NUM>. Specifically, the self-position calculation unit <NUM> calculates a position or posture of the user terminal <NUM> with respect to an environment or object around the user terminal <NUM>, with the image information, the distance information, and the IMU information being input thereto, using a simultaneous localization and mapping (SLAM) technology. At this time, the self-position calculation unit <NUM> may determine an origin or a scale in SLAM on the basis of the object information, the plane information, or the like. The self-position calculation unit <NUM> transmits a calculation result to the virtual object calculation unit <NUM> and the generation unit <NUM>.

The virtual object calculation unit <NUM> generates information regarding a virtual object to be arranged on the display screen. More specifically, the virtual object calculation unit <NUM> calculates arrangement information (information regarding a position, an orientation, or the like), scale information, or the like on the virtual object to be arranged on the display screen of the display unit <NUM>, on the basis of the self-position of the user terminal <NUM> calculated by the self-position calculation unit <NUM>, the detection result of the detection unit <NUM>, the input information input to the input unit <NUM>, the information regarding the virtual object stored in the storage unit <NUM>, and the like. Here, the virtual object calculation unit <NUM> calculates a scale of the virtual object based on a scale of the real space. More specifically, the virtual object calculation unit <NUM> determines the scale of the virtual object to be displayed on the display screen to generate scale information by appropriately expanding or contracting the scale of the virtual object from a scale of a real object on which the virtual object is based. The virtual object calculation unit <NUM> transmits a calculation result to a movement information generation unit <NUM>, which will be described later.

The generation unit <NUM> functions to generate various kinds of information for controlling the moving body <NUM>. More specifically, the generation unit <NUM> generates information for controlling a movement of the moving body <NUM>, an operation of an imaging device (second imaging device) included in the moving body <NUM>, a display of the display unit <NUM>, and the like. The functions of the generation unit <NUM> are implemented by the movement information generation unit <NUM>, an imaging information generation unit <NUM>, and a display information generation unit <NUM> included in the generation unit <NUM>.

The movement information generation unit <NUM> generates movement information related to the virtual object for controlling a movement of the moving body <NUM>. Specifically, the movement information generation unit <NUM> generates a position and an orientation of a waypoint as the movement information on the basis of the self-position of the user terminal <NUM>, the information input to the input unit <NUM>, the arrangement information and the scale information on the virtual object, and the waypoint information. For example, the movement information generation unit <NUM> may generate a route of the moving body <NUM> as the movement information. At this time, the movement information generation unit <NUM> may generate scale-related movement information according to the scale information on the virtual object, or may generate movement information by adapting the movement information to match the scale of the real space. Furthermore, the movement information generation unit <NUM> can also correct the movement information on the basis of an operation of the user through the input unit <NUM> or the like to generate new movement information. The operation by the user will be described in detail later. The movement information generation unit <NUM> transmits the generated movement information to the display information generation unit <NUM>, the prediction unit <NUM>, and the storage unit <NUM>.

The imaging information generation unit <NUM> functions to generate imaging information for controlling an imaging range of the imaging device included in the moving body <NUM> on the basis of an operation of the user. More specifically, the imaging information generation unit <NUM> generates imaging information regarding a direction in which the imaging device faces, an angle of view, or the like on the basis of the input information generated by the input unit <NUM> or the like. The imaging information generation unit <NUM> transmits the generated imaging information to the prediction unit <NUM> and the storage unit <NUM>.

The display information generation unit <NUM> functions to generate display information regarding contents to be displayed on the display unit <NUM>. More specifically, the display information generation unit <NUM> generates computer graphics (CG) to be displayed on the display unit <NUM> as the display information on the basis of the position of the imaging unit <NUM>, the arrangement information and the scale information on the virtual object, the waypoint information, a prediction result of the prediction unit <NUM>, which will be described later, and the like. The display information generation unit <NUM> can generate a video showing the movement of the moving body <NUM> as the display information. Furthermore, the display information generation unit <NUM> can also generate display information for displaying a simulation video to be captured by the imaging device included in the moving body <NUM>. The display information generation unit <NUM> transmits the generated display information to the display control unit <NUM>.

The prediction unit <NUM> functions to predict an operation of the moving body <NUM>. More specifically, the prediction unit <NUM> can predict a movement of the moving body <NUM> and an operation of the imaging device included in the moving body <NUM>. The functions of the prediction unit <NUM> are implemented by a movement prediction unit <NUM> and an imaging prediction unit <NUM>. The prediction unit <NUM> transmits a prediction result to the display information generation unit <NUM>.

The movement prediction unit <NUM> functions to predict a movement of the moving body <NUM> on the basis of the movement information. For example, the movement prediction unit <NUM> can predict a route of the moving body <NUM>. Furthermore, the imaging prediction unit <NUM> predicts an image to be captured by the imaging device included in the moving body <NUM> on the basis of the movement information and the imaging information. More specifically, the imaging prediction unit <NUM> predicts an image to be captured by the imaging device on the basis of the route along which the imaging device passes, the posture of the imaging device, and the like.

In the present embodiment, a virtual object is displayed on the display unit <NUM>, and the user can cause the information processing apparatus <NUM> to generate movement information, imaging information, or the like by performing a predetermined operation while viewing the display. Here, an example of a method of generating the virtual object will be described. Note that the method of generating the virtual object is not limited to what will be described below, and the virtual object may be generated by any method.

The method for generating the virtual object will be described with reference to <FIG>. <FIG> is a diagram illustrating a state in which a user U1 steers the moving body <NUM> to image a tower <NUM> and a forest <NUM>. In addition, <FIG> is a diagram illustrating a virtual object <NUM> generated on the basis of the imaged tower <NUM>. In addition, <FIG> is a diagram illustrating a virtual object <NUM> generated on the basis of the imaged forest <NUM>. Further, <FIG> is a diagram illustrating a route <NUM> along which the moving body <NUM> has moved.

First, the method for generating the virtual object according to the present embodiment will be briefly described. Here, it is assumed that the user U1 wants an imaging device <NUM> to capture a video including the tower <NUM>. In the present embodiment, first, the user U1 steers the moving body <NUM> using a steering device <NUM> such that the imaging device <NUM> images the tower <NUM> existing in a real space and the forest <NUM> existing around the tower <NUM> in advance. Next, on the basis of the captured image, a three-dimensional virtual object is generated using various CG technologies. In the present embodiment, movement information is further generated with waypoints being set in the route along which the moving body <NUM> has passed. Hereinafter, the method of generating the virtual object will be described in more detail with reference to <FIG>.

First, as illustrated in <FIG>, the user U1 causes the moving body (drone) <NUM> to fly using the steering device <NUM>. The moving body <NUM> includes an airframe <NUM>, a propeller <NUM>, and the imaging device <NUM>. The moving body <NUM> can fly by driving the propeller <NUM>. In addition, the.

user U1 steers the propeller <NUM> or the like to control an orientation, a posture, a speed, and the like of the airframe <NUM>. Furthermore, the imaging device <NUM> included in the moving body <NUM> images a landscape around the moving body <NUM>. Here, an image including the tower <NUM> and the forest <NUM> on the periphery thereof as illustrated in <FIG> is captured by the imaging device <NUM>.

On the basis of the captured image, virtual objects of the tower <NUM> and the forest <NUM> are generated using various known CG technologies. More specifically, the virtual object <NUM> of the tower <NUM> existing in the real space as illustrated in <FIG> and the virtual object <NUM> of the forest <NUM> existing in the real space as illustrated in <FIG> are generated. The information regarding the virtual objects generated in this way is stored in the storage unit <NUM> included in the information processing apparatus <NUM>.

Furthermore, the user U1 may cause the moving body <NUM> to actually implement a flight when a desired image is captured. More specifically, the user U1 steers the moving body <NUM> so that the moving body <NUM> passes along the route <NUM> that turns around the tower <NUM> as illustrated in <FIG>. The moving body <NUM> calculates the route <NUM> along with the moving body <NUM> has moved using a sensor included in the moving body <NUM>, and records a calculation result in a recording medium or the like included in the moving body <NUM>. Here, waypoints may be set in the route <NUM> in accordance with a predetermined rule. For example, a waypoint may be set for each predetermined distance. In addition, a waypoint density may be adjusted according to a curvature of the route <NUM>.

<FIG> illustrates an example of the route <NUM> along which waypoints <NUM> are set. Note that, although <NUM> waypoints 406a to <NUM> are illustrated in <FIG>, the number of waypoints is not limited thereto. The number of waypoints set in the route <NUM> may be two or more and <NUM> or less, or may be <NUM> or more. Also, note that waypoints are also illustrated in the drawings used for the following description, but the number of waypoints is not limited to that illustrated in the drawings.

The information regarding the route <NUM> along which the waypoints are set as described above may be stored as the movement information in the storage unit <NUM> included in the information processing apparatus <NUM>.

Here, a specific example of an operation by the user for causing the information processing apparatus <NUM> to generate movement information, imaging information, or the like on the basis of the virtual objects generated as described above will be described.

A case where movement information is generated on the basis of an operation of a user will be described. First, with reference to <FIG>, a case where, when movement information is recorded in advance in the information processing apparatus <NUM>, the movement information is corrected to generate new movement information will be described. <FIG> is a diagram illustrating a state in which a plane on a desk <NUM> existing in a real space is detected by a user terminal 10a. <FIG> and <FIG> are diagrams illustrating a state in which a waypoint 408a is selected on the basis of an operation of a user U2. Further, <FIG> and <FIG> are diagrams illustrating a state in which the position of the waypoint 408a is adjusted on the basis of an operation of the user U2.

As illustrated in <FIG>, it is assumed that the desk <NUM> exists in the real space before the eyes of the user U2. The user U2 possesses the user terminal 10a, and a start button <NUM> for starting displaying virtual objects and setting waypoints is displayed on a display screen <NUM> of the user terminal 10a. Here, the display screen <NUM> also functions as the input unit <NUM> that receives a touch operation, a pinch operation, or the like from the user U2. When the user U2 touches the start button <NUM>, the user terminal 10a detects a plane <NUM> on the desk <NUM>.

Then, as illustrated in <FIG>, an image 612a of a virtual object 422a of the tower and an image 614a of a virtual route 404a of the moving body <NUM> generated in advance are displayed on a display screen 610a of the user terminal 10a. Here, the virtual route is a route in a virtual space in which the virtual object is arranged. A scale of the virtual route is appropriately expanded or contracted to form a real route along which the moving body <NUM> actually moves. Hereinafter, in a case where it is not necessary to particularly distinguish the virtual route and the real route from each other, they will also be referred to simply as "route". At this time, the virtual object 422a is displayed on the display screen 610a as if it was arranged on the desk <NUM>. Note that, although the virtual object 422a is illustrated as being arranged on the desk <NUM> in <FIG>, the virtual object 422a is illustrated for explanation here, and the virtual object 422a is not actually arranged on the desk <NUM>.

Here, the image 614a of the virtual object 422a may be sized, for example, to be put on the desk <NUM> as illustrated in <FIG>. In addition, an image 616a of the waypoint 408a for adjusting the virtual route 404a is displayed in the image 614a of the virtual route 404a. In the present embodiment, the user U2 can also adjust the size of the image 612a of the virtual object displayed on the display screen 610a (that is, a distance from the user terminal 10a to the virtual object 422a) by performing a pinch operation or the like on the display screen 610a. At this time, the user terminal 10a may store a ratio between the size of the tower existing in the real space, on which the virtual object 422a is based, and the size of the virtual object 422a.

In the present embodiment, since the image 612a of the virtual object 422a is displayed on a plane (on the desk <NUM>), the user U2 can feel as if the virtual object 422a was arranged on the ground. Therefore, the user U2 can operate the user terminal 10a more intuitively.

Here, it is assumed that the moving body <NUM> is flying in advance by manual steering or the like, and the virtual route 404a is generated on the basis of the flight. In addition, it is assumed that waypoints are set in the virtual route 404a in advance. Specifically, as illustrated in <FIG>, <FIG> waypoints indicated by circles are set in the virtual route 404a. Further, one waypoint 408a of the <NUM> waypoints corresponds to the image 616a of the waypoint displayed on the display screen 610a.

The user U2 can select a waypoint to be adjusted by touching the image 616a of the waypoint displayed on the display screen 610a. Here, it is assumed that the user U2 selects the waypoint 408a in the virtual route 404a. Furthermore, in a state where the waypoint 408a is selected, the user U2 can adjust a position of the waypoint 408a by performing a pinch operation, a drag operation, or the like on the display screen 610a. Furthermore, in the present embodiment, the user U2 can also adjust an orientation of the imaging device included in the moving body <NUM> by operating the display screen 610a. For example, in a state where the waypoint 408a is selected, the user U2 can designate an orientation of the imaging device of the moving body <NUM> when the moving body <NUM> passes through a position corresponding to the waypoint 408a, by performing a pinch operation, a drag operation, or the like on the display screen 610a.

Next, in the state where the waypoint 408a is selected, the user U2 shifts a position of the user terminal 10a. For example, as illustrated in <FIG> and <FIG>, the user U2 pulls the user terminal 10a toward the user U2. Accordingly, the position of the selected waypoint 408a shifts according to the movement of the user terminal 10b. In addition, the virtual route 404a changes to a virtual route 404b according to a position of a waypoint 408b after the movement. By changing the virtual route <NUM>, a route along which the moving body <NUM> actually moves is adjusted. As described above, in the present embodiment, the route of the moving body <NUM> is adjusted on the basis of the operation by the user U2 for moving the user terminal 10a. In this way, the route of the moving body <NUM> is adjusted, and movement information indicating the adjusted route is newly generated. On the basis of the movement information, the moving body <NUM> can fly around the tower <NUM>.

Here, the case where the waypoints are set in the virtual route 404a in advance has been described. In a case where the waypoints are not set in the virtual route 404a in advance, the user U2 can also set a waypoint by touching a part of an image <NUM> of a virtual route <NUM> displayed on the display screen <NUM>. The waypoint set in this way can also be adjusted according to the above-described method.

As described above, according to the present embodiment, in a state where the waypoints are set, the user U2 can minutely adjust a waypoint by performing an operation on the display screen <NUM> and an operation for moving the user terminal 10a. Therefore, it is possible to more intuitively generate information for moving the moving body <NUM>.

The method of generating new movement information by adjusting the virtual route 404a in a case where the virtual route 404a of the moving body <NUM> is set in advance has been described above. Next, two methods of setting a route of the moving body <NUM> in a case where the route of the moving body <NUM> is not set in advance will be described with reference to <FIG> and <FIG>. <FIG> and <FIG> are diagrams illustrating a state in which a route of the moving body <NUM> is newly set on the basis of an operation of the user U2.

Note that, in either method, the user U2 operates the user terminal <NUM> to set a route of the moving body <NUM> and a waypoint. The waypoint set according to the methods to be described with reference to <FIG> and <FIG> may also be adjusted according to the method described above with reference to <FIG>.

A first method of setting a route of the moving body <NUM> will be described with reference to <FIG>. First, the user U2 touches a part of a display screen 610c of a user terminal 10c (for example, a designation point 616c shown on the display screen 610c). The user U2 moves the user terminal 10c, for example, downward, as indicated by a broken line, while touching the designation point 616c. The user terminal 10c stores the route along which the user terminal 10c has moved as a route of the moving body <NUM>. At this time, the user terminal 10c may set a waypoint in the route and store the waypoint together with the route.

Next, a second method of setting a route of the moving body <NUM> will be described with reference to <FIG>. In the second method, as illustrated in the upper part of <FIG>, the user U2 sets a waypoint 408d by touching a display screen 610d of a user terminal 10d. An image 616d of the set waypoint 408d is displayed on the display screen 610d.

Next, as illustrated in the lower part of <FIG>, the user U2 shifts a position of the user terminal 10d, and sets a waypoint 408e by touching a display screen 610e of a user terminal 10e at a new position. Thereafter, the movement of the user terminal <NUM> and the setting of the waypoint <NUM> are repeated, and a plurality of set waypoints <NUM> are connected to one another, thereby generating a route of the moving body <NUM>. The generated route and waypoints <NUM> are stored in the user terminal <NUM>.

In this way, according to the present embodiment, even in a case where a route of the moving body <NUM> is not set in advance, the user U2 can set a route of the moving body <NUM> by performing an operation on the display screen <NUM> and an operation for moving the user terminal <NUM>.

Here, the position of the waypoint set by operating the user terminal <NUM> will be described in more detail with reference to <FIG>. <FIG> is a diagram illustrating a state in which a position of an imaging unit 110a included in a user terminal 10f is set as a waypoint <NUM>. In addition, <FIG> is a diagram illustrating a display screen 610f in a case where the position of the imaging unit 110a included in the user terminal 10f is set as the waypoint <NUM>. In addition, <FIG> is a diagram illustrating a state in which a position away from a user terminal <NUM> by a predetermined distance is set as a waypoint <NUM>. Further, <FIG> is a diagram illustrating a display screen <NUM> in a case where the position away from the user terminal <NUM> by the predetermined distance is set as the waypoint <NUM>.

First, a position of a waypoint to be set will be described with reference to <FIG>. As illustrated in <FIG>, the virtual object 422a of the tower is arranged on the desk <NUM>. The imaging unit 110a included in the user terminal 10f captures an image in front of the imaging unit 110a. That is, the imaging unit 110a captures an image in a range in which the virtual object 422a is included. Here, the position of the imaging unit 110a is designated as a waypoint.

At this time, as illustrated in <FIG>, an image 612f of the virtual object of the tower arranged on the desk <NUM> is displayed on the display screen 610f of the user terminal 10f. The user can set a waypoint by touching the display screen 610f or doing the like while viewing the display screen 610f. In addition, the user can set a waypoint <NUM> at a new position by moving the user terminal 10f.

At this time, an image displayed on the display screen <NUM> may correspond to that to be actually captured at a waypoint by the imaging device of the moving body <NUM>. In this case, the user can check an image to be captured by the imaging device included in the moving body <NUM> in advance.

Next, a method in which the user terminal <NUM> sets a position away from the imaging unit 110a by a predetermined distance as a waypoint will be described with reference to <FIG> and <FIG>. Specifically, a position away forward from the imaging unit 110a by a distance d and slightly deviating downward from an optical axis <NUM> of the imaging unit 110a is set as a waypoint <NUM>.

At this time, as illustrated in <FIG>, an image <NUM> of the virtual object of the tower and an image <NUM> of the waypoint are displayed on a display screen <NUM> of the user terminal <NUM>. Further, a guide surface <NUM> connecting the user terminal <NUM> and the image <NUM> of the waypoint to each other is displayed on the display screen <NUM>. The user can set a waypoint <NUM> by performing a touch operation on the display screen <NUM> or the like while viewing the image <NUM> of the waypoint arranged on the guide surface <NUM>.

At this time, since the waypoint <NUM> is positioned to be lower than the optical axis <NUM> of the imaging unit 110a, it is considered that the user can more easily recognize the position of the waypoint <NUM>, referring to the display screen <NUM>.

The method of setting a waypoint by operating the display screen <NUM> has been described above. Next, a variation on the method of setting a waypoint will be described with reference to <FIG> and <FIG>. Specifically, a method of setting a waypoint using a designation object that designates a route of the moving body <NUM> will be described.

<FIG> is a diagram illustrating a state in which a waypoint is set using a designation bar <NUM>. <FIG> is a diagram illustrating a display screen <NUM> when the waypoint is set by the designation bar <NUM>.

In the present embodiment, a spherical designation object <NUM> is provided at a tip of the designation bar <NUM>. Here, the designation bar <NUM> may be a touch pen or the like that can perform various operations by touching a user terminal <NUM>. In addition, it is assumed that the user terminal <NUM> includes a sensor capable of detecting a three-dimensional position of the designation object <NUM> included in the designation bar <NUM>. Specifically, the user terminal <NUM> includes, for example, a distance measuring sensor such as a ToF sensor or a stereo camera. The user terminal <NUM> acquires position information on the designation object <NUM> on the basis of sensor information of the distance measuring sensor. The position information on the designation object <NUM> is expressed in a three-dimensional manner as (x, y, z). Here, z is a direction of gravity (vertical direction). The x and y directions are orthogonal to each other while being perpendicular to the z direction.

Here, a waypoint is set on the basis of the position information. Specifically, for example, when the user performs a touch operation or the like on the display screen <NUM>, the user terminal <NUM> sets the position of the designation object <NUM> as a waypoint.

At this time, an image <NUM> of the designation bar and an image <NUM> of the designation object are displayed on the display screen <NUM> of the user terminal <NUM>. Therefore, the user can set a waypoint while checking the position of the designation object <NUM> through the display screen <NUM>.

The operation for generating movement information (more specifically, information including waypoints) for controlling a movement of the moving body <NUM> has been described above. Next, two methods of generating imaging information for controlling an imaging range of the imaging device included in the moving body <NUM> will be described with reference to <FIG> and <FIG>. <FIG> is a diagram illustrating a state in which an orientation of the imaging device of the moving body <NUM> is set by shifting an orientation of a user terminal 10i. In addition, <FIG> is a diagram illustrating a state in which an angle of view of the imaging device of the moving body <NUM> is set by performing a pinch operation on a display screen of a user terminal 10j.

First, a method of setting an imaging direction of the imaging device of the moving body <NUM> will be described with reference to <FIG>. For example, the user selects one of the waypoints included in the route of the moving body <NUM>. In this state, as illustrated in <FIG>, the user can adjust an imaging direction (that is, an imaging range 134a and 134b) of the imaging unit 110a included in the user terminal 10i by shifting an orientation of the user terminal 10i. Direction information regarding the adjusted imaging direction of the imaging unit 110a is generated as imaging information. That is, the user terminal 10i can generate the direction information on the basis of posture information on the user terminal 10i. On the basis of the direction information, the imaging device of the moving body <NUM> can capture an image in the same direction as the adjusted direction of the imaging unit 110a at the set waypoint.

In addition, the user terminal <NUM> according to the present embodiment can generate angle-of-view information for controlling an angle of view of the imaging device of the moving body <NUM> on the basis of a pinch-out operation or a pinch-in operation on the display screen by the user.

Next, a method of setting an angle of view of the imaging device of the moving body <NUM> will be described with reference to <FIG>. For example, the user selects one of the waypoints included in the route of the moving body <NUM>. In this state, the user can adjust an imaging range <NUM> of the imaging unit 110a to set an angle of view of the imaging device when the moving body <NUM> passes through the selected waypoint by performing a pinch-out operation or a pinch-in operation on the display screen of the user terminal 10j. That is, the user terminal 10j can generate angle-of-view information for controlling an angle of view of the imaging device of the moving body <NUM> on the basis of a pinch-out operation or a pinch-in operation on the display screen by the user.

The generation of the direction information and the angle-of-view information by the user terminal <NUM> on the basis of the operation of the user has been described above. The imaging device of the moving body <NUM> can capture an image on the basis of the direction information and the angle-of-view information. Note that the direction information and angle-of-view information described above may be generated at the time of setting the position of the waypoint, or may be generated after the position of the waypoint is set.

Next, a simulation of an operation of the moving body <NUM> by the user terminal <NUM> based on the movement information and the imaging information will be described with reference to <FIG> and <FIG>. Specifically, the user terminal <NUM> simulates a movement of the moving body <NUM> and an image to be captured by the imaging device included in the moving body <NUM>. <FIG> is a diagram illustrating a display screen <NUM> displaying a result of simulating the movement of the moving body <NUM>. In addition, <FIG> is a diagram illustrating a display screen <NUM> displaying a result of simulating the image to be captured by the imaging device included in the moving body <NUM>.

As illustrated in <FIG>, an image 612i of the virtual object of the tower arranged on the desk and an image <NUM> of the moving body simulatively shown as a triangle are displayed on the display screen <NUM> of the user terminal <NUM>. When a simulation of a movement of the image <NUM> of the moving body is started, the image <NUM> of the moving body moves along a virtual route <NUM> formed by connecting images 616a to <NUM> of waypoints to each other. By checking the movement of the image <NUM> of the moving body, the user can predict how the moving body <NUM> will actually move.

Furthermore, a user terminal <NUM> according to the present embodiment can also simulate an image to be captured by the imaging device included in the moving body <NUM>. Specifically, in a case where the moving body <NUM> has ever flied along the waypoints set as described above, the user terminal <NUM> can display an image predicted to be captured by the imaging device of the moving body <NUM>. As illustrated in <FIG>, an image predicted to be captured is displayed on the display screen <NUM> of the user terminal <NUM>. By viewing the display screen <NUM>, the user can predict an image to be captured by the imaging device included in the moving body <NUM>. Note that the user can also stop a moving image displayed on the display screen <NUM> by touching a stop button <NUM> shown at the center of the display screen <NUM> or doing the like.

Hereinafter, a method of imaging a landscape using the moving body <NUM> will be described. First, three methods of imaging a landscape without using the above-described technology of the present disclosure will be described. Thereafter, a method of imaging a landscape using the moving body <NUM> according to the technology of the present disclosure will be described.

Note that, in the following description, it is assumed that the moving body <NUM> (for example, a drone) is caused to fly around a construction such as a tower and capture an impressive video, for example, for commercial use. In such a case, the moving body <NUM> needs to fly in a three-dimensional space. The capturing of the impressive video is possible only when various conditions such as a position, a speed, and a camera orientation of the moving body <NUM> for flight are appropriately controlled. Therefore, in order to capture an impressive video, an advanced technique for steering the moving body <NUM> is required.

In addition, it is difficult for the user to manually operate the moving body <NUM> to fly along the same trajectory repeatedly. Furthermore, in a case where an image is captured outdoors or in a case where an image is captured in a wide range, it is necessary to consider daylight conditions, entrance and exit of people, and the like, and thus, an imaging timing is important. Therefore, a video with good conditions is obtained by repeating the capturing of the image.

First, a method in which the user steers the moving body <NUM> by manual operation using a steering device such that the imaging device of the moving body <NUM> images a landscape will be described with reference to <FIG>. Here, it is assumed that the movement of the moving body <NUM>, the orientation of the imaging device included in the moving body <NUM>, and the like are operated by the steering device. <FIG> is a flowchart illustrating a manual operation-based imaging method. Hereinafter, the manual operation-based imaging method will be described in line with the flowchart illustrated in <FIG>.

First, the user checks a difference in image according to imaging conditions (Step S101). More specifically, the user manually operates the moving body <NUM> to actually fly around a construction to check a difference in how the image is viewed according to the imaging conditions such as an orientation of the imaging device of the moving body <NUM> or a distance between the moving body <NUM> and the construction. Note that the user is preferably a person who is accustomed to steering the moving body <NUM>.

Next, the user captures a video using the moving body <NUM> (Step S103). More specifically, the user controls a flight of the moving body <NUM> and an orientation of the imaging device by manual operation to capture an impressive video, such that the video is captured by the imaging device of the moving body <NUM>.

At this time, the user may cause any known type of mobile terminal such as a tablet terminal to display a two-dimensional map screen together with a video that is being captured by the imaging device included in the moving body <NUM>, thereby displaying a route along which the moving body <NUM> is flying. Further, the user may set a waypoint in the route on the basis of a predetermined rule. Thus, the user can set the waypoint while checking the video that is being captured.

Next, the user checks the video captured in Step S103 (Step S105). In a case where the flight of the moving body <NUM> and the orientation of the imaging device have been controlled as intended (Step S107: YES), the process proceeds to Step S109. On the other hand, in a case where the flight of the moving body <NUM> and the orientation of the imaging device have not been controlled as intended (Step S107: NO), the process returns to Step S103.

Even though the flight of the moving body <NUM> and the orientation of the imaging device have been controlled as intended (Step S107: YES), in a case where the imaging device fails to capture an impressive video as intended because, for example, there has been a timing at which the sun is hidden or an unintended person has crossed in front of the imaging device (Step S109: NO), the process returns to Step S103. On the other hand, in a case where an impressive video has been captured as intended (Step S109: YES), the imaging method illustrated in <FIG> ends.

The method of capturing a video by manual operation has been described above. According to such a method, in order to obtain an intended video, it is necessary to repeatedly reproduce the same flight of the moving body <NUM> and the same orientation of the imaging device by manual operation. Therefore, a lot of effort and time are required to obtain a desired video, and a lot of manpower is needed every time a video is captured.

Next, a method in which, while the moving body <NUM> is caused to automatically fly, the imaging device captures a video will be described with reference to <FIG> is a flowchart illustrating a method of causing the imaging device to capture a video by causing the moving body <NUM> to automatically fly. Hereinafter, the description will be given in line with the flowchart illustrated in <FIG>.

First, processing in Steps S201 to S207 is performed. Since the processing in Steps S201 to S207 is substantially the same as that in Steps S101 to S107, the description thereof is omitted here.

In a case where the flight of the moving body <NUM> and the orientation of the imaging device have been controlled as intended (Step S207: YES), data on the imaging conditions is stored (Step S209). More specifically, when the flight of the moving body <NUM> and the orientation of the imaging device have been controlled as intended, various imaging conditions such as a position, a speed, and an imaging device orientation of the moving body <NUM> during flight are recorded. The imaging conditions are recorded in any know type of recording medium or the like included in the moving body <NUM>. Note that information regarding the position, the speed, or the like of the moving body <NUM> is acquired by a GPS, an IMU, or the like included in the moving body <NUM>.

Next, an imaging operation is reproduced (Step S211). More specifically, the flight of the moving body <NUM> and the orientation of the imaging device are automatically reproduced on the basis of the imaging conditions recorded in Step S209. A video captured at this time is checked by the user.

In a case where the video has not been captured as intended (Step S213: NO), the process returns to Step S211, and the imaging operation is reproduced again. On the other hand, in a case where the video has been captured as intended (Step S213: YES), for example, when daylight conditions and the like are met, the capturing of the image illustrated in <FIG> ends.

The automatic flight-based imaging method has been described above. According to such a method, the user does not need to manually operate the same moving body <NUM>, and thus, the burden on the user is reduced.

Note that, on the basis of the data recorded in Step S209, a map can be displayed on a display screen of, for example, a tablet terminal or the like to depict a virtual route of the moving body <NUM> on the map, such that the virtual route is minutely adjusted. However, in a case where the virtual route is displayed on a two-dimensional map screen, it is difficult to intuitively adjust, for example, an altitude in the virtual route.

Furthermore, since the self-position of the moving body <NUM> is calculated using a global positioning system (GPS), an IMU, or the like, an error of about <NUM> to <NUM> occurs in its position relative to a construction or the like, depending on the GPS.

In the above-described methods, the user needs to go to a site where the moving body <NUM> actually flies in order to steer the moving body <NUM> and set waypoints. At that point, a method may be considered in which a route along which the moving body <NUM> flies is designated in advance using a map or the like displayed on a display screen of a tablet terminal or the like, such that the moving body <NUM> flies along the designated route.

More specifically, while a map is displayed on a display screen of, for example, a tablet terminal, the user sets waypoints by touching the display screen. Note that a position of a waypoint may be set on the basis of longitude and latitude. At this time, the user may set a speed and an altitude of the moving body <NUM> at a designated waypoint. Further, the user can also set an orientation of the moving body <NUM> at a designated waypoint. For example, the moving body <NUM> can be set to be oriented in a traveling direction.

A plurality of waypoints are set in the same manner, and a route of the moving body <NUM> is set by connecting the waypoints to one another. The user can record or transmit information indicating the set route in or to the moving body <NUM>, such that the imaging device captures a video while the moving body <NUM> flies along the set route.

According to such a method, the user can set a route along which the moving body <NUM> flies before going to a site where the moving body <NUM> flies. Therefore, the user can set a route along which the moving body <NUM> flies while staying at an office, home, or the like. However, it cannot be seen what video will be captured by the imaging device of the moving body <NUM> until the moving body <NUM> actually flies and the imaging device captures the video. Further, in a case where a position of a waypoint has been corrected, it cannot be seen how a video to be captured by the imaging device will change, unless the moving body <NUM> actually flies and the imaging device captures the video.

In addition, the method in which the waypoints are set by touching the map displayed on the display screen is convenient in a case where the moving body <NUM> roughly flies in a wide range. However, in a case where the imaging device of the moving body <NUM> dynamically captures an image around a construction, it is considered difficult to minutely set a route and the like of the moving body <NUM>. Furthermore, since the two-dimensional map is displayed on the display screen, it is necessary to set an altitude of the moving body <NUM> as a numerical value, and thus, it is not possible to intuitively set a waypoint.

Next, an imaging method according to the present disclosure will be described with reference to <FIG>. <FIG> is a flowchart illustrating a procedure until a virtual object is generated. In addition, <FIG> is a flowchart illustrating a procedure until a video is captured on the basis of generated movement information and imaging information. In addition, <FIG> is a diagram illustrating displaying processing by the information processing apparatus <NUM>. Hereinafter, the imaging method according to the present disclosure will be described with reference to <FIG>. In the following description, <FIG>, which have been described above, will be appropriately referred to.

An operation of the user in Step S301 illustrated in <FIG> is substantially the same as that in Step S101. However, in the imaging method according to the present disclosure, the user steering the moving body <NUM> may be a person who is not accustomed to steering the moving body <NUM>.

Next, the user causes the imaging device of the moving body <NUM> to capture a video (Step S303). The user causes the imaging device of the moving body <NUM> to capture an image of a subject on which a virtual object is based. At this time, the user may control a flight of the moving body <NUM> and an orientation of the imaging device by manual operation to capture an impressive video, such that the video is captured by the imaging device of the moving body <NUM>. For example, the user may cause the moving body <NUM> to turn around the tower <NUM> as illustrated in <FIG> such that the imaging device <NUM> captures a video. At this time, it is assumed that the imaging device <NUM> captures a video including the tower <NUM> and the forest <NUM>.

Next, the user checks the captured video (Step S305). More specifically, the user checks that the video captured by the imaging device <NUM> is an intended video.

Next, a virtual object is generated (Step S307). More specifically, information regarding a three-dimensional virtual object is generated using various known CG technologies on the basis of information such as the image captured in Step S303, and the position and the posture of the moving body <NUM> and the orientation of the imaging device <NUM> at the time of capturing the image. For example, information regarding the virtual object <NUM> of the tower illustrated in <FIG> and the virtual object <NUM> of the forest illustrated in <FIG> is generated.

Note that, in the processing for generating a three-dimensional virtual object on the basis of the captured image, more accurate values of the position and the posture of the moving body <NUM> may be calculated not only by using the GPS and the IMU but also through bundle adjustment. Accordingly, a position or a posture of the moving body <NUM> relative to an environment such as a construction is more accurately calculated. Here, the bundle adjustment is a method of estimating various parameters with high accuracy from an image. The information regarding the virtual object generated at this time is recorded in the storage unit <NUM> included in the user terminal <NUM>. At this time, information regarding the route <NUM> and the waypoints <NUM> along which the moving body <NUM> has moved as illustrated in <FIG> may be recorded in the storage unit <NUM>.

The processing until the virtual object is generated has been described above with reference to <FIG>. Next, a procedure until a desired video is captured will be described with reference to <FIG> and <FIG>. Note that processing in Steps S401 to S405 illustrated in <FIG> and <FIG> is mainly performed by the information processing apparatus <NUM> according to an embodiment of the present disclosure.

The information processing apparatus <NUM> performs processing for displaying a virtual object (Step S401). The processing for displaying a virtual object will be described with reference to <FIG> is a flowchart illustrating the processing for displaying a virtual object. Hereinafter, the processing for displaying a virtual object will be described in line with the flowchart illustrated in <FIG>. The processing illustrated in <FIG> is executed, for example, when the start button <NUM> displayed on the display screen <NUM> of the user terminal 10a is touched as described with reference to <FIG>.

First, the acquisition unit <NUM> acquires image information and sensor information (Step S501). More specifically, the acquisition unit <NUM> acquires image information including the desk <NUM> imaged by the imaging unit <NUM>. In addition, the acquisition unit <NUM> acquires IMU information or information on the distance from the user terminal 10a to the desk <NUM>, or the like detected by the sensor unit <NUM>. The acquisition unit <NUM> transmits the acquired image information and distance information to the detection unit <NUM> included in the processing unit <NUM>. In addition, the acquisition unit <NUM> transmits the acquired image information, distance information, and IMU information to the self-position calculation unit <NUM> included in the processing unit <NUM>.

Next, the plane detection unit <NUM> detects a plane on the basis of the image information and the distance information transmitted from the acquisition unit <NUM> (Step S503). Here, the plane detection unit <NUM> detects a flat plane <NUM> on the desk <NUM>. The plane detection unit <NUM> transmits a detection result to the virtual object calculation unit <NUM>.

Next, the self-position calculation unit <NUM> calculates a self-position of the user terminal <NUM> on the basis of the image information, the distance information, and the IMU information (Step S505). More specifically, the self-position calculation unit <NUM> calculates a position and a posture of the user terminal <NUM> with respect to the desk <NUM> or an environment on the periphery thereof. The self-position calculation unit <NUM> transmits a calculation result to the virtual object calculation unit <NUM>.

Next, the virtual object calculation unit <NUM> calculates a position, an orientation, a scale, and the like of a virtual object to be arranged on the basis of the calculation result of the self-position calculation unit <NUM> and the information regarding the virtual object recorded in the storage unit <NUM> (Step S507). The virtual object calculation unit <NUM> transmits a calculation result to the movement information generation unit <NUM>.

Next, the movement information generation unit <NUM> sets a route of the moving body <NUM> on the basis of the calculation result of the virtual object calculation unit <NUM> and the waypoint information recorded in the storage unit <NUM> (Step S509). For example, the movement information generation unit <NUM> sets a virtual route to turn around the virtual object arranged on the desk <NUM>. The movement information generation unit <NUM> transmits information on the set virtual route to the display information generation unit <NUM>.

Next, the display information generation unit <NUM> generates display information (Step S511). More specifically, the display information generation unit <NUM> generates display information for displaying the virtual route of the moving body <NUM> around the virtual object arranged on the desk <NUM>, and transmits the generated display information to the display control unit <NUM>.

Next, the display control unit <NUM> controls a display of the display unit <NUM> so that an image of the virtual route around the virtual object arranged on the desk <NUM> is displayed (Step S513). Accordingly, an image <NUM> of the virtual object <NUM> of the tower on the desk <NUM> existing before the eyes of the user and an image <NUM> of the virtual route that turns therearound are displayed on the display screen of the display unit <NUM>.

The processing for displaying a virtual object has been described above with reference to <FIG>. Next, referring back to <FIG>, the imaging method according to the present disclosure will be described.

The information processing apparatus <NUM> generates movement information and imaging information (Step S403). For example, as described with reference to <FIG>, movement information such as waypoints and imaging information such as an orientation and a zoom factor of the imaging device are generated on the basis of the operation by the user to move the user terminal <NUM>, and the generated information is transmitted to the prediction unit <NUM>. Here, processing of the information processing apparatus <NUM> in the operation described with reference to <FIG> will be described.

Here, processing of the information processing apparatus <NUM> in the processing for adjusting a waypoint as described with reference to <FIG> will be described. As illustrated in <FIG>, when the user U2 touches the image 616a of the waypoint displayed on the display screen 610a, the input unit <NUM> transmits, to the movement information generation unit <NUM>, input information indicating that the waypoint 408a corresponding to the image 616a of the waypoint has been selected.

Next, as illustrated in <FIG> and <FIG>, when the user U2 pulls the user terminal <NUM> toward the user U2, the sensor unit <NUM> detects the movement of the user terminal <NUM> and transmits the detected sensor information to the self-position calculation unit <NUM>. The self-position calculation unit <NUM> calculates a position and a posture of the user terminal <NUM> on the basis of the sensor information. The self-position calculation unit <NUM> transmits a calculation result to the movement information generation unit <NUM>.

Next, the movement information generation unit <NUM> corrects the virtual route of the moving body <NUM> so that the position of the selected waypoint 408a is displaced as much as a distance by which the user terminal <NUM> has moved. Accordingly, information regarding the new virtual route is generated as movement information and transmitted to the prediction unit <NUM>.

Next, processing performed by the information processing apparatus <NUM> in the operation described with reference to <FIG> or <FIG> will be described. The acquisition unit <NUM> acquires, from the input unit <NUM>, input information based on an operation on the display screen <NUM> by the user. In addition, the acquisition unit <NUM> acquires image information from the imaging unit <NUM>, and distance information and IMU information from the sensor unit <NUM>. The acquisition unit <NUM> transmits the acquired information to the processing unit <NUM>.

The self-position calculation unit <NUM> calculates a self-position of the user terminal <NUM> on the basis of the transmitted sensor information, distance information, or the like, and transmits a calculation result to the generation unit <NUM>. The movement information generation unit <NUM> specifies a position of a waypoint on the basis of the calculation result, the input information, and the like. The movement information generation unit <NUM> sets a virtual route of the moving body <NUM> by connecting a plurality of specified waypoints to one another, and transmits the virtual route to the prediction unit <NUM> as movement information.

Next, processing of the information processing apparatus <NUM> when the waypoint is set using the designation bar <NUM> as described with reference to <FIG> and <FIG> will be described. First, the acquisition unit <NUM> acquires IMU information and sensor information from the sensor unit <NUM>. Further, the acquisition unit <NUM> acquires image information from the imaging unit <NUM> and transmits the image information to the object detection unit <NUM>.

The object detection unit <NUM> detects a designation object <NUM> included in an image on the basis of the image information. Further, the object detection unit <NUM> detects, for example, a distance and a direction from the imaging unit <NUM> to the designation object <NUM> on the basis of the sensor information, and transmits a detection result to the generation unit <NUM>.

The movement information generation unit <NUM> specifies a position of the designation object <NUM> on the basis of the detection result of the object detection unit <NUM>, and sets the position as a waypoint. The movement information generation unit <NUM> sets a virtual route by connecting a plurality of set waypoints to one another, and transmits the virtual route to the prediction unit <NUM> as movement information.

Next, processing of the information processing apparatus <NUM> when the orientation of the imaging device is set as described with reference to <FIG> will be described. The self-position calculation unit <NUM> calculates a posture of the user terminal <NUM> on the basis of the sensor information, and transmits a calculation result to the generation unit <NUM>. The imaging information generation unit <NUM> sets an orientation of the imaging device on the basis of the calculated posture. The imaging information generation unit <NUM> transmits the set orientation of the imaging device to the prediction unit <NUM> as direction information.

Next, processing of the information processing apparatus <NUM> when the angle of view of the imaging device is set as described with reference to <FIG> will be described.

The imaging information generation unit <NUM> acquires, from the input unit <NUM>, input information indicating that a pinch-in operation or a pinch-out operation has been performed by the user. The imaging information generation unit <NUM> generates angle-of-view information indicating an angle of view of the imaging device on the basis of the input information, and transmits the angle-of-view information to the prediction unit <NUM>.

Next, the information processing apparatus <NUM> performs a simulation of a movement of the moving body <NUM> and a video to be captured by the imaging device (Step S405). For example, the prediction unit <NUM> simulates a movement of the moving body <NUM> on the display screen <NUM>. Specifically, the movement prediction unit <NUM> predicts a movement of the moving body <NUM> on the basis of the movement information, and transmits a prediction result to the display information generation unit <NUM>.

Furthermore, the prediction unit <NUM> simulates a video to be captured. More specifically, the imaging prediction unit <NUM> predicts a video to be captured by the moving body <NUM> on the basis of the movement information and the imaging information, and transmits a prediction result to the display information generation unit <NUM>.

Next, the display unit <NUM> displays the prediction result (Step S407). More specifically, the display information generation unit <NUM> generates display information for displaying the prediction result on the basis of the prediction result, and transmits the display information to the display control unit <NUM>. The display control unit <NUM> controls a display of the display unit <NUM> so that the display unit <NUM> displays the prediction result on the basis of the display information. Accordingly, the prediction result is displayed on the display unit <NUM>. More specifically, the prediction result of the movement of the moving body <NUM> as illustrated in <FIG> or the prediction result of the video to be captured by the imaging device of the moving body <NUM> as illustrated in <FIG> is displayed.

Next, in a case where the simulation has been performed as intended (Step S409: YES), the process proceeds to Step S411. At this time, the storage unit <NUM> may store the movement information and the imaging information that have been used for the simulation. On the other hand, when the simulation has not been performed as intended (Step S409: NO), the process returns to Step S403.

Next, movement of the moving body <NUM> and image capturing by the imaging device are performed (Step S411). For example, the virtual route of the moving body <NUM> formed in Step S403 is converted into a real route along which the moving body <NUM> actually moves by being converted into a coordinate system of a real space by the movement information generation unit <NUM>. Information regarding the real route is transmitted to the moving body <NUM> by the communication control unit <NUM>. While the moving body <NUM> is flying along the generated real route, the imaging device images a landscape on the basis of the imaging information.

In a case where an impressive video as intended has been captured (Step S413: YES), the imaging processing illustrated in <FIG> ends. On the other hand, in a case where an impressive video as intended has not been captured (Step S413: NO), the process returns to Step S411.

The imaging method according to the present disclosure has been described above. The information processing apparatus <NUM> according to the present disclosure controls a display of a virtual object based on an object existing in a real space on a display screen, and generates movement information for controlling a movement of a moving body. Accordingly, in a case where a user desires to cause the moving body <NUM> such as a drone to fly around the object existing in the real space, on which the virtual object is based, the user can designate a route of the moving body <NUM> while viewing the virtual object. Therefore, the information processing apparatus <NUM> according to the present embodiment can more intuitively generate movement information for controlling a movement of the moving body <NUM>.

In addition, in the information processing apparatus <NUM> according to the present embodiment, the movement information generation unit <NUM> generates the movement information on the basis of an operation by the user U2 viewing the display screen <NUM>. The user U2 can designate movement information such as a route of the moving body <NUM> while viewing the virtual object displayed on the display screen <NUM>. Therefore, it is possible to more intuitively generate movement information for controlling a movement of the moving body <NUM>.

In addition, in the present embodiment, as illustrated in <FIG>, <FIG>, etc., the display of the display screen <NUM> includes an image <NUM> of the virtual route of the moving body <NUM>. Therefore, it becomes easier for the user U2 to imagine a route of the moving body <NUM>.

In addition, in the present embodiment, an image <NUM> of at least one waypoint (adjustment portion) for adjusting the route of the moving body <NUM> is displayed in at least a part of the image <NUM> of the virtual route displayed on the display screen <NUM>. The movement information generation unit <NUM> generates movement information on the basis of an operation for moving the image <NUM> of the waypoint. Therefore, the user U2 can designate a route of the moving body <NUM> only by moving the image <NUM> of the waypoint as a marker of the route, thereby making it possible to more intuitively generate movement information for controlling a movement of the moving body <NUM>.

In addition, in the present embodiment, as described with reference to <FIG>, the movement information generation unit <NUM> shifts a position of the waypoint <NUM> on the basis of an operation for shifting a position of the display screen <NUM>. Therefore, the user can more intuitively designate a route of the moving body <NUM>.

In addition, in the present embodiment, the image <NUM> of the virtual object is displayed to be superimposed on an image captured by an imaging unit <NUM> included in a user terminal <NUM>. Accordingly, the user U2 can recognize the virtual object 422a as if it existed in the real space. Therefore, the user U2 can more intuitively designate a route of the moving body <NUM>. In addition, by matching a viewpoint of the imaging unit <NUM> with a waypoint, an image captured at the waypoint can be displayed on the display screen. Therefore, when the waypoint has changed in position, it is also possible to predict in advance how an image to be captured will change.

In addition, in the present embodiment, as described with reference to <FIG>, the movement information generation unit <NUM> generates the movement information on the basis of an operation by the user U2 for shifting a viewpoint of the imaging unit <NUM>. More specifically, the movement information generation unit <NUM> generates the movement information on the basis of a predetermined shift of the viewpoint in position. The display screen <NUM> includes an image captured by the imaging unit <NUM>. Therefore, when the moving body <NUM> actually moves, it becomes easier for the user U2 to imagine a landscape to be captured by the imaging device included in the moving body <NUM>, thereby making it possible to cause the imaging device included in the moving body <NUM> to capture a more desired video.

Note that the route of the moving body <NUM> may be a route from the viewpoint of the imaging unit <NUM> as described with reference to <FIG>. In addition, the route of the moving body <NUM> may be positioned away from the viewpoint of the imaging unit <NUM> by a predetermined distance. For example, as described with reference to <FIG>, a position away forward from the viewpoint of the imaging unit <NUM> by a distance d and lower than an optical axis of the imaging unit <NUM> within an angle of view may be designated as the route of the moving body <NUM>. In this case, as illustrated in <FIG>, the waypoint or the like is displayed on the display screen <NUM>, such that the user U2 can more intuitively designate a route of the moving body <NUM>.

In addition, in the present embodiment, as described with reference to <FIG> and <FIG>, the movement information generation unit <NUM> generates the movement information on the basis of an operation for moving a designation object designating a route of the moving body <NUM>. More specifically, in the present embodiment, the route of the moving body <NUM> is designated on the basis of a route along which the designation object <NUM> provided at a tip of a designation bar <NUM> has moved. Thus, the user can designate a route of the moving body <NUM> by a simple operation for moving the designation object <NUM>. In addition, as illustrated in <FIG>, an image <NUM> of the designation object <NUM> is displayed on the display screen <NUM>. Accordingly, the user U2 can recognize a position of the designation object <NUM> via the display screen <NUM>. Therefore, it becomes easier for the user U2 to imagine a route of the moving body <NUM>.

In addition, in the present embodiment, the moving body <NUM> includes an imaging device. The imaging device captures a landscape around the moving body <NUM>. In addition, the information processing apparatus <NUM> according to the present embodiment includes an imaging information generation unit <NUM> generating imaging information for controlling an imaging range of the imaging device included in the moving body <NUM> on the basis of an operation of the user. Therefore, the user can designate an imaging range of the imaging device included in the moving body <NUM> on the basis of various operations, thereby making it possible to cause the imaging device of the moving body <NUM> to capture a more appropriate video.

Furthermore, in the present embodiment, the imaging information generation unit <NUM> generates direction information regarding an imaging direction of the imaging device included in the moving body <NUM> as the imaging information. Therefore, the user can cause the imaging device of the moving body <NUM> to capture a more appropriate video.

In addition, in the present embodiment, an image captured by the imaging unit <NUM> is displayed on the display screen. In addition, as described with reference to <FIG>, the imaging information generation unit <NUM> generates the direction information on the basis of an operation for shifting an orientation of the imaging unit <NUM>. Therefore, the user can generate direction information while guessing a video to be captured by the imaging device of the moving body <NUM>, thereby making it possible to cause the imaging device to capture a more appropriate video.

In addition, in the present embodiment, the imaging information generation unit <NUM> can generate angle-of-view information for controlling an angle of view of the imaging device of the moving body <NUM> as the imaging information on the basis of a pinch-out operation or a pinch-in operation on the display screen by the user U2. Therefore, the user can easily designate an imaging range of the imaging device of the moving body <NUM>.

In addition, in the present embodiment, the information processing apparatus <NUM> further includes a movement prediction unit <NUM> predicting a movement of the moving body <NUM> on the basis of the movement information. More specifically, the movement prediction unit <NUM> can simulate a movement of the moving body <NUM>. Therefore, the user can check a route of the moving body <NUM> in advance on the basis of the simulation of the movement of the moving body <NUM>. In the present embodiment, as described with reference to <FIG>, a result of simulating the movement of the moving body <NUM> (that is, a prediction result) is displayed on the display screen <NUM>. Therefore, the user can more easily check a route of the moving body <NUM> by viewing the display screen <NUM>.

In addition, in the present embodiment, the information processing apparatus <NUM> further includes an imaging prediction unit <NUM> predicting an image to be captured by the imaging device of the moving body <NUM> on the basis of the movement information and the imaging information. In the present embodiment, the imaging prediction unit <NUM> can simulate an image to be captured by the imaging device. The user can check an image to be captured on the basis of a result of the simulation. In addition, in the present embodiment, as described with reference to <FIG>, the result of the simulation by the imaging prediction unit <NUM> is displayed on the display screen <NUM>. Therefore, the user can easily check a prediction result of the imaging prediction unit <NUM>.

In addition, in the present embodiment, the moving body <NUM> is three-dimensionally movable. Accordingly, the user can three-dimensionally designate a route of the moving body <NUM>. Therefore, the user can more intuitively generate movement information for controlling a movement of the moving body <NUM>.

Furthermore, according to the present embodiment, once the route of the moving body <NUM> is set, it is possible to cause the moving body <NUM> to fly along the same route repeatedly without manpower, or cause the imaging device to capture similar videos repeatedly.

Next, an example of a hardware configuration of the user terminal <NUM> constituting the information processing system <NUM> according to an embodiment of the present disclosure, such as the user terminal <NUM> described above, will be described in detail with reference to <FIG>.

<FIG> is a functional block diagram illustrating a configuration example of a hardware configuration of the user terminal <NUM> constituting the information processing system <NUM> according to an embodiment of the present disclosure.

The user terminal <NUM> constituting the information processing system <NUM> according to the present embodiment mainly includes a CPU <NUM>, a ROM <NUM>, and a RAM <NUM>. In addition, the user terminal <NUM> further includes a host bus <NUM>, a bridge <NUM>, an external bus <NUM>, an interface <NUM>, an input device <NUM>, an output device <NUM>, a storage device <NUM>, a drive <NUM>, a connection port <NUM>, and a communication device <NUM>.

The CPU <NUM> functions as an arithmetic processing device and a control device, and controls an overall operation or a partial operation in the user terminal <NUM> in accordance with various programs recorded in the ROM <NUM>, the RAM <NUM>, the storage device <NUM>, or a removable recording medium <NUM>. The ROM <NUM> stores programs, operation parameters, and the like used by the CPU <NUM>. The RAM <NUM> primarily stores programs used by the CPU <NUM>, parameters appropriately changing when executing the programs, and the like. They are connected to each other by the host bus <NUM> configured as an internal bus such as a CPU bus. For example, the acquisition unit <NUM>, the processing unit <NUM> (each functional unit illustrated in <FIG>), the display control unit <NUM>, and the communication control unit <NUM> illustrated in <FIG> can be configured by the CPU <NUM>.

The host bus <NUM> is connected to the external bus <NUM> such as a peripheral component interconnect/interface (PCI) bus via the bridge <NUM>. In addition, the input device <NUM>, the output device <NUM>, the storage device <NUM>, the drive <NUM>, the connection port <NUM>, and the communication device <NUM> are connected to the external bus <NUM> via the interface <NUM>.

The input device <NUM> is an operation means operated by the user, for example, a mouse, a keyboard, a touch panel, a button, a switch, a lever, a pedal, or the like. Alternatively, the input device <NUM> may be, for example, a remote control means (so-called remote controller) using infrared rays or other radio waves, or an external connection device <NUM> such as a mobile phone or a PDA corresponding to an operation of the user terminal <NUM>. In addition, the input device <NUM> includes an input control circuit or the like generating an input signal on the basis of information input by the user, for example, using the above-described operation means and outputting the input signal to the CPU <NUM>. By operating the input device <NUM>, the user of the user terminal <NUM> can input various kinds of data to the user terminal <NUM> and give processing operation instructions.

The output device <NUM> includes a device capable of visually or auditorily notifying the user of acquired information. Examples of such a device include display devices such as a CRT display device, a liquid crystal display device, a plasma display device, an EL display device, and a lamp, audio output devices such as a speaker and a headphone, printer devices, and the like. The output device <NUM> outputs, for example, results obtained by various types of processing performed by the user terminal <NUM>. Specifically, the display device displays the results obtained by various types of processing performed by the user terminal <NUM> as a text or an image. On the other hand, the audio output device converts an audio signal including reproduced audio data, acoustic data, or the like into an analog signal and outputs the analog signal.

The storage device <NUM> is a data storage device configured as an example of the storage unit of the user terminal <NUM>. The storage device <NUM> includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like. The storage device <NUM> stores programs executed by the CPU <NUM>, various kinds of data, and the like. For example, the storage unit <NUM> illustrated in <FIG> can be configured by the storage device <NUM>.

The drive <NUM> is a reader/writer for a recording medium, and is built in or externally attached to the user terminal <NUM>. The drive <NUM> reads out information recorded on the mounted removable recording medium <NUM> such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the information to the RAM <NUM>. Furthermore, the drive <NUM> can also write a record on the mounted removable recording medium <NUM> such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. The removable recording medium <NUM> is, for example, a DVD medium, an HD-DVD medium, a Blu-ray (registered trademark) medium, or the like. Alternatively, the removable recording medium <NUM> may be a compact flash (CF) (registered trademark), a flash memory, a secure digital (SD) memory card, or the like. Alternatively, the removable recording medium <NUM> may be, for example, an integrated circuit (IC) card on which a non-contact IC chip is mounted, an electronic device, or the like.

The connection port <NUM> is a port for direct connection to the user terminal <NUM>. Examples of the connection port <NUM> include a universal serial bus (USB) port, an IEEE <NUM> port, a small computer system interface (SCSI) port, and the like. Other examples of the connection port <NUM> include an RS-232C port, an optical audio terminal, a high-definition multimedia interface (HDMI) (registered trademark) port, and the like. By connecting the external connection device <NUM> to the connection port <NUM>, the user terminal <NUM> directly acquires various kinds of data from the external connection device <NUM> or provide various kinds of data to the external connection device <NUM>.

The communication device <NUM> is, for example, a communication interface including a communication device or the like for connection to a communication network (network) <NUM>. The communication device <NUM> is, for example, a communication card or the like for wired or wireless local area network (LAN), Bluetooth (registered trademark), or wireless USB (WUSB). Alternatively, the communication device <NUM> may be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), any type of modem for communication, or the like. For example, the communication device <NUM> can transmit and receive signals and the like to and from the Internet and other communication devices according to a predetermined protocol such as TCP/IP. In addition, the communication network <NUM> connected to the communication device <NUM> includes a network or the like connected in a wired or wireless manner, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, or the like.

An example of the hardware configuration capable of implementing the functions of the user terminal <NUM> constituting the information processing system <NUM> according to an embodiment of the present disclosure has been described above. Each of the above-described components may be configured using a general-purpose member, or may be configured by hardware specific to the functions of each component. Therefore, it is possible to appropriately change the hardware configuration to be used according to the technical level at the time of carrying out the present embodiment. Note that, although not illustrated in <FIG>, it is obvious that various components corresponding to the user terminal <NUM> constituting the information processing system <NUM> are included.

Note that a computer program for implementing each function of the user terminal <NUM> constituting the information processing system <NUM> according to the present embodiment as described above can be created and installed on a personal computer or the like. In addition, a computer-readable recording medium storing such a computer program can also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Furthermore, the above-described computer program may be distributed, for example, via a network without using a recording medium. In addition, the number of computers executing the computer program is not particularly limited. For example, the computer program may be executed by a plurality of computers (for example, a plurality of servers or the like) in cooperation with each other.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited thereto. It is obvious to a person having ordinary knowledge in the technical field of the present disclosure that various changes or modifications can be made within the scope of the technical idea set forth in the claims, and it is of course to be understood that the changes or modifications also fall within the technical scope of the present disclosure.

For example, in the above-described embodiment, the user U1 causes the moving body <NUM> to fly in advance so that the imaging device <NUM> mounted on the moving body <NUM> captures an image for generating a virtual object, but the present technology is not limited thereto. For example, in a case where a virtual object has been generated in any method, for example, in a case where a virtual object has been generated on the basis of an image captured by a user other than the user U1 using the moving body <NUM> in the past, the already-generated virtual object may be used.

In addition, the moving body <NUM> has been described as a drone in the above-described embodiment, but the moving body <NUM> may be any movable device. For example, the technology of the present disclosure can also be applied to any kind of aerial vehicle that can fly like the drone. Furthermore, the technology of the present disclosure can also be applied to a manipulator corresponding to a hand, an arm, or the like of a robot. In this case, the information processing apparatus may control, for example, a display of a virtual object to be handled by the manipulator on the display screen. In addition, the information processing apparatus can generate movement information for controlling a movement of the moving body using, for example, a fingertip of the manipulator as the moving body. Accordingly, it is possible to more intuitively generate movement information for controlling a movement of the fingertip or the like of the manipulator.

In addition, in the above-described embodiment, information regarding a virtual object, a waypoint, and the like is recorded in the information processing apparatus <NUM>. Alternatively, the information regarding the virtual object, the waypoint, and the like may be recorded in various servers connected to the network. In this case, the information processing apparatus <NUM> can receive information recorded in an appropriate server via the network, and generate movement information, imaging information, and the like.

In addition, in the above-described embodiment, the description has been given assuming that the user terminal <NUM> is mainly a smartphone, a tablet terminal, or the like. Alternatively, the user terminal <NUM> may be a general-purpose personal computer (PC), a game machine, a robot, or a wearable device such as a head mounted display (HMD) or a smart watch.

Claim 1:
An information processing apparatus (<NUM>) for implementing functions of a user terminal (<NUM>), the information processing apparatus comprising:
a display control unit (<NUM>) that controls display of a virtual object (<NUM>, <NUM>) on a display screen (<NUM>) of the user terminal, the virtual object being based on an object (<NUM>, <NUM>) existing in a real space; and
a movement information generation unit (<NUM>) that generates movement information, on the basis of an operation by a user viewing the display screen, for controlling a movement of a moving body (<NUM>) with respect to the object in the real space, the moving body being communicably connected to the user terminal,
wherein the display includes a route of the moving body, characterized in that at least one adjustment portion for adjusting the route is displayed in at least a part of the route, and the operation is an operation for shifting a position of the adjustment portion displayed on the display screen.