Patent Description:
Still images and videos are used to introduce and promote goods or facilities in industries such as tourism, real estate, used cars and weddings. Further, in recent years, in addition to still images taken by a general-purpose camera, wide-angle images such as spherical images are used that enhance the feeling of actually being there. Furthermore, contents created by combining the wide-angle images and the still images with text information are used for the promotion or introduction of goods or services as described above. As a technique of capturing the wide-angle images for creating such contents, a technique of using an application installed in an information terminal such as a smartphone in combination with a celestial-sphere camera is known.

"<NPL> discloses a technique regarding generating contents using wide-angle images. The above video on the Internet discloses a technique of performing an operation of setting hot spots for switching scenes at desired locations in a wide-angle image, while a user is shooting photographs, in order to generate contents more efficiently.

However, according to well-known techniques, when creating contents by associating a still image with a wide-angle image, images captured by an external digital camera are to be copied to an information terminal and a desired image selected from an image folder in the information terminal is to be associated with the wide-angle image. Such capturing images with the external camera and copying the images to the information terminal take time and efforts. Further there is a possibility that an image wrongly selected from the image folder is associated with the wide-angle image, and therefore there is room for improvement.

<CIT> discloses an apparatus, system, and method, each of which: obtains a first image in a first projection, and a second image in a second projection; obtains area specification information specifying an area in the second image; and associates location information to be used for superimposing the second image on the first image with the area specification information, or generates location information based on the area specification information.

<CIT> discloses an information processing apparatus calculates a first corresponding area in a first image in first projection, corresponding to a second image in second projection, through first homography transformation; transforms a projection of a peripheral area including the first corresponding area to generate a peripheral area image; calculate a second corresponding area in the peripheral area image through second homography transformation; calculate a third corresponding area in the second image, corresponding to a third image in the second projection, through third homography transformation; reversely transform a projection of the second corresponding area to generate first location information based on a first specific corresponding area in the first image; calculate a fourth corresponding area in the peripheral area image through the second homography transformation; and reversely transform a projection of the fourth corresponding area to generate second location information based on a second specific corresponding area in the first image.

The invention provides an information processing method, carrier means, and apparatus according to claims <NUM>, <NUM>, and <NUM>, respectively.

A more complete appreciation of the embodiments and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings, wherein:.

However, the disclosure of this patent specification is not intended to be limited to the specific terminology so selected. Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views thereof, embodiments of this disclosure are described. In embodiments, a description is given of an example in which an information processing apparatus is an information terminal that communicates with a spherical imaging device. Further, in embodiments, a description is given of an example in which a program executable by a computer implementing the information processing apparatus is an application program to be installed in the information terminal.

<FIG> is a schematic diagram illustrating an example of a configuration of a spherical image content creation system <NUM> including an information terminal <NUM>, according to the present embodiment. The spherical image content creation system <NUM> illustrated in <FIG> includes a spherical imaging device <NUM> configured to capture a spherical image and the information terminal <NUM> communicable with the spherical imaging device <NUM>.

Any suitable information terminal is used as the information terminal <NUM>, and a mobile information terminal such as a smartphone, a tablet computer, or a laptop computer is preferable. In the information terminal <NUM>, an application program is installed that controls the spherical imaging device <NUM> to capture a spherical image and creates content using the captured spherical image. The content created as above is hereinafter referred to as a "spherical image content". The information terminal <NUM> constitutes an information processing apparatus or a computer according to the present embodiment, which has a spherical image content creation function. Further, the information terminal <NUM> and the spherical imaging device <NUM> in combination constitute the spherical image content creation system <NUM>.

In the embodiment illustrated in <FIG>, the spherical imaging device <NUM> and the information terminal <NUM> are connected by, for example, a wireless connection such as a wireless local area network (LAN) or Bluetooth (registered trademark). An image in a predetermined format captured by the spherical imaging device <NUM> is transmitted to the information terminal <NUM> by wireless communication. The information terminal <NUM> creates the spherical image content using the received image with the spherical image content creation function. In another example, the spherical imaging device <NUM> and the information terminal <NUM> are connected with each other in any other way than the wireless connection as described above, such as a wired LAN or a universal serial bus (USB).

In the embodiment illustrated in <FIG>, the information terminal <NUM> is further connected to the Internet <NUM> through a mobile communication network such as Long-Term Evolution (LTE) or <NUM>, or via a communication device such as an access point, a mobile router, or a broadband router.

In the embodiment illustrated in <FIG>, a content publishing server <NUM> is provided on the Internet <NUM>. The content publishing server <NUM> receives the spherical image content transmitted from the information terminal <NUM> and stores and manages the received spherical image content. Further, the content publishing server <NUM> receives a viewing request for the spherical image content from the information terminal <NUM> or other information terminal, and transmits the requested spherical image content to the request source information terminal. Thus, the spherical image content is viewed on a display device of the request source information terminal. For example, the content publishing server <NUM> is configured as a computer in an on-premises environment or a cloud environment.

In an example embodiment, the content publishing server <NUM> is configured as a web server or a web application server. In such embodiment, the content publishing server <NUM> receives a request for a content registration request including a spherical image content to be registered in accordance with HyperText Transfer Protocol (HTTP), stores the spherical image content in an appropriate storage area, and manages the content in a database, for example. The content publishing server <NUM> further receives a request for an image viewing request designating a spherical image content, reads the spherical image content as the request target, and transmits a response including the read content. The request source information terminal that receives the response makes the received spherical image content viewable on the display device using a web browser or a specific application. In the present embodiment, the content or the spherical image content is a collection of one or a plurality pieces of data as material. Examples of the material of this content or the spherical image content include, but are not limited to, a spherical image, a normal image, a spherical video, a normal video, text (characters), and sound. Although, in the embodiment, a description is given of an example in which a content is a spherical image content including at least a spherical image, in another example, instead of the spherical image, a content uses a wide-angle image other than the spherical image.

A description is now given of examples of configurations of the spherical imaging device <NUM> and the information terminal <NUM>, according to the present embodiment, with reference to <FIG> and <FIG> and <FIG>. <FIG> is a sectional view of the spherical imaging device <NUM>, according to the present embodiment. The spherical imaging device <NUM> illustrated in <FIG> includes an imaging body <NUM>, a casing <NUM> that holds the imaging body <NUM> and components such as a controller and a battery, and an operation button <NUM> provided on the casing <NUM>.

The imaging body <NUM> illustrated in <FIG> includes two image forming optical systems 20A and 20B and two image sensors 22A and 22B. Each of the image sensors 22A and 22B is, for example, a charge-coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. Each of the image forming optical systems <NUM> is configured as a fisheye lens consisting of, for example, seven lenses in six groups. In the embodiment illustrated in <FIG>, the fish-eye lens has a full angle of view of larger than <NUM> degrees (=<NUM> degrees/n, where n denotes the number of optical systems and n is <NUM>), preferably has an angle of view of <NUM> degrees or larger. The number of the image forming optical systems <NUM> and the image sensors <NUM> is any other suitable number, such as <NUM> or more.

The relative positions of the optical elements (lenses, prisms, filters, and aperture stops) of the two image forming optical systems 20A and 20B are defined with reference to the image sensors 22A and 22B. More specifically, positioning is made such that the optical axis of the optical elements of each of the image forming optical systems 20A and 20B is positioned at the central part of the light receiving area of corresponding one of the image sensors <NUM> orthogonally to the light receiving area, and such that the light receiving area serves as the imaging plane of corresponding one of the fisheye lenses.

In the embodiment illustrated in <FIG>, the image forming optical systems 20A and 20B have the same specification, and are combined facing the opposite directions such that the optical axes thereof match with each other. The image sensors 22A and 22B convert the light distribution of the received light into an image signal, and sequentially output image frames to the image processing block on the controller.

As described below in detail, the images captured by the respective image sensors 22A and 22B are combined so as to generate an image over a solid angle of 4π steradian (hereinafter, such an image is referred to as a "spherical image"). The spherical image is obtained by capturing images of all directions that can be seen from an image capturing point. Although in the embodiments, a description is given of an example case in which a spherical image is preferably generated, in another example, a so-called panoramic image obtained by photographing <NUM> degrees only in a horizontal plane or an image that is a part of the image obtained by photographing omnidirectionally or <NUM> degrees in a horizontal plane is generated. The spherical image is either a still image or a moving image (movie).

<FIG> is a diagram illustrating an example of a hardware configuration of the spherical imaging device <NUM>, according to the present embodiment. The spherical imaging device <NUM> includes a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, an image processing block <NUM>, a moving image compression block <NUM>, a still image compression block <NUM>, a dynamic random access memory (DRAM) <NUM> connected via a DRAM interface <NUM>, and an acceleration sensor <NUM> connected via an external sensor interface <NUM>.

The CPU <NUM> controls operation of each of hardware components and overall operation of the spherical imaging device <NUM>. The ROM <NUM> stores a control program described in a code readable by the CPU <NUM> and various kinds of parameters. The image processing block <NUM> is connected to a first image sensor 130A and a second image sensor 130B (corresponding to the image sensors 22A and 22B in <FIG>, respectively), and receives image signals of images captured by the first image sensor and the second image sensor. The image processing block <NUM> includes, for example, an image signal processor (ISP), and performs, for example, shading correction, Bayer interpolation, white balance correction, and gamma correction on the image signals received from the first image sensor 130A and the second image sensor 130B. Further, the image processing block <NUM> combines a plurality of images obtained from the first image sensor 130A and the second image sensor 130B, to generate a spherical image as described above.

The moving image compression block <NUM> is a codec block configured to compress and decompress a video in a moving picture experts group (MPEG)-<NUM> advanced video coding (AVC)/H. <NUM> format, for example. The moving image compression block <NUM> is used to generate video data of the generated spherical image. The still image compression block <NUM> is a codec block configured to compressing and decompress a still image in a joint photographic experts group (JPEG) or tagged image file format (TIFF) format, for example. The still image compression block <NUM> is used to generate still image data of the generated spherical image. The DRAM <NUM> provides a storage area for temporarily storing data when performing various types of signal processing and image processing. The acceleration sensor <NUM> detects acceleration components of three axes, which are used to detect the vertical direction to perform zenith correction on the spherical image. The acceleration sensor <NUM> is just one example. In another example, in addition to the acceleration sensor, at least one of functions of a triaxial angular velocity sensor and an electronic compass is provided.

The spherical imaging device <NUM> further includes an external storage interface <NUM>, a USB interface <NUM>, a serial block <NUM>, and a video output interface <NUM>. The external storage interface <NUM> is connected to an external storage <NUM>. The external storage interface <NUM> controls reading and writing of data from and to the external storage <NUM> such as a memory card inserted in a memory card slot.

The USB interface <NUM> is connected to a USB connector <NUM>. The USB interface <NUM> controls USB communication with an external device such as the information terminal <NUM> or a personal computer connected via the USB connector <NUM>. The serial block <NUM> controls serial communication with an external device such as the information terminal <NUM> or a personal computer, and is connected to a wireless module <NUM>. The video output interface <NUM> is an interface to connect the spherical imaging device <NUM> to an external display such as a high-definition multimedia interface (HDMI, registered trademark), through which captured images are output to the external display as a video.

When the power is turned on by an operation to a power switch, the control program mentioned above is loaded to a main memory. The CPU <NUM> controls operation of each hardware component of the spherical imaging device <NUM> according to the control program loaded into the main memory, while temporarily saving data required for the control in the memory. Thus, the spherical imaging device <NUM> implements functional units and operations described below.

<FIG> is a diagram illustrating an example of a hardware configuration of the information terminal <NUM>, according to the present embodiment. The information terminal <NUM> illustrated in <FIG> includes a CPU <NUM>, a random access memory (RAM) <NUM>, a built-in storage <NUM>, an input device <NUM>, an external storage <NUM>, a display <NUM>, a wireless module <NUM>, a USB connector <NUM>, and a built-in camera <NUM>.

The CPU <NUM> controls operation of each of hardware components and overall operation of the information terminal <NUM>. The RAM <NUM> provides a work area for the CPU <NUM>. The built-in storage <NUM> stores an operating system and a control program, such as an application program that executes processes in the information terminal <NUM> according to the present embodiment, each of the operating system and the control program being written in codes interpretable by the CPU <NUM>.

The input device <NUM> is an input device such as a touch screen, and provides a user interface. The input device <NUM> constitutes input means in the present embodiment. The external storage <NUM> is a removable storage medium mounted, for example, in a memory card slot, and records various types of data, such as image data in a video format and still image data. The wireless module <NUM> establishes a connection for wireless LAN communication between the information terminal <NUM> and an external device such as the spherical imaging device <NUM>. The USB connector <NUM> establishes a USB connection between the information terminal <NUM> and an external device.

Although, a description given hereinafter is of an example in which the wireless module <NUM> is used as communication means that allows the information terminal <NUM> to communicate with the spherical imaging device <NUM>, limitation to any specific standard is not intended. In another example, the connection to an external device is established through other wireless communication such as Bluetooth (registered trademark) or wireless USB, the USB connector <NUM>, or through wired communication such as a wired LAN.

The display <NUM> displays an operation screen that accepts operation inputs from a user, displays a monitor image of an image captured by the spherical imaging device <NUM> that is ready to be captured or being captured, and displays a moving image or still image stored for playback or viewing. The display <NUM> and the input device <NUM> allows a user, through the operation screen, to make instructions for image capturing or changing various kinds of setting in the spherical imaging device <NUM>. The display162 constitutes built-in display means in the present embodiment. In another example, the information terminal <NUM> further includes a video interface such as HDMI (registered trademark). In this example, the information terminal <NUM> outputs a screen to external display means such as a display device or a projector device via the video interface.

The built-in camera <NUM> is a standard camera that the information terminal <NUM> includes, which captures a normal still image or a moving image, to generate still image data and moving image data. The built-in camera <NUM> constitutes imaging means in the present embodiment.

When power is supplied to the information terminal <NUM> and the power thereof is turned on, the programs are read from the ROM <NUM> or the built-in storage <NUM>, and loaded to the RAM <NUM>. The CPU <NUM> controls operation of each hardware element of the information terminal <NUM> according to the programs loaded into the RAM <NUM>, while temporarily saving data required for the control in the memory. Thus, the information terminal <NUM> implements functional units and operations described below. The above-mentioned application program is a program that causes a computer to operate as each means in the present embodiment. The above-mentioned program is installed in a computer, and the computer executes the installed program, thereby implementing the information terminal <NUM> according to the present embodiment.

The content publishing server <NUM> has the similar or substantially the similar hardware configuration as that of the information terminal <NUM> illustrated in <FIG>, and redundant description thereof is omitted below, in order to simplify the description. In addition, any other suitable hardware components are added, deleted, or changed to, from, or in the content publishing server <NUM> as appropriate according to the intended use.

A detailed description is now given of the spherical image content creation function, according to the present embodiment, with reference to <FIG> is a block diagram illustrating an example of a functional configuration relating to spherical image content creation of the information terminal <NUM>, according to the present embodiment.

As illustrated in <FIG>, a functional block <NUM> of the information terminal <NUM> includes a communication unit <NUM>, a built-in camera control unit <NUM>, a display control unit <NUM>, an input unit <NUM>, a spherical image content creation unit <NUM>, and a content storage unit <NUM>.

The communication unit <NUM> controls the wireless module <NUM>, which is an interface with an external device, to perform communication with the spherical imaging device <NUM>. The built-in camera control unit <NUM> controls the built-in camera <NUM> as the imaging means.

The display control unit <NUM> controls the display <NUM> as the display means. The input unit <NUM> controls the input device <NUM> as the input means.

In a preferable example embodiment, the spherical image content creation unit <NUM> is provided as an application that integrates a function of controlling the spherical imaging device <NUM> to capture a spherical image and an imaging function of the built-in camera <NUM> of the information terminal <NUM>, and collectively manages plural types of images (e.g., a spherical image and a normal image) captured with these functions.

The spherical image content creation unit <NUM> includes a spherical imaging device control unit <NUM>, a built-in imaging function invoking unit <NUM>, an image information management unit <NUM>, and a user interface (UI) unit <NUM>.

The spherical imaging device control unit <NUM> controls the spherical imaging device <NUM> as an extraneous source via the communication unit <NUM>. The spherical imaging device control unit <NUM> issues commands to the spherical imaging device <NUM> to control the spherical imaging device <NUM> to capture a spherical image and transmit the captured spherical image to the information terminal <NUM>. Further, the spherical imaging device control unit <NUM> acquires a spherical image from the spherical imaging device <NUM> via the communication unit <NUM>. The spherical imaging device control unit <NUM> constitutes external device control means and first acquisition means in the present embodiment. The spherical image is either a still image or a moving image (movie). A description is given hereinafter basically is of an example in which the spherical image is a still image.

The built-in imaging function invoking unit <NUM> invokes an imaging function of the built-in camera <NUM> by using the built-in camera control unit <NUM>. Further, the built-in imaging function invoking unit <NUM> acquires an image captured by the built-in camera <NUM> from the built-in camera control unit <NUM>. The built-in imaging function invoking unit <NUM> constitutes invoking means and second acquisition means in the present embodiment.

There are three example implementation ways for directly or indirectly controlling the built-in camera <NUM> from the application. The first example way is to invoke a camera application itself for capturing an image using the built-in camera <NUM>. The second example way is to invoke a camera function for capturing an image using the built-in camera <NUM>. The third example way is to implement a function itself for controlling the built-in camera <NUM> in the spherical image content creation unit <NUM>. From the viewpoint of user operability, the second and third example ways are preferable. However, both implementation ways have a common feature of controlling the built-in camera <NUM> directly or indirectly from the application.

Further, the configuration illustrated in <FIG> supports any of the above three ways, and the built-in imaging function invoking unit <NUM> invokes the built-in camera control unit <NUM> as a camera function and invokes the imaging function of the built-in camera <NUM>. However, no limitation is intended thereby.

Although in the embodiment, a description is given of an example in which a normal still image (also referred to as a "still image") is acquired as an image captured by the built-in camera <NUM>, in another example the image captured by the built-in is a moving image. Further, in the embodiment, in a case in which a spherical image is in an image folder (camera roll), the built-in camera <NUM> can be used to capture a normal image.

The image information management unit <NUM> manages a spherical image acquired by the spherical imaging device control unit <NUM> in association with a normal image acquired from the built-in imaging function invoking unit <NUM>. Thus, the image information management unit <NUM> stores a spherical image content in the content storage unit <NUM>. The spherical image content includes one or more spherical images and one or more captured images. The image information management unit <NUM> constitutes associating means in the present embodiment. In another example, in addition to or in alterative to storing the spherical image content in the content storage unit <NUM>, the image information management unit <NUM> transmits (so-called uploads) the created spherical image content to the content publishing server <NUM> as an extraneous source to register the transmitted content therein.

The image information management unit <NUM> illustrated in <FIG> includes an association position specifying unit <NUM>. The association position specifying unit <NUM> specifies a particular position in a spherical image captured by the spherical imaging device <NUM> as an association destination to which the normal image or the like is associated. The association position specifying unit <NUM> constitutes position specifying means in the present embodiment.

In an example embodiment, the association position specifying unit <NUM> specifies a central area of a display area of the spherical image when displayed on the display <NUM> as the particular position in the spherical image. In another example embodiment, the association position specifying unit <NUM> specifies a designated area designated by an input to the input device <NUM> in the display area when the spherical image is displayed on the display <NUM> as the particular position in the spherical image.

In another example, in addition to associating the normal image with the particular position in the spherical image, the image information management unit <NUM> associates the normal image with a content including the spherical image, the content being a higher-level unit than the spherical image. Further, in addition to the normal image acquired by the built-in imaging function invoking unit <NUM>, the image information management unit <NUM> can manage other images in association with the spherical image or the content. For example, the image information management unit <NUM> manages either one or both of images stored in the built-in storage <NUM> of the information terminal <NUM> or the external storage <NUM> and images acquired from extraneous sources such as images received from an image management system (e.g., the content publishing server <NUM> or another service) communicable with the information terminal <NUM>, in association with the spherical image or the content. Furthermore, the image information management unit <NUM> can also manage a second spherical image in association with a particular position of a first spherical image. Moreover, in this case, the image information management unit <NUM> can manage the first spherical image in association with a particular position of the second spherical image.

The UI unit <NUM> manages the transition of screens relating to spherical image content creation, and provides a graphical user interface (GUI) such as a screen and buttons constituting the screen. The UI unit <NUM> constitutes first providing means in the present embodiment for providing a first instruction unit as a GUI component that receives an instruction for controlling the spherical imaging device <NUM> to perform imaging. Further, the UI unit <NUM> constitutes second providing means in the present embodiment for providing a second instruction unit as a GUI component that receives an instruction for invoking the imaging function of the built-in camera <NUM>.

A detailed description is now given of an operation of creating a spherical image content, according to the present embodiment, with reference to <FIG> to <FIG>. <FIG>, <FIG>, <FIG>, <FIG> and <FIG> are flowcharts, each illustrating an operation of creating a spherical image content performed by the information terminal <NUM>, according to the present embodiment. Note that in <FIG>, <FIG>, <FIG>, <FIG> and <FIG>, main processes relating to associating a plurality of types of images, according to the present embodiment, and the descriptions of processes relating to other functions and exception handling are omitted. <FIG> to <FIG> are diagrams, each illustrating an example of a screen displayed on the display <NUM> of the information terminal <NUM>, according to the present embodiment.

<FIG> illustrates an example of a top screen <NUM> of a spherical image content creation application. The top screen <NUM> illustrated in <FIG>, includes a "Create Content" button <NUM> for starting the creation of a spherical image content. The operation illustrated in <FIG> and <FIG> starts from step S100 of <FIG> in response to a user operation of tapping the "Create Content" button <NUM> on the top screen <NUM> illustrated in <FIG>, for example.

In step S101, the information terminal <NUM> displays a content information input screen <NUM> illustrated in <FIG>. The content information input screen <NUM> illustrated in <FIG> allows a user to enter information relating to a content such as a content name and desired alphanumeric characters indicating the content by using a text box <NUM> and a text box <NUM>, for example. The content information input screen <NUM> further includes a " Insert Map" button <NUM> for inserting a floor plan. The user taps the "Insert Map" button <NUM> to invoke a process of inserting an image of a particular floor plan into the content. The content information input screen <NUM> further includes an "OK" button <NUM> for finishing the input of the basic information of the content and proceeding to the next operation of creating items in the content.

In step S102, the information terminal <NUM> branches the operation depending on an instruction received from the user on the content information input screen <NUM> illustrated in <FIG>. When the information terminal <NUM> determines in step S102 that the "Insert Map" button <NUM> is tapped and thereby an instruction for inserting a floor plan (insert floor plan) is received, the operation proceeds to step S103.

In step S103, as illustrated in <FIG>, the information terminal <NUM> displays a source selection menu that receives a selection of a source from which the floor plan is to be acquired. The source selection menu includes a "Photograph by Camera" button <NUM> and a "Select Image from Camera Roll" button <NUM>. The "Photograph by Camera" button <NUM> allows the user, when selected, to select the built-in camera <NUM> as the source. The "Select Image from Camera Roll" button <NUM> allows the user, when selected, to select an image folder in the information terminal <NUM> as the source. In response to tapping either one of the buttons, the operation proceeds to step S104.

Although in the embodiment, the description given is of an example in which the content information input screen <NUM> illustrated in <FIG> includes "Photograph by Camera" button <NUM> and the "Select Image from Camera Roll" button <NUM>, in another example, any other suitable button is provided such as a button for selecting an external image management system on a network as the source.

In step S104, the information terminal <NUM> performs a floor plan registration process. The floor plan is a base image to which spherical images are to be associated in the application. In the present embodiment, the floor plan (also referred to as a "floor map" or a "sketch drawing") is used as a base image as a base image associating the entire contents with each other. In another example, an image other than the floor plan is used as the base image. In still another example, no floor plan is provided, and the entire content is configured by two types of images, i.e., the spherical image and the normal image.

<FIG> is a flowchart illustrating an operation of registering the floor plan. The operation illustrated in <FIG> is invoked by the process of step S104 illustrated in <FIG> and starts from step S201. In step S202, the information terminal <NUM> determines whether imaging by the built-in camera <NUM> is selected. When the determination result in step S202 indicates that the "Photograph by Camera" button <NUM> is tapped in the source selection menu and therefore imaging by the built-in camera <NUM> is selected (S202: YES), the operation proceeds to step S203.

In step S203, the information terminal <NUM> invokes the built-in camera control unit <NUM> by the built-in imaging function invoking unit <NUM>. <FIG> illustrates an imaging screen <NUM> for imaging a floor plan with the built-in camera, which is displayed in response to the invocation of the built-in camera control unit <NUM>. When an imaging button <NUM> is tapped on the imaging screen <NUM> illustrated in <FIG> and thereby the execution of the imaging process is completed, the operation proceeds to step S204. In step S204, the information terminal <NUM> acquires a normal image captured by the built-in camera <NUM>.

By contrast, when the determination result in step S202 indicates that imaging by the built-in camera is not selected (S202: NO), the operation proceeds to step S207. In step S207, the information terminal <NUM> determines whether the information terminal <NUM> is selected as the source. When the determination result in step S207 indicates that the "Select Image from Camera Roll" button <NUM> is tapped in the source selection menu and therefore the information terminal <NUM> is selected (S207: YES), the operation proceeds to step S208.

In step S208, the information terminal <NUM> reads a list of images stored in the information terminal <NUM> (e.g., in an image folder) and displays the list in a manner that a desired image is selected. In step S209, the information terminal <NUM> receives selection from the list of images stored in the information terminal <NUM>, acquires the selected image. Then, the operation proceeds to step S205.

In step S205, the information terminal <NUM> determines whether the image is acquired (acquired by imaging, or acquired from the information terminal itself or an extraneous source). When the determination result indicates that the image is acquired (S205: YES), the operation proceeds to step S206. In step S206, the information terminal <NUM> registers the acquired image as a floor plan in association with the content, and records "with floor plan". In step S210, the operation returns to the operation of <FIG>. Thus, control is returned to step S101, and the content information input screen <NUM> that is updated is displayed.

<FIG> illustrates the content information input screen <NUM> that is updated to reflect the registration of the floor plan. On the content information input screen <NUM> illustrated in <FIG>, an image <NUM> of the floor plan is newly displayed. Although in the embodiment, the description given is of an example in which the floor plan is an image captured by the built-in camera <NUM>, i.e., an example in which the image of the floor plan is acquired by imaging a drawing at hand, such photograph is just an example of the floor plan. In another example, the floor plan is an image such as a drawing or an illustration drawn in advance. For example, such image is stored in an image folder (camera roll).

By contrast, when the determination result in step S205 indicates that no image is acquired (S205: NO), such as when no imaging is performed or when no selection of the desired image is performed, the operation proceeds to S210 and returns to the operation of <FIG>. Further, when the determination result in step S207 indicates that no source is selected, such as when an area other than the "Photograph by Camera" button <NUM> and "Select Image from Camera Roll" button <NUM> is tapped in the source selection menu (S207: NO), the operation proceeds to step S210, and returns to the operation of <FIG>. In another example, the source selection menu is configured so that an image is acquired from an extraneous source such as an image management system (e.g., the content publishing server <NUM> or another service) communicable with the information terminal <NUM>, in addition to the "Photograph by Camera" button <NUM> and the "Select Image from Camera Roll" button <NUM>.

Referring again to <FIG>, in step S102, when the information terminal <NUM> determines that an instruction for proceeding to the next process of creating items of the content is received in response to tapping of the "OK" button <NUM> on the content information input screen <NUM> illustrated in <FIG> or <FIG> (S102: NEXT SCREEN), the operation proceeds to step S105.

In step S105, the operation by the information terminal <NUM> branches depending on the presence or absence of the floor plan. When the determination result in step S105 indicates that the floor plan is registered (S105: YES), the operation proceeds to step S106.

In step S106, the information terminal <NUM> displays a content creation screen (with floor plan) <NUM> to register a spherical image in association with a position on the floor plan. In step S107, the operation by the information terminal <NUM> branches depending on an instruction received from the user on the content creation screen (with floor plan) <NUM>. When the floor plan is registered and no spherical image is registered yet, the content creation screen (with floor plan) <NUM> is in the state as illustrated in <FIG>.

The content creation screen (with floor plan) <NUM> illustrated in <FIG> includes a display area <NUM> in which an image of the floor plan is displayed. In response to detection of position designation operation such as tapping by a user, the display area <NUM> is configured to acquire the coordinates of a position on the display area <NUM> corresponding to the position designation operation. Optionally, the image of the floor plan displayed in the display area <NUM> is enlarged or reduced in response to an operation such as a pinch.

<FIG> is a diagram for describing how association with the position on the floor plan is performed. The display area <NUM> is displayed on the display in a predetermined size, and the image of the floor plan is arranged in the display area <NUM> at a predetermined scale. Therefore, a designated position <NUM> in the display area <NUM> is converted from a relative position (X1/(X1+X2), Y1/(Y1+Y2)) as illustrated in <FIG> to coordinates on the image of the floor plan. When the display area <NUM> is, for example, tapped on the content creation screen (with floor plan) <NUM>, a coordinate value on the image of the floor plan corresponding to the designated position at which the operation is performed in the display area <NUM> is identified, and thereby an instruction for registering a spherical image at the identified position is received. Although in <FIG>, the upper left corner is set as the origin and the coordinate value is in the range of <NUM> to <NUM>, this is just one example. The origin is set in any other suitable way and the coordinate value is in any other suitable range.

In the content creation screen (with floor plan) <NUM> displayed in step S106, a function to "save" a content and a function to confirm a registered spherical image in the content ("image confirmation") can be invoked, in addition to the registration of the spherical image ("image registration"). However, when no spherical image is registered yet, a GUI component for the "image confirmation" is not displayed. On the other hand, a GUI component to "save" a content may be displayed so that the content can be temporarily saved.

When the determination result in step S107 indicates that an instruction for "image registration" is received on the content creation screen (with floor plan) <NUM>, the operation proceeds to step S108. In step S108, the information terminal <NUM> displays the source selection menu illustrated in <FIG>. The source selection menu includes a "Photograph by Spherical Camera" button <NUM> and a "Select Spherical Image from Camera Roll" button <NUM>. The "Photograph by Spherical Camera" button <NUM> allows the user, when selected, to select the spherical imaging device <NUM>, which is external to the information terminal <NUM>, as a source from which a spherical image to be registered is be acquired. The "Select Spherical Image from Camera Roll" button <NUM> allows the user, when selected, to select an image folder in the information terminal <NUM>, as a source of a spherical image to be registered. In response to tapping either one of the buttons, the operation proceeds to step S109. The "Photograph by Spherical Camera" button <NUM> constitutes the first instruction unit in the present embodiment.

In step S109, the information terminal <NUM> sets the content and the position coordinate value on the image of the floor plan corresponding to the designated position as an association destination to which the spherical image is to be registered. In step S110, the information terminal <NUM> performs a spherical image registration process.

By contrast, when the determination result in step S105 indicates that no floor plan is registered (S105: NO), the operation proceeds to step S111. In step S111, the information terminal <NUM> displays a content creation screen (without floor plan) <NUM>. In step S112, the operation by the information terminal <NUM> branches depending on an instruction received from the user on the content creation screen (without floor plan) <NUM>. When no floor plan is registered and no spherical image is registered yet, the content creation screen (without floor plan) <NUM> is in the state as illustrated in <FIG>. The content creation screen (without floor plan) <NUM> illustrated in <FIG> includes the "Photograph by Spherical Camera" button <NUM> and "Select Spherical Image from Camera Roll" button <NUM>, in substantially the same manner as the screen illustrated in <FIG>. In response to tapping either one of the buttons, an instruction for registering a spherical image ("image registration") is received, and the operation proceeds to step S112.

In the content creation screen (without floor plan) <NUM> displayed in step S111, a function to "save" a content and a function to confirm a registered spherical image in the content ("image confirmation") can be invoked, in addition to the registration of the spherical image ("image registration"). Although in a state in which no spherical image is registered yet, a GUI component for the "image confirmation" is not displayed, a GUI component to "save" the content may be displayed for temporary storage.

When the determination result in step S112 indicates that an instruction for "image registration" is received on the content creation screen (without floor plan) <NUM>, the operation proceeds to step S113. In step S113, the information terminal <NUM> sets the content itself as an association destination with which a spherical image to be captured is to be associated. In step S110, the information terminal <NUM> performs the spherical image registration process.

In the embodiment, the description given is of an example in which the screen illustrated in <FIG> includes only the "Photograph by Spherical Camera" button <NUM> and the "Select Spherical Image from Camera Roll" button <NUM>, and the screen illustrated in <FIG> includes only the "Photograph by Spherical Camera" button <NUM> and the "Select Spherical Image from Camera Roll" button <NUM>. In another example, the screen includes a button for selecting, as the source, a spherical image that is already captured and stored in the spherical imaging device <NUM> but not transferred yet, or a button for selecting, as the source, a spherical image stored in an external image management system on a network.

<FIG> is a flowchart illustrating an operation of registering a spherical image. The operation illustrated in <FIG> is invoked by the process of step S110 illustrated in <FIG> and starts from step S300. In step S301, the information terminal <NUM> determines whether imaging by the spherical imaging device <NUM>, which is external to the information terminal, is designated When the determination result in step S301 indicates that imaging by the spherical imaging device <NUM> is selected (S301: YES), as the "Photograph by Spherical Camera" button <NUM> of the source selection menu is tapped on the content creation screen (with floor plan) <NUM> illustrated in <FIG>, or as the "Photograph by Spherical Camera" button <NUM> is tapped on the content creation screen (without floor plan) <NUM> illustrated in <FIG>, the operation proceeds to step S302.

In step S302, the information terminal <NUM> controls the spherical imaging device <NUM> by the spherical imaging device control unit <NUM> to capture a spherical image. <FIG> illustrates a spherical imaging screen <NUM> for capturing an image by the spherical imaging device <NUM>. When the imaging button <NUM> is tapped on the spherical imaging screen <NUM>, an imaging command is issued from the spherical imaging device control unit <NUM> to the spherical imaging device <NUM> via the communication unit <NUM>. In response to the imaging command, the spherical imaging device <NUM> performs imaging operation. When the imaging is completed, an image transfer to the information terminal <NUM> is started. <FIG> illustrates a transfer screen <NUM> indicating that the spherical image is being transferred from the spherical imaging device <NUM> to the information terminal <NUM>. In step S303, the information terminal <NUM> acquires the spherical image transferred from the spherical imaging device <NUM>, which is an external imaging device, by the spherical imaging device control unit <NUM>, and the operation proceeds to step S313.

By contrast, when the determination result in step S301 indicates that imaging by the spherical imaging device <NUM> is not selected (S301: NO), the operation proceeds to step S304. In step S304, the information terminal <NUM> determines whether a captured image stored in the spherical imaging device <NUM>, which is an external imaging device, is selected as the source. When the determined result in step S304 indicates that the captured image stored in the spherical imaging device <NUM> is selected as the source (S304: YES), the operation proceeds to.

In step S305, the information terminal <NUM> acquires a list of spherical images that have been captured (and has not been transferred to the information terminal <NUM> yet) from the spherical imaging device <NUM> by the spherical imaging device control unit <NUM>, and displays the list in a manner that a desired spherical image is selected. In step S306, the information terminal <NUM> receives a selection of a particular image from the list of the spherical images stored in the spherical imaging device <NUM>. In step S303, the information terminal <NUM> receives the selected spherical image from the spherical imaging device <NUM>, and the operation proceeds to step S313.

By contrast, when the determination result in step S304 indicates that a captured image stored in the spherical imaging device <NUM>, which is an external imaging device, is not selected as the source (S304: NO), the operation proceeds to step S307. The determination in step S304 is performed in a case in which the menu illustrated in <FIG> and the content creation screen (without floor plan) <NUM> illustrated in <FIG> includes a button for selecting a captured spherical image (and not transferred to the information terminal <NUM> yet) stored in the spherical imaging device <NUM> as the source. However, in another example, such process is omitted. In this case, the operation proceeds directly to step S307.

In step S307, the information terminal <NUM> determines whether images stored in the information terminal <NUM> are selected as the source. When the determination result in step S307 indicates that the images stored in the information terminal <NUM> are selected as the source (S307: YES), the operation proceeds to step S308.

In step S308, the information terminal <NUM> reads out a list of spherical images stored in the information terminal <NUM>. A format of a spherical image is same as that of a general-purpose image, such as JPEG. Further, to an image, information for identifying whether the image is a spherical image is attached as meta information. For example, when a user selects an image other than a spherical image, the information terminal <NUM> prompts the user to select a different image, based on this meta information. In another example, based on the meta information, the information terminal <NUM> displays, from the acquired list of images, spherical images, and other normal captured images in a manner that two types of images are distinguishable. In still another example, the information terminal <NUM> extracts only spherical images in advance based on the meta information and include the extracted spherical images in the list to be displayed. Further, it is assumed that spherical images that have already been captured and has been acquired are stored in the information terminal <NUM>. In step S309, the information terminal <NUM> receives a selection of a particular spherical image from the list of the spherical images stored in the information terminal <NUM>, and reads the selected image. Then, the operation proceeds to step S313.

By contrast, when the determination result in step S307 indicates that images stored in the information terminal <NUM> are not selected as the source (S307: NO), the operation proceeds to step S310. In step S310, the information terminal <NUM> acquires a list of spherical images from an image management system (e.g. the content publishing server <NUM>) communicable with the information terminal <NUM> via a network. It is assumed that spherical images that have already been captured have been uploaded to the image management system. In step S311, the information terminal <NUM> receives a selection of a particular spherical image from the list of the spherical images stored in the image management system, and in step S312, receives the selected spherical image via the network. Then, the operation proceeds to step S313.

The processes from step S310 to step S312 are performed in a case in which the source selection menu illustrated in <FIG> and the content creation screen (without floor plan) <NUM> illustrated in <FIG> includes a button for selecting an image management system on the network as the source. However, in another example, such processes are omitted. In this case, the operation proceeds directly to step S316.

In step S313, the information terminal <NUM> displays a screen for confirming a spherical image to be used. In step S314, the information terminal <NUM> determines whether the spherical image to be used is confirmed. <FIG> illustrates a spherical image confirmation screen <NUM> for confirming the spherical image. The spherical image confirmation screen <NUM> includes a button <NUM> for confirming the use of the spherical image that is currently displayed and a button <NUM> for redoing the imaging or the selection.

When the determination result in step S314 indicates that the image is confirmed (S314: YES), as the button <NUM> is tapped on the spherical image confirmation screen <NUM>, the operation proceeds to step S315. In step S315, the information terminal <NUM> registers the content by associating the confirmed spherical image with the association destination that is set in step S109 or S113 of <FIG>. In step S316, the operation returns to the operation of <FIG>. Note that the registration of this association destination can be moved to a desired position afterwards. By contrast, when the determination result in step S314 indicates that the redo is designated (S314: NO), as the button <NUM> is tapped on the spherical image confirmation screen <NUM> illustrated in <FIG>, the operation returns to step S301. Although the description given is of an example in which the redo process is performed in a state where the same source is selected, in another example the redo process is performed from the selection of the source.

When the spherical image registration process illustrated in <FIG> ends, the operation returns to step S105 again, and the content creation screen that is updated is displayed in step S106 or step S111.

<FIG> illustrates the content creation screen updated in step S111 in a state in which no floor plan is registered and one spherical image is registered.

The content creation screen (without floor plan) <NUM> illustrated in <FIG> includes a display area <NUM> in which the spherical image is displayed and a "Save" button <NUM> for saving the content. The display area <NUM> includes, a GUI component <NUM> for invoking a screen for entering information indicating a place where the spherical image is captured. The content creation screen (without floor plan) <NUM> further includes, the "Select Spherical Image from Camera Roll" button <NUM> and an icon <NUM> for photographing with the spherical imaging device.

The display area <NUM> of a particular spherical image is a GUI component for invoking confirmation of a registered spherical image ("image confirmation"). The "Save" button <NUM> is a GUI component for invoking a content saving process. As one or more spherical images are registered, "image confirmation" is displayed, and the GUI component of "save" of the content is displayed as needed.

<FIG> illustrates the content creation screen updated in step S106 in a state in which a floor plan is registered and two spherical images are registered.

The content creation screen (with floor plan) <NUM> illustrated in <FIG> includes the display area <NUM> of a floor plan above a display area <NUM> in which a spherical image that similar to the content creation screen (without floor plan) <NUM> illustrated in <FIG>. Note that <FIG> illustrates a state in which two spherical images are registered respectively in association with two positions on the floor plan. A marker 442a and a marker 442b are attached respectively to two positions on the display area <NUM> of the floor plan. More specifically, <FIG> illustrates a state in which a spherical image corresponding to the marker 442b at a position indicated by "<NUM>" among the two marker 442a and marker 442b is selected.

In the display area <NUM>, the spherical image currently being selected is displayed. The display area <NUM> further includes a GUI component <NUM> for invoking a screen for entering information indicating a place where the spherical image is captured. Further, an icon 442c having a fan shape indicating an angle at which the spherical image is being displayed is superimposed on the display area <NUM> of the floor plan. The display area <NUM> of the spherical image is a GUI component for invoking confirmation ("image confirmation") of the spherical image that is registered and currently being selected. The content creation screen (with floor plan) <NUM> illustrated in <FIG> includes a "Save" button <NUM> in substantially the same manner as the content creation screen illustrated in <FIG>.

Hereinafter, a description is given of an embodiment in which a normal image as an annotation is added to a spherical image using the content creation screen (without floor plan) <NUM> illustrated in <FIG>. However, as indicated above, the similar or substantially the similar operation is performed using the content creation screen (with floor plan) <NUM>.

When the display area <NUM> corresponding to a particular spherical image is tapped on the content creation screen (without floor plan) <NUM> illustrated in <FIG>, the information terminal <NUM> receives an instruction for "image confirmation". When the determination result in step S107 or step S112 indicates that the "image confirmation" is accepted on the content creation screen (with floor plan) <NUM> or the content creation screen (without floor plan) <NUM>, the operation proceeds to step S114 illustrated in <FIG> via a point B illustrated in <FIG>.

In step S114, the information terminal <NUM> displays an image confirmation screen <NUM> as illustrated in <FIG>, the screen allowing a user to confirm a designated spherical image. The image confirmation screen <NUM> illustrated in <FIG> includes a "Back" button <NUM>, a display area <NUM> in which the designated spherical image is displayed, and a menu button <NUM>. In this screen, a displayed part of the spherical image displayed in the display area <NUM> is moved in response to an operation of instructing a direction such as a flick.

When the menu button <NUM> is tapped on the image confirmation screen <NUM> illustrated in <FIG>, a menu including an "Add Annotation" button <NUM> for performing image annotation is displayed on the image confirmation screen <NUM>, as illustrated in <FIG>. When the "Add Annotation" button <NUM> is tapped, the information terminal <NUM> receives an instruction for adding an annotation. When the "Back" button <NUM> is tapped on the image confirmation screen <NUM> illustrated in <FIG>, the information terminal <NUM> receives an instruction for returning to the content creation screen (e.g., the content creation screen (without floor plan) <NUM> of <FIG>).

When the determination result in step S115 that the instruction of "Add annotation" is accepted on the image confirmation screen <NUM> illustrated in <FIG>, the operation proceeds to step S116. In step S116, the information terminal <NUM> specifies a position to be associated, by the association position specifying unit <NUM>. When the annotation addition is invoked through the menu button <NUM> on the image confirmation screen <NUM> illustrated in <FIG>, a position corresponding to substantially the center of the portion currently being displayed in the display area <NUM> of the spherical image is specified as an association position to which the annotation is added.

The description given above is of an example in which the "Add Annotation" button <NUM> is displayed in response to tapping the menu button <NUM> on the image confirmation screen <NUM> illustrated in <FIG>. However, this is just one example. In another example, the "Add Annotation" button <NUM> is displayed in response to a long tap (holding down) of a predetermined position (e.g., a position <NUM>) in the display area <NUM> on the image confirmation screen <NUM> illustrated in <FIG>. When annotation addition is invoked by tapping an area in the display area <NUM>, a position in the spherical image corresponding to a position when the area in the display area <NUM> is tapped is specified as the association position to which the annotation is to be added.

<FIG> is a diagram for describing how to associate an annotation with a position in a spherical image. As illustrated in a spherical image format <NUM> of <FIG>, a spherical image is generated in a format in which a horizontal axis represents the longitude from -<NUM> degrees to +<NUM> degrees and a vertical axis represents the latitude from -<NUM> degrees to +<NUM> degrees.

When displaying such spherical image on a device having a flat surface such as the display <NUM> of the information terminal <NUM>, the spherical image is mapped to a spherical object <NUM>, and a virtual camera <NUM> is arranged, for example, at the center of the spherical object <NUM>. Thereby, the spherical image is displayed as a projection image <NUM> viewed from the virtual camera <NUM>. In other words, the display area <NUM> is displayed on the display in a predetermined size, and the projection image <NUM> generated from the spherical image is arranged in the display area <NUM>. A position in the spherical image corresponding to the center position of the projection image <NUM> currently being displayed is acquired from a direction of the virtual camera <NUM>. On the other hand, a certain position (e.g., a tapped position) <NUM> in the display area <NUM> is converted to a coordinate <NUM> on the spherical image format <NUM>, from relative positions (X1/(X1+X2), Y1/(Y1+Y2)) as illustrated in <FIG>, via a points <NUM> in a range corresponding to the projection image <NUM> on the spherical object <NUM>. Although <FIG> illustrates an example in which the center is set as the origin, and the coordinate values are set in the range of longitude from -<NUM> degrees to +<NUM> degrees and latitude from -<NUM> degrees to +<NUM> degrees, in another example, the origin is set in any other suitable way and the range is set in any other suitable way. For example, the spherical image format <NUM> is represented in a format in which the horizontal axis indicates the longitude from <NUM> degree to <NUM> degrees and the vertical axis indicates the latitude from <NUM> degree to <NUM> degrees. In a case in which the wide-angle image is an image other than the spherical image, such as an image having an angle of view of <NUM> degrees or more, a coordinate of the image is set that supports the angle of view, instead of the coordinate of the spherical image described above.

Referring again to <FIG>, in step S117, the information terminal <NUM> sets the specified position in the spherical image as the association destination. In step S118, the information terminal <NUM> displays an annotation setting screen <NUM> as illustrated in <FIG>. The annotation setting screen <NUM> illustrated in <FIG> includes a "Select Image" button <NUM>. In response to a user's tapping on the "Select Image" button <NUM>, a process of registering an image captured by the built-in camera <NUM> of the information terminal <NUM> is invoked. Although in the embodiment, a description is given of an example in which only an image is added as an annotation, in another example, a text input field is provided such that text is also added as an annotation in addition to an image.

In step S119, the information terminal <NUM> displays a source selection menu that accepts a selection of a source from which a normal image is to be acquired, as illustrated in <FIG>. The source selection menu includes a "Photograph by Camera" button <NUM> for selecting the built-in camera and a "Select Image from Camera Roll" button <NUM> for selecting an image folder in the information terminal <NUM> as the source. When either one of the buttons is tapped, the operation proceeds to step S120. The "Photograph by Camera" button <NUM> constitutes a second instruction unit in the present embodiment.

In step S120, the information terminal <NUM> performs a normal image registration process.

<FIG> is a flowchart illustrating an operation of registering a normal image. The operation illustrated in <FIG> is invoked by the process of step S120 illustrated in <FIG> and starts from step S400. In step S401, the information terminal <NUM> determines whether imaging by the built-in camera is designated. When the determination result in step <NUM> indicates that the "Photograph by Camera" button <NUM> is tapped in the source selection menu illustrated in <FIG> and therefore imaging by the built-in camera <NUM> is selected (S401: YES), the operation proceeds to step S402.

In step S402, the information terminal <NUM> invokes the built-in camera control unit <NUM> by the built-in imaging function invoking unit <NUM>. <FIG> illustrates an imaging screen <NUM> for imaging a normal image with the built-in camera, which is displayed in response to the invocation of the built-in camera control unit <NUM>. When an imaging button <NUM> is tapped on the imaging screen <NUM> illustrated in <FIG> and thereby the execution of the imaging process is completed, the operation proceeds to step S410. In step S403, the information terminal <NUM> acquires the normal image captured by the built-in camera <NUM>, and the operation proceeds to step S410.

By contrast, when the determination result in step S401 indicates that imaging by the built-in camera <NUM> is not selected (S401: NO), the operation proceeds to step S404. In step S404, the information terminal <NUM> determines whether the information terminal <NUM> is selected as the source. When the determination result in step S404 indicates that the information terminal <NUM> is selected as the source (S404: YES), the operation proceeds to step S405.

In step S405, the information terminal <NUM> reads a list of images stored in the information terminal <NUM> and displays the list in a manner that a desired image is selected. In step S406, the information terminal <NUM> receives a selection of a particular image from the list of the images stored in the information terminal <NUM>, and reads the selected image. Then, the operation proceeds to step S410.

By contrast, when the determination result in step S404 indicates that images stored in the information terminal <NUM> are not selected as the source (S404: NO), the operation proceeds to step S407. In step S407, the information terminal <NUM> acquires a list of normal images from an image management system (e.g. the content publishing server <NUM>) communicable with the information terminal <NUM> via a network, and displays the acquired list. In step S408, the information terminal <NUM> receives a selection of a particular normal image from the list of the normal images stored in the image management system, and in step S409, receives the selected normal image via the network. Then, the operation proceeds to step S410. The processes from step S407 to step S409 are performed in a case in which the source selection menu illustrated in <FIG> includes a button for selecting the image management system as the source. However, in another example, such processes are omitted. In this case, the operation proceeds directly to step S413.

In step S410, the information terminal <NUM> displays a screen for confirming a spherical image to be used. In step S410, the information terminal <NUM> determines whether the normal image to be used is confirmed. <FIG> illustrates an image confirmation screen <NUM> for confirming the normal image. The image confirmation screen <NUM> includes a button <NUM> for confirming the use of the image that is currently being displayed and a button <NUM> for redoing the imaging or the selection.

When the determination result in step S411 indicates that the image is confirmed (S411: YES), as the button <NUM> is tapped on the image confirmation screen <NUM>, the operation proceeds to step S412. In step S412, the information terminal <NUM> registers the content by associating the confirmed image with the association destination. In step S413, the operation returns to the operation of <FIG>. This association destination is the coordinate value of the particular position in the spherical image or a direction indicating a particular position, which are specified in step S117. Note that the registration of this association destination can be moved to a desired position afterwards. By contrast, when the determination result in step S411 indicates that the redo is designated (S411: NO), as the button <NUM> is tapped on the image confirmation screen <NUM>, the operation returns to step S401.

When the operation returns to the operation of <FIG>, the operation returns to step S114 via the point B. In step S114, the information terminal <NUM> displays the image confirmation screen <NUM> that is updated, as illustrated in <FIG>. In the image confirmation screen <NUM> illustrated in <FIG>, a marker <NUM> indicating the position with which the normal image is associated is displayed in the display area <NUM> of the spherical image. The coordinate of the marker <NUM> corresponds to the coordinate set as the association destination when the normal image is associated and registered in step S412. Further, since there are one or more annotations, a button <NUM> for deleting an annotation and a button <NUM> for editing an annotation are displayed. However, a further detailed description of deleting and editing annotations is omitted, in order to simplify the description.

When the determination result in step S115 indicates that the information terminal <NUM> receives an instruction for returning to an upper level, as the "Back" button <NUM> is tapped on the image confirmation screen <NUM> illustrated in <FIG> or <FIG>, the operation proceeds to step S105 illustrated in <FIG> via the point C. In this case, an operation of registering another spherical image in association with a particular position on the floor plan of the content or in association with the content itself is performed.

It is assumed that the content creation screen (without floor plan) <NUM> as illustrated in <FIG> or the content creation screen (with floor plan) <NUM> in which a spherical image is registered at a particular position on a floor plan is displayed. When the determination result in step S107 or step S112 indicates that an instruction for "save" is accepted on the content creation screen (with floor plan) <NUM> or the content creation screen (without floor plan) <NUM>, the operation proceeds to step S121 illustrated in <FIG> via a point A.

In step S121, the information terminal <NUM> stores the created spherical image content in the content storage unit <NUM>. Then, the operation ends in step S122. In another example, instead of storing the spherical image content in the content storage unit <NUM> or in addition to storing the spherical image content in the content storage unit <NUM>, the spherical image content is uploaded to the content publishing server <NUM>.

In another example, any other suitable functions are provided, although a detailed description thereof is omitted below. Examples of the other suitable functions include deleting and replacing the registered spherical image and the normal image, before storing or uploading the spherical image content. Examples of the other suitable functions further include performing image processing such as blurring a particular area of the spherical image or the normal image, stamping, and reducing, before storing or uploading the spherical image content. Examples of the other suitable functions still further include inputting, changing, or deleting the name of the content or the spherical image, before storing or uploading the spherical image content.

For example, in a content that introduces a piece of real estate and equipment, a normal image obtained by photographing equipment provided in each of plural rooms is associated with a particular position of a spherical image of the corresponding room. On the other hand, in some cases, a normal image is obtained by photographing view from an entrance or a veranda. In this case, such normal image can be associated with the entire content rather than the spherical image of each room. In still another example, spherical images of plural rooms are associated with each other at a predetermined position (e.g., example, a position corresponding to the door).

<FIG> are diagrams illustrating a data structure in which a spherical image content generated by the information terminal <NUM> is stored, according to the present embodiment. <FIG> illustrates a content information management table that stores information such as a content identifier (ID) and a name in association with each other. <FIG> illustrates a spherical image management table that stores a spherical image ID in association with a storage location in which a spherical image identified by the corresponding spherical image ID is stored. <FIG> illustrates a normal image management table that stores a normal image ID in association with a storage location in which a normal image identified by the corresponding normal image ID is stored.

<FIG> illustrates a content element management DB that stores an ID of a content element, which is a target to be associated with a spherical image, such as a room ID, a name of the content element, a spherical image ID of a spherical image with the content element is to be associated, and coordinates on a floor plan, in association with each other. The coordinates on the floor plan are obtained as described above with reference to <FIG>, for example, and stored. In the table, Null indicates that there is no value. In other words, Null in the table indicates that a content or a content element identified by a content ID <NUM> is a content including no floor plan or a room (content element) that is not associated with a position on the floor plan.

<FIG> illustrates a floor plan management table that stores a floor plan ID, a content ID, and a normal image ID indenting a normal image corresponding to the floor plan in association with each other. <FIG> illustrates an annotation management table that stores an annotation ID, a spherical image ID identifying a spherical image with which an annotation identified by the corresponding annotation ID is associated, coordinates at which the annotation is placed on the spherical image, a description of the annotation, and a normal image ID identifying a normal image to be added as the annotation, in association with each other. The coordinates on the spherical image are obtained as described with reference to <FIG>. In another example, the annotation management table is configured such that the entire content is associated instead of associating the spherical image.

<FIG> is a diagram illustrating a file structure when a spherical image content generated by the information terminal <NUM> according to the present embodiment is exported. In <FIG>, under a folder having a name of content, a file of an image of a floor plan, a file of an image of view, and folders each having a name of a spherical image are arranged. Under each folder having the name of the spherical image, a file of the spherical image and one or more files of normal images associated with the spherical image are arranged.

By the above-described operations, the spherical image content having a data structure schematically illustrated in <FIG> having a normal image associated with a certain position in the spherical image is created. Further, when viewing the spherical image content by using a viewer application, a normal image is displayed as being embedded in the spherical image as an annotation. For example, in the viewer application, in response to a predetermined operation (e.g., tap, click, etc.) on an image element (e.g., a marker, icon, or symbol) arranged at an annotation position in the spherical image, the normal image associated with the annotation position pops up.

As described heretofore, according to the described embodiment, in an information processing apparatus (a computer) including imaging means, a program implementing the information processing apparatus configured to integrate a function of controlling an external imaging device to capture an image and a function of imaging by the imaging means that the information processing apparatus (the computer) includes, and manage a plurality of types of captured images collectively, a storage medium storing the program, and the information processing apparatus are provided.

In the above-described embodiment, a function of controlling an external imaging device to capture an image and a function of capturing an image with imaging means that the information processing apparatus itself includes are integrated into a single application. Therefore, a user is saved from having to use the function of controlling the external imaging device to capture an image and the function of capturing an image by the imaging means in separate applications.

Further, when using the different imaging functions in different applications respectively, images are to be temporarily stored in an image folder such as a camera roll and registered in a content via the image folder. However, in this case, although the images can be stored in chronological order in the image folder, the images to be registered in different contents are sometimes stored in a mixed manner. In some cases, this makes it difficult to correctly register the images in the content. By contrast, in the above-described embodiment, since a plurality of types of images are collectively managed, complications caused by the intervening image folder are reduced.

Further, integrating the capturing of the spherical image and the capturing of the normal image for content creation reduces time and efforts of a creator of the content. Since the creator can perform up to photographing a normal image on the content creation application installed in the information terminal <NUM>, as long as the spherical imaging device <NUM> and the information terminal <NUM> are provided, photographic equipment such as a general-purpose is not required. Thus, capturing a normal image or movie, which is required in the related art, is no more required. Furthermore, a content, a floor plan (optional), a spherical image, and a normal image are associated without time and efforts. This reduces time and efforts of classifying and managing the captured normal images.

In the embodiment, the description given above is of an example in which a content to be created is a spherical image content including a spherical image as a wide-angle image. However, the spherical image is just one example of the wide-angle image. In another example, the wide-angle image is a panoramic image obtained by photographing <NUM> degrees in a horizontal plane or an image that is a part of the image obtained by photographing omnidirectionally or <NUM> degrees in a horizontal plane. In the disclosure, the wide-angle image refers to an image having a relatively large angle of view as compared with a normal image, and preferably an image having an angle of view of at least a horizontal angle of view of <NUM> degrees or more. In still another example, the wide-angle image is an image having a large angle of view captured by one or more imaging devices. The normal image is either a still image or a moving image. The normal image an image having a narrower angle of view than the wide-angle image, and more specifically, an image having an angle of view of <NUM> to <NUM> degrees, preferably <NUM> to <NUM> degrees.

In the embodiment, the description given above is of an example in which the spherical image and the normal image are each a still image. However, the spherical image and the normal image are each not limited to a still images. In another example, the spherical image and the normal image is a semi-moving image (e.g., a time lapse movie, an interval-shot images, animation), which includes a series of still images. In this case, image association is performed by using time information such as a time and a frame interval as the association destination, in addition to the particular position in the spherical image.

Each of the functions of the described embodiments may be implemented by one or more processing circuits or circuitry. Processing circuitry includes a programmed processor, as a processor includes circuitry. A processing circuit also includes devices such as an application specific integrated circuit (ASIC), digital signal processor (DSP), field programmable gate array (FPGA) and conventional circuit components arranged to perform the recited functions.

The functional blocks as described above are implemented by a computer-executable program written by programming languages such as an assembler language, C, and object-oriented programming languages such as C++, C#, and Java (registered trademark). The program may be distributed through a telecommunication line or as being stored in a computer-executable storage medium such as a ROM, an electrically erasable and programmable read only memory (EEPROM), an electrically programmable read only memory (EPROM), a flash memory, a flexible disk, a compact disk read only memory (CD-ROM), a compact disk rewritable (CD-RW), a digital versatile disk read only memory (DVD-ROM), a digital versatile disk random access memory (DVD-RAM), a digital versatile disk rewritable (DVD-RW), a Blu-ray disk, an SD card, and a magneto-optical disk (MO).

Claim 1:
An information processing method performed by an information processing apparatus (<NUM>) including communication means (<NUM>) and imaging means (<NUM>), the information processing apparatus (<NUM>) being installed with an application program, the method comprising:
executing the application program to:
control, via the communication means (<NUM>), an external imaging device (<NUM>) to capture a wide-angle image and transmit the captured wide-angle image to the information processing apparatus (<NUM>);
control the imaging means (<NUM>) to capture an image;
specify a particular position in the wide-angle image captured by the external imaging device (<NUM>);
associate the wide-angle image with the image captured by the imaging means (<NUM>) by associating the captured image with the particular position in the wide-angle image;
and further characterized by executing the application program to :
create an image element arranged at the particular position of the wide-angle image, wherein, the image element is configured so that, when the image element and the wide-angle image are displayed on a display, a predetermined operation on the image element causes the image captured by the imaging means (<NUM>) to be displayed on the display at the particular position.