PANORAMA PACKET

One or more techniques and/or systems are provided for generating a panorama packet and/or for utilizing a panorama packet. That is, a panorama packet may be generated and/or consumed to provide an interactive panorama view experience of a scene depicted by one or more input images within the panorama packet (e.g., a user may explore the scene through multi-dimensional navigation of a panorama generated from the panorama packet). The panorama packet may comprise a set of input images may depict the scene from various viewpoints. The panorama packet may comprise a camera pose manifold that may define one or more perspectives of the scene that may be used to generate a current view of the scene. The panorama packet may comprise a coarse geometry corresponding to a multi-dimensional representation of a surface of the scene. An interactive panorama of the scene may be generated based upon the panorama packet.

DETAILED DESCRIPTION

An embodiment of generating a panorama packet is illustrated by an exemplary method100ofFIG. 1. At102, the method starts. At104, a set of input images depicting a scene are identified (e.g., a user may capture one or more photos of a building and outdoor space). At106, a camera pose manifold is estimated based upon the set of input images. For example, a graph of the set of input images may be mapped onto a geometric shape (e.g., based upon focal points of respective input images), and the camera pose manifold is defined by the graph. The camera pose manifold may comprise rotational data and/or translational data that may be used to generate a current view of the scene depicted by the set of input images (e.g., a panorama of the scene may be generated, and a current view of the panorama may be created based upon a view of the scene along the camera pose manifold).

At108, a coarse geometry may be constructed based upon the set of input images. The coarse geometry may correspond to a multi-dimensional representation of a surface of the scene. For example, a structure from motion techniques, stereo mapping techniques, utilization of depth values, an image feature matching technique, and/or other techniques may be used to construct the coarse geometry from the set of input images. In an example, the set of inputs images may be projected onto the coarse geometry (e.g., during generation of a panorama) to create textured coarse geometry (e.g., color values of pixels of input images may be assigned to geometry pixels of the coarse geometry).

In some embodiments, a graph may be defined for inclusion within the panorama packet. The graph may specify relational information between respective input images within the set of input images. In an example, the graph comprises a first node representing a first input image, a second node representing a second input image, and a first edge between the first node and the second node. The first edge may represent translation view information between the first input image and the second input image (e.g., a translated view of the scene may correspond to a portion of the scene that is not depicted by a single input image, but may be based upon a view derived from multiple input images which may be projected onto the coarse geometry to obtain the translated view). In this way, the graph may be utilized to generate one or more current views provided during an interactive panorama view experience of the scene through a panorama generated using the panorama packet.

At110, the panorama packet may be generated. The panorama packet may comprise the set of input images, the camera pose manifold, the coarse geometry, the graph, and/or other information. In an example, the set of input images may be retained within the panorama packet, such as during panorama generation, without modification to the set of input images (e.g., the set of input images may not be fused together during an interactive panorama view experience of the scene). In an example, the panorama packet may be stored according to a single file format (e.g., a file that may be consumed by an image viewing interface). The panorama packet may be utilized (e.g., by an image viewing interface) to provide an interactive panorama view experience of the scene through a panorama created from the panorama packet. At112, the method ends.

FIG. 2illustrates an example of a system200for generating a panorama packet206. The system200comprises a packet generating component204. The packet generating component204is configured to identify a set of input images202. In an example, one or more input images may be selected for identification as the set of input image202based upon various criteria, such as a relatively similar name, a relatively similar description, captured by the same camera, captured by the same image capture program, image features depicting a similar scene, images taken within a temporal threshold, etc. The set of input images202may depict a scene, such as a building and outdoor space, from various viewpoints.

The packet generating component204may be configured to estimate a camera pose manifold210, such as based on the camera position and/or orientation information for respective input images, for example. The camera pose manifold210may comprise one or more focal points for view perspectives of the scene (e.g., a view perspective from which a user may view the scene through a panorama generated based upon the panorama packet206). The packet generating component204may be configured to construct a coarse geometry212corresponding to a multi-dimensional representation of a surface of the scene. In some embodiments, the packet generating component204may be configured to generate a graph214representing relational information between respective input images within the set of input images202, which may be used to derive a current view of the panorama. The packet generating component204may generate the panorama packet206based upon the set of input images202, the camera pose manifold210, the coarse geometry212, the graph214, and/or other information used to generate a panorama.

An embodiment of utilizing a panorama packet is illustrated by an exemplary method300ofFIG. 3. At302, the method starts. A panorama packet (e.g., panorama packet206ofFIG. 2) may comprise a set of input images, a camera pose manifold, a coarse geometry, a graph, and/or other information that may be used to generate a panorama. In an example, an image viewing interface may provide an interactive panorama view experience of a scene depicted by the panorama. For example, a user may explore the scene by navigating the panorama in multi-dimensional space (e.g., three-dimensional space). The image viewing interface may display one or more current views of the scene responsive to the user navigating the panorama.

At304, a request for a current view of the scene associated with the panorama packet is received. For example, the current view may correspond to navigational input through the panorama (e.g., the user may navigate towards a building depicted within the panorama of the scene). At306, responsive to the current view corresponding to an input image within the panorama packet, the current view may be presented based upon the input image (e.g., an input image may adequately depict the building from a view perspective defined by the camera pose manifold).

At308, responsive to the current view of the scene corresponding to a translated view between a first input image (e.g., depicting a first portion of the building) and a second input image (e.g., depicting a second portion of the building), one or more input images are projected onto the coarse geometry to generate a textured coarse geometry. In an example, a first portion of the first input image is blended with a second portion of the second input image to define textured data (e.g., color values) for a first portion of the coarse geometry (e.g., a blending technique performed based upon overlap between the first and second input images). In another example, a portion of the geometry (e.g., an occluded portion) may be inpainted because of a lack of textured data for the portion. The translated view may be obtained based upon a view perspective, defined by the camera pose manifold, of the textured coarse geometry. In an example, the set of input images are projected onto proxy geometry corresponding to a multi-dimensional reconstruction of the scene to create textured proxy geometry, which may be used to fuse the panorama using a shared artificial focal point corresponding to an average center viewpoint of the set of input images. In another example, the set of input images are retained within the panorama packet, and are not stitched and/or fused together during generation of the current view. In this way, the current view is presented based upon the translated view. At310, the method ends.

FIG. 4illustrates an example of a system400for displaying a current view414of a panorama406. The system400may comprise an image viewing interface component404. The image viewing interface component404may be configured to provide an interactive panorama view experience of a scene corresponding to a panorama packet402(e.g., panorama packet206ofFIG. 2). The panorama packet402may comprise a set of input images depicting the scene, such as a building and outdoor space. The panorama packet402may comprise a camera pose manifold, as well as a coarse geometry onto which the set of input images may be projected to generate textured coarse geometry. One or more current views of the scene may be identified using a graph comprised within the panorama packet402(e.g., the graph may comprise relationship information between respective input images). In this way, a current view may be obtained from an input image or the textured coarse geometry (e.g., if the current view is not adequately depicted by a single input image, then the current view may be derived from a translated view of the textured coarse geometry along the camera pose manifold). It may be appreciated that in an example, navigation of the panorama406may correspond to multi-dimensional navigation, such as three-dimensional navigation, and that merely one-dimensional and/or two-dimensional navigation are illustrated for simplicity.

In an example, the set of input images of the panorama packet comprise a first input image408(e.g., depicting a building and a portion of a cloud), a second input image410(e.g., depicting a portion of the cloud and a portion of a sun), a third input image412(e.g., depicting a portion of the sun and a tree), and/or other input images depicting overlapping portions of the scene and/or non-overlapping portions of the scene (e.g., a fourth input image may depict the entire sun, a fifth input image may depict the building and the cloud, etc.). A user may navigate to a top portion of the building depicted by the scene. The image viewing interface component404may be configured to provide the current view414based upon the first input image408, which may adequately depict the top portion of the building.

FIG. 5illustrates an example of a system500for displaying a current view514of a panorama506. The system500may comprise an image viewing interface component504. The image viewing interface component504may be configured to provide an interactive panorama view experience of a scene corresponding to a panorama packet502(e.g., panorama packet206ofFIG. 2). The panorama packet502may comprise a set of input images depicting the scene; a coarse geometry onto which the set of input images may be projected to generate textured coarse geometry; a camera pose manifold; and/or a graph specifying relational information between respective input images. One or more current views of the scene may be identified using a graph comprised within the panorama packet. In this way, a current view may be obtained from an input image or the textured coarse geometry (e.g., if the current view is not adequately depicted by a single input image, then the current view may be derived from a translated view of the textured coarse geometry along the camera pose manifold). It may be appreciated that in an example, navigation of the panorama506may correspond to multi-dimensional navigation, such as three-dimensional navigation, and that merely one-dimensional and/or two-dimensional navigation are illustrated for simplicity.

In an example, the set of input images of the panorama packet comprise a first input image508(e.g., depicting a building and a portion of a cloud), a second input image510(e.g., depicting a portion of the cloud and a portion of a sun), a third input image512(e.g., depicting a portion of the sun and a tree), and/or other input images depicting overlapping portions of the scene and/or non-overlapping portions of the scene (e.g., a fourth input image may depict the entire sun, a fifth input image may depict the building and the cloud, etc.). A user may navigate to towards the cloud and sun depicted within the scene. The current view514of the cloud and sun may correspond to a translated view between the second input image510and the third input image512(e.g., the current view514may correspond to a point along an edge connecting the second input image510and the third input image512within the graph of the panorama packet502). Accordingly, the image viewing interface component504may be configured to project one or more input images onto the coarse geometry to generate the textured coarse geometry. The translated view may be obtained based upon a view perspective, as defined by the camera pose manifold, of the textured coarse geometry. The image viewing interface component504may be configured to provide the current view514based upon the translated view.

FIG. 6illustrates an example of a system600configured for generating an intermediary panorama606to provide an interactive panorama view experience612of a scene. The system600comprises an image viewing interface component604. The image viewing interface component604may be configured to provide the interactive panorama view experience612based upon a set of input images608, coarse geometry, a camera pose manifold, a graph, and/or other information within a panorama packet602. The image viewing interface component604may be configured to generate the intermediary panorama606of the scene using the set of input images. In an example, the intermediary panorama606may correspond to a fused panorama (e.g., one or more input images may be fused together). In another example, the intermediary panorama606may correspond to a stitched panorama (e.g., one or more input images are stitched together). The image viewing interface component604may be configured to blend the intermediary panorama606with the set of input images608using a blending technique610to generate a panorama of the scene. In this way, the interactive panorama view experience612for the panorama may be provided (e.g., a user may be able to explore the scene by multi-dimensional navigation).

FIG. 7illustrates an example of a system700configured for generating a first panorama706of a first region of a scene to provide an interactive panorama view experience712of the scene. The system700comprises an image viewing interface component704. The image viewing interface component704may be configured to provide the interactive panorama view experience712based upon a set of input images, coarse geometry, a camera pose manifold, a graph, and/or other information within a panorama packet702. The image viewing interface component704may be configured to segment the scene into one or more regions based upon a content segmentation technique710. For example, a first region may correspond to a background of the scene and a second region may correspond to a foreground of the scene. The image viewing interface component704may generate the first panorama706for the first region because parallax error and/or other error occurring in the background (e.g., which may result from a stitching process used to generate the first panorama706) may have an adverse, but possibly marginal, effect on visual quality of the interactive panorama view experience712. Accordingly, one or more input images corresponding to the first region may be stitched together to make the first panorama706. The image viewing interface component704may represent the second region using one or more input images708corresponding to the second region. For example, a visualization, such as a spin movie, may be used to represent objects within the second region, such as the foreground of the scene. In this way, the first panorama706may be used for the background and the one or more input images708may be used for the foreground to provide the interactive panorama view experience712.

FIG. 8illustrates an example of a system800configured for generating a first partial panorama806and/or a second partial panorama808to provide an interactive panorama experience812. The system800comprises an image viewing interface component804. The image viewing interface component804may be configured to provide the interactive panorama view experience812based upon a set of input images, coarse geometry, a camera pose manifold, a graph, and/or other information within a panorama packet802. The image viewing interface component804may be configured to cluster respective input images within the panorama packet802based upon an alignment detection techniques810. For example, one or more input images having a first focal point alignment above a threshold may be grouped into a first cluster; one or more input images having a second focal point alignment above the threshold may be grouped into a second cluster; etc. The image viewing interface component804may be configured to generate the first partial panorama806based upon the first cluster (e.g., the first partial panorama806may correspond to a first portion of the scene depicted by the one or more input images within the first cluster). The image viewing interface component804may be configured to generate the second partial panorama808based upon the second cluster (e.g., the second partial panorama808may correspond to a second portion of the scene depicted by the one or more input images within the second cluster). In this way, the first partial panorama806(e.g., to display a current view corresponding to the first portion of the scene) and/or the second partial panorama808(e.g., to display a current view corresponding to a second portion of the scene) may be used to provide the interactive panorama view experience.

Still another embodiment involves a computer-readable medium comprising processor-executable instructions configured to implement one or more of the techniques presented herein. An example embodiment of a computer-readable medium or a computer-readable device that is devised in these ways is illustrated inFIG. 9, wherein the implementation900comprises a computer-readable medium908, such as a CD-R, DVD-R, flash drive, a platter of a hard disk drive, etc., on which is encoded computer-readable data906. This computer-readable data906, such as binary data comprising at least one of a zero or a one, in turn comprises a set of computer instructions904configured to operate according to one or more of the principles set forth herein. In some embodiments, the processor-executable computer instructions904are configured to perform a method902, such as at least some of the exemplary method100ofFIG. 1and/or at least some of the exemplary method300ofFIG. 3, for example. In some embodiments, the processor-executable instructions904are configured to implement a system, such as at least some of the exemplary system200ofFIG. 2, at least some of the exemplary system400ofFIG. 4, at least some of the exemplary system500ofFIG. 5, at least some of the exemplary system600ofFIG. 6, at least some of the exemplary system700ofFIG. 7, and/or at least some of the exemplary system800ofFIG. 8, for example. Many such computer-readable media are devised by those of ordinary skill in the art that are configured to operate in accordance with the techniques presented herein.

Generally, embodiments are described in the general context of “computer readable instructions” being executed by one or more computing devices. Computer readable instructions are distributed via computer readable media as will be discussed below. Computer readable instructions are implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform particular tasks or implement particular abstract data types. Typically, the functionality of the computer readable instructions are combined or distributed as desired in various environments.

FIG. 10illustrates an example of a system1000comprising a computing device1012configured to implement one or more embodiments provided herein. In one configuration, computing device1012includes at least one processing unit1016and memory1018. In some embodiments, depending on the exact configuration and type of computing device, memory1018is volatile, such as RAM, non-volatile, such as ROM, flash memory, etc., or some combination of the two. This configuration is illustrated inFIG. 10by dashed line1014.

In other embodiments, device1012includes additional features or functionality. For example, device1012also includes additional storage such as removable storage or non-removable storage, including, but not limited to, magnetic storage, optical storage, and the like. Such additional storage is illustrated inFIG. 10by storage1020. In some embodiments, computer readable instructions to implement one or more embodiments provided herein are in storage1020. Storage1020also stores other computer readable instructions to implement an operating system, an application program, and the like. Computer readable instructions are loaded in memory1018for execution by processing unit1016, for example.

Device1012includes input device(s)1024such as keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, or any other input device. Output device(s)1022such as one or more displays, speakers, printers, or any other output device are also included in device1012. Input device(s)1024and output device(s)1022are connected to device1012via a wired connection, wireless connection, or any combination thereof. In some embodiments, an input device or an output device from another computing device are used as input device(s)1024or output device(s)1022for computing device1012. Device1012also includes communication connection(s)1026to facilitate communications with one or more other devices.

It will be appreciated that layers, features, elements, etc. depicted herein are illustrated with particular dimensions relative to one another, such as structural dimensions and/or orientations, for example, for purposes of simplicity and ease of understanding and that actual dimensions of the same differ substantially from that illustrated herein, in some embodiments.

Further, unless specified otherwise, “first,” “second,” or the like are not intended to imply a temporal aspect, a spatial aspect, an ordering, etc. Rather, such terms are merely used as identifiers, names, etc. for features, elements, items, etc. For example, a first object and a second object generally correspond to object A and object B or two different or two identical objects or the same object.

Also, although the disclosure has been shown and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based upon a reading and understanding of this specification and the annexed drawings. The disclosure includes all such modifications and alterations and is limited only by the scope of the following claims.