Determining two-dimensional images using three-dimensional models

Systems and methods for determining two-dimensional (2D) images are presented. For instance, data indicative of a three-dimensional (3D) model of a geographic area can be obtained. A 2D output image can be generated depicting at least a portion of the geographic area based at least in part on the 3D model. Each pixel in the output image can then be reprojected to the 3D model. A plurality of aerial images depicting the geographic area can be obtained. A source image can then be determined for each pixel in the output image from the plurality of aerial images. The source image can be determined based at least in part on the reprojection of the pixel in the output image to the three-dimensional model.

FIELD

The present disclosure relates generally to aerial imagery, and more particular to determining images of a geographic area based at least in part on aerial imagery.

BACKGROUND

Geographic information systems provide for the archiving, retrieving, and manipulating of data that has been stored and indexed according to geographic coordinates of its elements. A geographic information system can be used for storing, manipulating, and displaying graphical representations of a geographic area. In particular, such graphical representations can include a two-dimensional (2D) image of the geographic area. For instance, such 2D image can be a mosaic image. An interactive geographic information system can present the image of the geographic area to a user in a suitable user interface, such as a browser. A user can navigate the mosaic image by controlling a virtual camera that specifies what portion of the mosaic image is rendered and presented to a user.

Conventional techniques for creating such images can include capturing a plurality of aerial images depicting overlapping portions of the geographic area and stitching projections of the aerial images on smoothed terrain to create an output image. As another example, images can be created by rendering a textured three-dimensional (3D) model of the geographic area from a desired viewing angle. Such 3D model can include a polygon mesh, such as a triangle mesh, used to model the geometry (e.g. terrain, buildings, and other objects) of the geographic area. Geographic imagery, such as aerial or satellite imagery, can be texture mapped to the 3D model so that the 3D model provides a more accurate and realistic representation of the scene.

However, such conventional techniques may cause inconsistent depictions of the geographic area. For instance, such techniques may result in mosaic images having visible seams and/or buildings that lean in inconsistent manners. As another example, such techniques can result in mosaic images that include unwanted artifacts or flickering.

SUMMARY

One example aspect of the present disclosure is directed to a computer-implemented method of determining a mosaic image. The method includes obtaining, by the one or more computing devices, data indicative of a three-dimensional model of the geographic area. The method further includes generating, by the one or more computing devices, a two-dimensional output image depicting at least a portion of the geographic area based at least in part on the three-dimensional model of the geographic area. The method further includes reprojecting, by the one or more computing devices, each pixel in the output image to the three-dimensional model. The method further includes obtaining, by the one or more computing devices, a plurality of aerial images depicting the geographic area. The method further includes determining, by the one or more computing devices, for each pixel in the output image, a source image from the plurality of aerial images based at least in part on the reprojection of the pixel in the output image to the three-dimensional model.

Other example aspects of the present disclosure are directed to systems, apparatus, tangible, non-transitory computer-readable media, user interfaces, memory devices, and electronic devices for determining mosaic images and/or mosaic image textures from aerial images.

DETAILED DESCRIPTION

Example aspects of the present disclosure are directed to determining a two-dimensional (2D) “mosaic” image of a geographic area. In particular, data indicative of a three-dimensional (3D) model of the geographic area can be obtained. An output image of the geographic area can then be generated at least in part by capturing one or more virtual snapshots of the 3D model from a desired perspective or viewpoint. Each pixel in the output image can be reprojected onto the 3D model. In particular, a camera ray associated with each pixel in the output image can be traced back to the corresponding source point on the 3D model. A texture of the image mosaic can then be determined. In particular, determining the texture can include selecting a source image for each pixel in the output image. The source image can be selected from a plurality of aerial images that depict the geographic area.

More particularly, the data indicative of a 3D model of the geographic area can be a polygon mesh of the geographic area. The polygon mesh can include a plurality of polygons (e.g. triangles) interconnected by vertices and edges to model the geometry of the geographic area. For instance, in some implementations, the polygon mesh can be generated at least in part from a plurality of aerial images depicting the geographic area. The plurality of aerial images may depict overlapping portions of the geographic area. The aerial images may be captured, for instance, by one or more airplanes, helicopters, unmanned aerial vehicles, satellites, air balloons, and/or various other aircrafts. The aerial images may be captured from different perspectives, such as nadir perspectives and/or various oblique perspectives.

An output image can be generated based at least in part on the 3D model. The output image can represent at least a portion of the geographic area from a desired viewpoint. For instance, the desired view point may be a nadir viewpoint, an oblique viewpoint, or other viewpoint. A nadir viewpoint can be a viewpoint having camera rays that project directly downward. An oblique viewpoint may be a viewpoint having camera rays that project downward at an angle. In some implementations, desired viewpoint may be an orthogonal oblique viewpoint having camera rays that project downward in the same direction at the same angle.

For instance, the output image may be generated by capturing a plurality of snapshots of the 3D model using a virtual camera. The one or more snapshots can each depict a portion of the geographic area. The one or more snapshots can be taken from the perspective of the desired viewpoint. In some implementations, the virtual camera can be a perspective camera or an orthographic camera. In this manner, the output image can be a perspective projection or an orthographic projection of the 3D model to the desired view point.

In other implementations, the camera may be associated with a hybrid perspective-orthographic projection that corresponds to various aspects of a perspective camera and an orthographic camera. For instance, the hybrid perspective-orthographic projection can have associated camera rays that originate from different points, and project in different directions relative to the 3D model. In this manner, towards the center of a particular snapshot or tile associated with the hybrid perspective-orthographic projection, the projection may approximate a perspective or pinhole projection, and towards the edges of the snapshot or tile, the projection may approximate an orthographic projection.

According to example aspects of the present disclosure, each pixel of the generated output image can then be reprojected onto the 3D model. In particular, each pixel of the output image can be reprojected onto a corresponding source point of the 3D model. For instance, in some implementations, a camera ray associated with each pixel can be traced from the pixel in the output image to the source point of the 3D model. One or more candidate source images can be identified for each pixel of the output image based at least in part on the reprojection. In particular, in some implementations, each aerial image of the plurality of aerial images that sees the source point of the 3D model can be identified as a candidate source image for the corresponding pixel of the output image.

A candidate source image can then be selected as a source image for each pixel. For instance, a candidate source image can be selected as source image based at least in part on the pose of the candidate source image. As used herein, the pose of an image refers to the position and orientation of a camera that captured the image relative to a reference. In this manner, the source image can be selected by determining a direction of the camera rays in the candidate source images that are associated with the pixels corresponding to the source point in the 3D model. For instance, in some implementations, the source image can be selected based at least in part on a consistency in direction between the camera ray of the source image and the camera ray associated with output image (e.g. the camera ray projecting from the corresponding pixel of the output image to the source point of the 3D model). In other implementations, the source image may be selected based at least in part on a consistency in direction between the camera ray of the source image and parallel to the normal of the viewpoint.

As another example, the source image for a pixel may further be selected based at least in part on the source image selected for one or more neighboring pixels. For instance, the source image for the pixel can be selected such that the same source image is used in neighboring pixel(s). In this manner, the source image can be selected to minimize or reduce a number of source images used and to increase the consistency between source images used for neighboring pixels.

In some implementations, the source images may be selected based at least in part on an optimization having simultaneous objectives of increasing the consistency in the directions of the camera rays associated with the source image pixels and the corresponding mosaic pixels, and increasing the consistency of source images used for neighboring pixels. In this manner, the source image for a pixel can be selected based at least in part on a trade off between the two objectives of the optimization.

Once the source images for each pixel in the output image are determined, an image, such as a mosaic image, can be generated using the selected source images. For instance, the source images for each pixel can be mapped to the corresponding pixels of the output image to generate the mosaic image. The mosaic image can then be provided for display, for instance, in a user interface associated with a geographic information system. For instance, the image may be provided for display in a viewport associated with the geographic information system that allows a user to interact with the image. For instance, the user may be able to select a desired viewing angle (e.g. nadir, oblique, etc.), zoom in or out, pan in various directions, and/or perform various other interactions with the image.

With reference now to the figures, example aspects of the present disclosure will be discussed in greater detail. For instance,FIG. 1depicts an overview of an example system100for determining 2D images according to example embodiments. The system100can include a server110for hosting a geographic information system120. The server110can be any suitable computing device, such as a web server. The server110can be in communication with a user device130over a network140, such as the Internet. The user device130can be any suitable computing device, such as a laptop, desktop, smartphone, tablet, mobile device, wearable computing device, or other computing device.

The server110can host an interactive geographic information system120that serves geographic data stored, for instance, in a geographic database118. For instance, the geographic database118can include geographic data for rendering a two-dimensional (2D) mosaic image of a geographic area. The geographic data can include a plurality of source images depicting the geographic area, a three-dimensional (3D) model (e.g. polygon mesh) representing the geometry of the geographic area, and one or more textures for mapping to the polygon mesh. The geographic data can be stored in a hierarchical tree data structure, such a quadtree or octree data structure, that spatially partitions the geographic data according to geospatial coordinates.

The polygon mesh can include a plurality of polygons (e.g. triangles) interconnected by vertices and edges to model the geometry of the geographic area. The polygon mesh can be represented in any suitable format, such as a depth map, height field, closed mesh, signed distance field, or any other suitable type of representation. The polygon mesh can be a stereo reconstruction generated from aerial imagery of the geographic area. The imagery can be taken by overhead cameras, such as from a satellite or aircraft, at various oblique or nadir perspectives. In the imagery, features are detected and correlated with one another. The points can be used to determine a stereo mesh from the imagery. In this way, a three-dimensional model can be determined from two-dimensional imagery. The geographic data can also include a plurality of textures that can be mapped to the polygon mesh to provide a three-dimensional model of the geographic area. The textures can be generated from aerial or satellite imagery of the geographic area.

System100can further include a mosaic generator122configured to generate a 2D image, such as a 2D mosaic image, of the geographic area. For instance, in example embodiments, the mosaic generator122can be configured to generate an output image by creating a projection of the 3D model from a desired viewpoint. In some implementations, the projection can be created by capturing one or more snapshots of the geographic area with a virtual camera from the perspective of the viewpoint. An output image can be generated using the snapshots. Each pixel in the output image can then be traced back to a corresponding source point on the 3D model. A source image can be selected for each pixel in the output image from one or more candidate source images (e.g. from the plurality of aerial images) that see the source point. A mosaic image can then be generated based at least in part on the source images. In some implementations, the mosaic image can be generated by texturing the output image using the source images.

Referring still toFIG. 1, the user device130can implement a user interface134that allows a user132to interact with the geographic information system120hosted by the server110. The user interface134can be a browser or other suitable client application that can render an image of the geographic area on a display device associated with the user device130. The user132can interact with the user interface134to pan, tilt, zoom, or otherwise navigate to view different portions of the mosaic image of the geographic area from different perspectives.

In response to the user interactions with the user interface134, requests for geographic data can be provided from the user device130over the network140to the server110. The server110can provide geographic data, such as a mosaic image and one or more textures, to the user device130. The user device130can then render one of the textures in conjunction with the mosaic image from a desired viewpoint to present the mosaic image of the geographic area to the user.

FIG. 2depicts a flow diagram of an example method (200) of determining an image of a geographic area according to example embodiments of the present disclosure. Method (200) can be implemented by one or more computing devices, such as one or more of the computing devices depicted inFIG. 6. In particular implementations, the method (200) can be implemented by the geographic information system120and mosaic generator122ofFIG. 1. In addition,FIG. 3depicts steps performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the steps of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, or modified in various ways without deviating from the scope of the present disclosure.

At (202), method (200) can include obtaining data indicative of a 3D model. For instance, the 3D model can be represented as a polygon mesh. The polygon mesh can include a plurality of polygons (e.g. triangles) interconnected by vertices and edges to model the geometry of the geographic area. The polygon mesh can be represented in any suitable format, such as a depth map, height field, closed mesh, signed distance field, or any other suitable type of representation. The polygon mesh can be a stereo reconstruction generated from aerial imagery depicting the geographic area.

At (204), method (200) can include generating an output image depicting at least a portion of the geographic area based at least in part on the 3D model. For instance, in some implementations, the output image can be generated by rendering the 3D model from a desired view angle or viewpoint. In some implementations, the output image can be generated by creating a projection of the 3D model (or a portion of the 3D model) from the desired view angle or viewpoint. The projection can be created by taking one or more snapshots of the 3D model using a virtual camera (e.g. a perspective virtual camera, an orthographic virtual camera, or a hybrid perspective-orthographic virtual camera). In particular, the one or more snapshots can be taken from the desired viewpoint or view angle.

At (206), method (200) can include reprojecting each pixel in the output image to the 3D model. In particular, each pixel in the output image can have a corresponding source point on the 3D model. The source point can represent the same surface of the geographic area as is depicted in the pixel of the output image. Reprojecting each pixel in the output image to the 3D model can include tracing a camera ray from the pixel to the source point. The camera ray can be a vector projecting from the pixel in the output image to the source point on the 3D model. In this manner, the camera ray can be indicative of a viewing angle associated with the output image relative to the 3D model and/or the geographic area.

For instance,FIG. 3depicts a plurality of camera rays302being traced to a 3D model304. In particular, each camera ray302can correspond to a pixel in an output image (not shown) generated according to example aspects of the present disclosure. As indicated above, each camera ray302can be traced from the pixel in the output image to the corresponding source point on the 3D model304. As shown, the camera rays302are directed equally in the same direction, at the same angle (e.g. camera rays302are parallel). This can be indicative of an orthographic output image. It will be appreciated that the output image may be associated with a perspective projection or a hybrid perspective-orthographic projection. In such instances, camera rays302may be directed at different angles.

Referring back toFIG. 2, at (208), method (200) can include obtaining a plurality of aerial images. For instance the aerial images can depict at least a portion of a geographic area from various nadir or oblique perspectives. In some implementations, the aerial images may depict overlapping portions of the geographic area. In some implementations, at least a portion of the aerial images may be the aerial images used in creating the 3D model. As indicated above, the aerial images can be captured by an image capture device associated with a satellite or aircraft, such as an airplane, helicopter, unmanned aerial device or other aircraft.

At (210), method (200) can include determining a source image for each pixel in the output image. The source image for a pixel can be determined based at least in part on the camera ray associated with the pixel. The source image can be selected from one or more candidate source images associated with the pixel. The one or more candidate source images can be the aerial images from the plurality of aerial images that see (e.g. depict the same portion of the geographic area) the source point in the 3D model.

As will be described in greater detail with respect toFIG. 5, in some implementations, a candidate source image can be selected as a source image based at least in part on an alignment between a direction of the camera ray associated with the pixel and the pose of the candidate image. In further implementations, a candidate source image can be selected as the source image based at least in part on a source image selected for one or more other pixels that neighbor the pixel.

At (212), method (200) can include generating a “mosaic” image based at least in part on the determined source images. In some implementations, the mosaic can be generated by mapping the source images to the corresponding pixels of the output image. In this manner, the output image can be textured using one or more source images that view the corresponding surface from a viewing direction that aligns (e.g. within a threshold) with the perspective of the output image. At (214), method (200) can include storing the generated mosaic image in a memory for future use. For instance, the mosaic may be provided to a user device for rendering by the user device in a user interface associated with a geographic information system in response to a user interaction with the user interface.

For instance,FIG. 4depicts an example image400according to example embodiments of the present disclosure. For instance, image400can be a mosaic image generated in accordance with example aspects of the present disclosure. In particular, image400can be generated using 3D model402. As shown model402includes pixels404and406. Image400includes corresponding pixels408and410. As shown, pixels404and408depict a rooftop corner of building412, and pixels406and410depict a portion of the street at the foot of building412. As further shown, although they correspond to the same points, pixels404and406are separated by a greater distance than pixels408and410. In this manner, it can be important to select the same source image for adjacent or neighboring pixels to preserve visual continuity.

FIG. 5depicts a flow diagram of an example method (500) of determining one or more source images for a pixel to be implemented in a mosaic image. In particular, method (500) can correspond to example implementations of (210) of method (200) (e.g. for determining a source image for a single pixel). At (502), method (500) can include identifying one or more candidate source images for the pixel. As indicated above, the candidate source images can be one or more images from the plurality of aerial images that see (e.g. depict the same surface as) the source point on the 3D model. In this manner, each of the candidate source images can depict the surface or other point depicted by the pixel in the output image. For instance, the candidate source images can depict the surface depicted by the pixel in the output image from various perspectives.

At (504), method (500) can include identifying pose information associated with each of the candidate source images. For instance, the pose of each candidate source image can be accessed from a memory or determined using a suitable posing technique. The pose for each candidate source image can provide position and orientation information for the image capture device that captured the candidate source image. The pose of the images can provide data useful for selecting source images for generating a textured mosaic, such as the camera location and camera view direction of the candidate source image relative to various portions of the output image and/or polygon mesh. In particular, the pose for an image can be used to determine a camera ray associated with the pixel depicting the appropriate surface of the geographic area. In some implementations, the camera ray can be indicative of a viewing angle of the candidate image relative to the geographic area.

At (506), method (500) can include identifying one or more source images associated with one or more adjacent pixels. For instance, the one or more adjacent pixels can be pixels that neighbor the current pixel (e.g. the pixel for which a source image is currently being selected). In some implementations, identifying one or more source images associated with one or more adjacent pixels can include identifying source images that have been selected for one or more neighboring pixels. In some implementations, identifying one or more source images associated with one or more adjacent pixels can include identifying one or more source images to be selected for one or more adjacent pixels based at least in part on the pose of the images.

At (508), method (500) can include selecting a source image for the pixel. For instance, in some implementations the source image can be selected based at least in part on the direction of the camera ray. In particular, the source image can be the candidate source image having the camera ray that most closely aligns with the viewpoint associated with the output image. As another example, the source image can be a candidate source image that aligns with the viewpoint within a threshold. In further implementations, the source image can be selected based at least in part on the one or more source images associated with the adjacent pixel(s). For instance, the source image for the current pixel can be selected to be the same as a source image used for the adjacent pixel(s). In this manner, the source images for each pixel can be selected to reduce or minimize a number of source images used to texture the mosaic.

In some implementations, the source image can be selected based at least in part on the camera rays of the candidate source images and the neighboring source images. For instance, the source image can be selected based on an optimization having simultaneous objectives of increasing an alignment between the source image viewing angle (e.g. as evidenced by the camera ray associated with the source image) and the viewpoint of the output image, and decreasing a total number of source images used to texture or populate the mosaic. In particular, the optimization can simultaneously take into account the alignment between the source image and the output image, and one or more source images selected (or to be selected) for one or more adjacent pixels in the output image. In this manner, solving the optimization can include determining a trade-off between selecting the source image that most closely aligns with the viewpoint of the output image, and selecting the same source image for adjacent pixels.

In some implementations, method (500) can be repeated for each pixel of the output image to determine a source image for each pixel in the output image. For instance, in some implementations, the source images can be determined on a pixel-by-pixel basis. For instance, the source images may be selected in a serial manner, in which source images re determined for one pixel at a time. In other implementations, at least a portion of the source images may be determined in a parallel manner. For instance, source images may be determined simultaneously for two or more pixels.

FIG. 6depicts an example computing system600that can be used to implement the methods and systems according to example aspects of the present disclosure. The system600can be implemented using a client-server architecture that includes a server610that communicates with one or more client devices630over a network640. The system600can be implemented using other suitable architectures, such as a single computing device.

The system600includes a server610, such as a web server. The server610can host a geographic information system, such as a geographic information system associated with a mapping service. The server610can be implemented using any suitable computing device(s). The server610can have one or more processors612and one or more memory devices614. The server610can also include a network interface used to communicate with one or more client devices630over the network640. The network interface can include any suitable components for interfacing with one more networks, including for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.

The one or more processors612can include any suitable processing device, such as a microprocessor, microcontroller, integrated circuit, logic device, or other suitable processing device. The one or more memory devices614can include one or more computer-readable media, including, but not limited to, non-transitory computer-readable media, RAM, ROM, hard drives, flash drives, or other memory devices. The one or more memory devices614can store information accessible by the one or more processors612, including computer-readable instructions616that can be executed by the one or more processors612. The instructions616can be any set of instructions that when executed by the one or more processors612, cause the one or more processors612to perform operations. For instance, the instructions616can be executed by the one or more processors612to implement the geographic information system120and mosaic generator122ofFIG. 1.

As shown inFIG. 6, the one or more memory devices614can also store data618that can be retrieved, manipulated, created, or stored by the one or more processors612. The data618can include, for instance, aerial images, 3D model, and other data. The data618can be stored in one or more databases. The one or more databases can be connected to the server610by a high bandwidth LAN or WAN, or can also be connected to server610through network640. The one or more databases can be split up so that they are located in multiple locales.

The server610can exchange data with one or more client devices630over the network640. Although two client devices630are illustrated inFIG. 6, any number of client devices630can be connected to the server610over the network640. Each of the client devices630can be any suitable type of computing device, such as a general purpose computer, special purpose computer, laptop, desktop, mobile device, navigation system, smartphone, tablet, wearable computing device, a display with one or more processors, or other suitable computing device.

Similar to the server610, a client device630can include one or more processor(s)632and a memory634. The one or more processor(s)632can include one or more central processing units (CPUs), graphics processing units (GPUs) dedicated to efficiently rendering images or performing other specialized calculations, and/or other processing devices. The memory634can include one or more computer-readable media and can store information accessible by the one or more processors632, including instructions636that can be executed by the one or more processors632and data638. For instance, the memory634can store instructions636for implementing a user interface module for textured mosaic images determined according to example aspects of the present disclosure.

The client device630ofFIG. 6can include various input/output devices for providing and receiving information from a user, such as a touch screen, touch pad, data entry keys, speakers, and/or a microphone suitable for voice recognition. For instance, the client device630can have a display device635for presenting a user interface displaying mosaic images according to example aspects of the present disclosure.

The client device630can also include a network interface used to communicate with one or more remote computing devices (e.g. server610) over the network640. The network interface can include any suitable components for interfacing with one more networks, including for example, transmitters, receivers, ports, controllers, antennas, or other suitable components.

The network640can be any type of communications network, such as a local area network (e.g. intranet), wide area network (e.g. Internet), cellular network, or some combination thereof. The network640can also include a direct connection between a client device630and the server610. In general, communication between the server610and a client device630can be carried via network interface using any type of wired and/or wireless connection, using a variety of communication protocols (e.g. TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g. HTML, XML), and/or protection schemes (e.g. VPN, secure HTTP, SSL).