System and method of texturing a 3D model from video

A method and system for applying photo texture acquired from a video resource to a 3D model operates within a 3D modeling system. The modeling system includes a modeling application operating on a workstation and a storage device containing a video resource. A 3D model is created or edited within the 3D modeling system. For a selected surface, the method and system allow selection of a video resource, selection of a video frame of the video resource, and selection of an area of the video frame to use as a photo texture to apply to the selected surface. The selected area of the video frame is copied and mapped to the selected surface of the 3D model.

FIELD OF TECHNOLOGY

The present disclosure relates to photo texturing of three-dimensional models and, more specifically, to a system and a method configured to photo texture a selected surface of the three-dimensional model by mapping to the selected surface an image of the surface acquired from a video frame of a video resource in which the selected surface appears.

BACKGROUND

Advances in computer-aided three-dimensional (3D) modeling—both in terms of the number of available software applications and the ease with which the available applications can be used—have facilitated the proliferation of 3D digital modeling through a number of technological areas. Using computer-aided drafting (CAD) programs, digital modeling software, and the like, professionals and novices alike can generate digital, 3D models of buildings, spaces, objects, etc.

One context in which computer-aided 3D modeling has found particular utility is the field of digital cartography. One function of digital cartography is to generate maps that provide accurate visual depictions of an area. In particular, some maps now allow a user to visualize a particular location as it actually appears, by integrating photographic elements from satellite imagery, aerial imagery, and surface imagery with traditional cartographic elements (e.g., information about road locations, geographic features, etc.) and advanced location data (e.g., from the Global Positioning System (GPS)). In some instances, digital maps may allow a user to view an element such as, for example, a building, as a 3D model.

Generally, 3D models depicted in digital maps are created using computer-aided 3D modeling programs. In some instances, the 3D models are untextured. That is, the 3D models depict the shape of the building, but do not accurately depict the building's visual characteristics, such as the building's façade. In other instances, the 3D models are textured using, for example, one or more photographic images of the real-world object modeled. The process of applying a photographic image to a model of a real-world object is referred to as “photo texturing.”

The tedious and time-consuming nature of the photo texturing process limits the capacity of most digital map systems to implement accurate 3D modeling of mapped areas. To model and texture a given object, images of the object must be acquired, usually from several angles. In some instances, this may require traveling to the object. Moreover, acquired images may be from a sub-optimal angle, or may not capture a clear view of the surface to be photo textured.

SUMMARY

In an embodiment, a computer-implemented method for applying photo texturing to a 3D model of a real-world object includes receiving a selection of a surface of the 3D model and receiving a request to apply to the selected surface a photo texture acquired from a video resource. Thereafter, the method receives a selection of the video resource from which to acquire the photo texture and a selection of a video frame of the video resource. The method proceeds to receive a selection of an area of the selected video frame to apply as the photo texture to the selected surface. Having received the selection of the area, the method applies the selected area of the selected video frame to the selected surface.

If desired, the computer-implemented method receives geolocation information for a point in a model space in which the 3D model is created and initiates a search of metadata associated with a plurality of video resources to determine one or more video resources according to the received geolocation information. Further, if desired, the computer-implemented method receives geolocation information for a point in a model space in which the 3D model is created and uses information stored as metadata with the selected video resource to determine a video frame of the video resource to display.

In another embodiment, a system includes a storage device storing a plurality of video resources, a display device, and a processor communicatively coupled to the storage device and to the display device. The system also includes a memory device storing a set of machine readable instructions executable by the processor and operable to cause the processor to facilitate the creation of a 3D model in a 3D space represented on the display device, to receive a selection of a surface of the 3D model, and to receive a request to apply to the selected surface a photo texture acquired from a video resource stored on the storage device. The instructions also cause the processor to receive a selection of the video resource from which to acquire the photo texture and to receive a selection of a video frame of the video resource. Lastly, the instructions cause the processor to receive a selection of an area of the selected video frame to apply as the photo texture to the selected surface, and to apply the selected area of the selected video frame to the selected surface.

In yet another embodiment, a tangible, non-transitory computer-readable medium stores machine instructions operable to cause a processor to create a 3D digital model of a real-world object, to receive a selection of a surface of the 3D model, and to receive a request to apply to the selected surface a photo texture acquired from a video frame of a video resource. In response to the request, the instructions cause the processor to receive a selection of the video resource that includes the video frame from which to acquire the photo texture and to receive a selection of the video frame of the video resource. Thereafter, the instructions cause the processor to receive a selection of an area of the selected video frame to apply as a photo texture to the selected surface, and apply the selected area of the selected video frame to the selected surface.

DETAILED DESCRIPTION

A computer-implemented method or a computer system or a computer-readable medium storing a set of instructions for execution on a processor operates to apply a photo texture, acquired from a video resource, to one or more surfaces of a 3D model of a real-world object. Upon receipt of a selection of a surface of the 3D model, and a request to apply to the selected surface a photo texture from a video resource, the method or apparatus receives a selection of a video resource from which a video frame will be selected to use to photo texture the selected surface. The method or apparatus receives a selection of the video frame to use, and identifies an area of the video frame to apply to the selected surface. The method or apparatus then photo textures the selected surface by applying the identified area of the video frame to the selected surface.

In various embodiments, the method and apparatus may receive geolocation information for one or more points in a 3D space in which the 3D model is created. The method and apparatus may initiate a search of video resources stored locally and/or video resources stored remotely (e.g., web-hosted). The method and apparatus may also initiate a search of metadata of a selected video resource to determine one or more potential video frames from which an area of the video frame could be selected.

Generally speaking, a 3D modeling system allows a user to design 3D models, for example, for architectural, civil and mechanical design purposes, or for digital modeling and/or rendering of any 3D object. For example, a 3D modeling system may allow an architect to render a new building design in 3D such that the building can be displayed from any angle. An interior designer may design a space and display the space to a client as it would appear as the client entered the space from one or more directions, in various lighting conditions, or with various design options. An engineer may design a consumer device, such as a mobile phone, viewing it from all angles to gauge its aesthetic appeal.

Generally, the realism of such models is limited by the amount of time and effort available to provide detail to each of the modeled surfaces. For example, an architect may spend countless hours programming the surfaces of a 3D model to have just the right surface texturing (e.g., a brick façade), to include doors and windows, to reflect light in just the right manner, etc. Sometimes, however, it is desirable to model an existing building (as opposed to a building being designed) in a fast and efficient manner. This may be the case where, for instance, an architect is modeling the site of a new building, e.g., including the existing buildings around the structure being designed. In these instances, it is significantly more efficient to apply to the models images of the existing buildings as they exist. By doing so, it is possible to create 3D models with a high degree of realism in an expeditious manner. The process of adding surface texture (e.g, color, detail, etc.) to a digitally rendered 3D model is called “texture mapping.” “Photo-texturing” is the process of texture mapping a digital image (such as a scanned photograph, a digital photograph, etc.) to a surface of a digitally rendered 3D model.

Of course, in order to apply a photo-texture to a surface of a 3D model, it is necessary first to have a photograph of the surface. Generally, where the object is a structure, this may require traveling to the location of the structure and acquiring suitable images (e.g., with a digital camera) of any surface of the structure that is to be photo-textured. Alternatively, one or more existing images may be used to photo-texture a surface or surfaces of the structure.

In any event, a user desiring to apply a photo texture to a surface of a 3D model, having determined an image to apply, may select a portion of the image that depicts a surface corresponding to the surface of the 3D model to which the photo texture will be applied. Selection of the area could be accomplished by drawing a polygonal (e.g., rectangular) shape over the digital image, locating crosshairs indicative of vertices of such a polygon, drawing a freehand selection area, sizing and/or locating a pre-determined polygon (e.g., corresponding to the shape of the surface to be photo-textured), etc. The selected portion of the image may then be mapped to the surface of the 3D model by UV mapping, as discussed below.

A database of digital photographic and/or video images may provide a ready source of existing images that could be used to photo-texture surfaces of structures modeled in 3D. The database may include photographic and/or video images of structures within some area. For example, a database may include photographic and/or video images of structures on one block, images of structures in a particular city, images of structures in a particular country, etc. In some embodiments, for ease of locating in the database a particular structure, each of the photographic and/or video images may be associated with information (i.e., the images may be “tagged”) indicating the location from which the image was captured. The location indication may be in any appropriately searchable form including, by way of example and not limitation: latitude and longitude, intersection, street address, zip code, distance and direction from a known, fixed point, etc. The identification of a geographic location of a real-world object (e.g., a structure, a camera) is referred to herein as “geolocation.” An object for which a geographic location has been identified is said to be “geolocated.” The association (e.g., by metadata) of geolocation information with a file (e.g., a digital image) is referred to herein as “geotagging.” A file (e.g., an image file) having associated metadata indicative of a geolocation (e.g., a geolocation of the camera at the time the image was captured) is said to be “geotagged.”

The database of digital photographic and/or video images may be populated with geotagged images recorded by, for example, a digital camera, operable to record still images or video images, and having a GPS unit that allows the camera to geolocate. The digital camera may be a digital still camera, a digital video camera, a digital single-lens reflex (DSLR) camera, etc. Throughout the remainder of this description, all devices for capturing still or video imagery for population in the database of digital still and video imagery will be referred to, respectively, as image capture devices and video capture devices and, collectively, as capture devices. In some embodiments, the metadata associated with the images may include more information than just a simple location of the image or video capture device at the time the images were captured. For example, the metadata may include the direction the capture device was facing when the images were captured, the focal point of the lens, the height and/or angle of the capture device with respect to the ground, the zoom level of the lens, the distance from the capture device to nearby objects, etc. In some embodiments, some or all of the photographic and/or video images may have less or no associated metadata and, specifically, may lack metadata indicative of the direction the capture device was facing as the images were captured and may lack metadata indicative of the location of the capture device when the images were captured.

FIG. 1A, depicts a block diagram of an embodiment of a system10for photo texturing a 3D model. The system10includes a user-side system12and a server side system14communicatively coupled to each other by network16. The user side system12includes a user workstation18. The user workstation18includes, or is communicatively coupled to, a display20and one or more input devices22, such as a keyboard, a mouse, a trackpad and/or touchpad, a touch-sensitive screen, a gesture tracking system, or any other known input method. Of course, the input device22may be separate from the display20or may include the display20(e.g., where the input device22is a touch-sensitive display). The user workstation includes a processor24, a memory26, and an input/output (I/O) interface28.

The I/O interface28couples the display20and the input devices22to a bus25in the workstation18, which bus25, in turn, communicatively couples the I/O interface28to a processor24and a memory sub-system26. The processor24may be any processor capable of executing one or more programs stored in the memory sub-system26(as described below) to perform the specific tasks associated with the 3D photo-texturing application herein described and, in particular, may be a general purpose processor, a digital signal processor, a field programmable gate array (FPGA), an application specific IC (ASIC), etc. The processor24may operate to retrieve one or more applications30from the memory sub-system26and to execute the applications30. An exemplary workstation18includes an operating system (OS)32such as Windows, Mac OS, Linux, Android, etc. The exemplary workstation18may also store in the memory sub-system26a software application34embodying the 3D modeling application. For simplicity, the term “software application” is used herein to refer both to an instance of a software application executing in the workstation18and to the set of computer instructions that defines the software application. However, it will be understood that while an instance of an application executes in the workstation18, machine-readable instructions of the application are stored on a non-transitory, computer-readable medium such as a persistent memory27, a system cache29, or both. The persistent memory27may be a hard disk, a flash drive, a CD, a DVD, a tape drive, etc. In at least some of the embodiments, the persistent memory27is significantly slower that the system cache29. In particular, the persistent memory27may have one or more of a slower read speed, a slower write speed, a lower bandwidth (i.e., the size of a data block the persistent memory27can supply to the CPU24at one time), etc.

The OS32executes on the processor24(“central processing unit” or “CPU”) that includes one or several CPU cores, depending on the embodiment. The workstation18also includes the system cache29that may be implemented as a quickly-accessible, physical memory such as random access memory (RAM). In general, the system cache29may be provided on a same chip as the CPU24or on a separate chip. In some embodiments, the system cache29is a dedicated memory region on a RAM chip. During operation, the system cache29operates as an active storage, so that software tasks such as the application30access memory pages (or simply “pages”) stored in the system cache29quickly and efficiently. While the application30, including the OS32and the 3D modeling application34, are generally described above as being stored on the persistent memory27and/or the system cache29, the applications30may be stored on the server side system14, on another workstation18on the user-side system12, or some combination of the two.

With continued reference toFIG. 1A, the workstation18and, in particular, the I/O interface28, may include a network interface card (NIC)36via which the workstation18may be coupled by the communication network16, which may be the Internet, for example. Specifically, the workstation18may communicate via the network16with a server40on the server-side system14, which may, in turn, be in communication, for example over a local area network (LAN)44, with one or more databases42.

The server40may be configured similarly to the workstation18, in that it may include one or more displays, CPUs, memory sub-systems, I/O interfaces, input devices, etc. Moreover, while described herein as a single server, the server40may be any number of servers40according to the architecture of the network16, the LAN44, server loading and/or demand parameters, etc. Additionally, multiple servers40need not be affiliated with the same entity or company, need not serve the same purpose, etc. For example, a first server40may be affiliated with a video sharing service, and have a first database42storing video imagery, while a second server40may be affiliated with a photo sharing service and have a second database42storing photo imagery, and a third server40may be affiliated with a digital cartography and navigation service and have a third database42storing maps, satellite imagery, etc.

As described in greater detail below, the server40generally operates to receive from the workstation18, via the network16, one or more requests generated by the 3D modeling application34and, in response to the request, to retrieve information from the database42and transmit the retrieved information back to the workstation18, again via the network16. The server40may complete one or more processing steps before and/or after retrieving the information from the database42, such as, by way of example and not limitation, calculating a value using as input the information from the database42or performing a lookup of a database index value. Additionally, in some embodiments, the server40may store one or more applications, including the 3D modeling application34. In these instances, the workstation18may receive the application34from the server40and execute the application34locally on the workstation18. In alternate embodiments, the application34may be stored and executed on the server40, and the server40may provide only display and/or user-interface data via the network16to the workstation18. That is, the workstation18may serve as a display and/or interface terminal for the application executed on the server40.

In still other embodiments, the application34is stored in the memory sub-system26and executed by the CPU24on the workstation18, and interacts with a second application (not shown) stored and executed on the server40. The second application may be, by way of example and not limitation, an interactive map application that retrieves (e.g., from the database42) and/or renders maps, satellite imagery, terrain data, etc. for display on an output device such as the display20, provides an interactive user interface to allow a user to zoom in on a desired location, pan to a desired location, select various types of data on the map or the satellite image, etc. In one such embodiment, the second application provides two-dimensional and three-dimensional representations of geographic regions. A user may view a two-dimensional satellite image of a certain location and dynamically switch to a three-dimensional view of the location. At the time when the second application transitions from a two-dimensional rendering to a three-dimensional rendering, the second application at least sometimes requests additional resources such as bitmap images, modeling data, etc. Further, in some cases, the second application invokes additional sets of functions, such as rendering functions for three-dimensional graphics, stored in a corresponding dynamically linked library (DLL), for example.

The information retrieved and/or used by the second application may be stored in the database42. Specifically, the database42may include map data, satellite imagery, terrain data, imagery captured from a pedestrian and/or vehicular perspective (e.g., views of buildings taken from a street or a pedestrian walkway), panoramic imagery, video captured from a pedestrian, aerial, or vehicular perspective, etc., any and/or all of which may or may not have associated with it meta-data including: a location from which an image or video was captured; a type of capture device and/or lens used to capture the image or video; an angle of a capture device relative to the ground and/or relative to a velocity of the capture device; a heading and or velocity of a capture device and/or the device's viewport when an image or video is captured; a zoom level of an image or video captured; and any other parameter desired. In some embodiments, the database42includes a data store46of map and satellite images and associated data and a data store48of images and video captured associated data. Of course, the data stores46and48may be on a single database42, may be on separate databases42, or may each or collectively be spread across multiple databases42.

Where the database42and, in particular, the data store46, includes video imagery, each video file, as generally known, comprises a number of individual images captured and stored sequentially such that playback appears to a viewer to show moving imagery. The video imagery may be captured by any of several means and, in particular, may be captured by a video capture device50such as a camera45having a video function, a dedicated video recorder47, or any other device capable of capturing video imagery, including, but not limited to a smart phone49, a tablet computer, a laptop computer, a netbook computer, a portable media player (PMP) device, etc. The video capture device50may be carried by a pedestrian52, mounted on and/or in a vehicle (not shown) (e.g., a bicycle, an automobile, a motorcycle, a cart, a tricycle, a snowmobile, etc.), etc. The video capture device50may, in some embodiments, include a video capture device having hardware and/or software operable to allow the video capture device50to capture a 360-degree view. In an embodiment of the video capture device50, one or more Global Positioning System (GPS) units (not shown) operate to determine the location of the video capture device50at the commencement of recording and/or as video is captured, and data from the GPS unit is stored as metadata with the video in the data store46. The video capture device50may also include various computer equipment (not shown), including one or more processors, memory subsystems, displays, input devices, communication media, etc., which may function to control the video capture device50, capture and store video and metadata, transmit the stored video imagery and metadata to another device (e.g., the data store46), etc. In some embodiments, the video capture device50stores data locally on a memory subsystem (e.g., on a memory card, a recordable optical medium, a FLASH device, etc.) and, at a later time, transfers the captured video data and metadata to the data store46. In some embodiments, the video capture device50is communicatively coupled to the data store46and stores data in the data store46as the data is captured.

In some embodiments, the video capture device50may be communicatively coupled directly to the workstation18via the network interface card36or via another interface on the I/O interface28, either while video data is captured or at a later point in time.FIG. 1Billustrates an embodiment in which the video capture device50is coupled to the NIC36.

With reference now toFIG. 2A, a person using the 3D modeling system to model a structure may create a 3D model of the structure.FIG. 2Adepicts a 3D model100of a structure102set on a ground plane104in a 3D program space106, as the 3D model100might be depicted on the display20. The structure100includes generally planar sides108,110,112, and114that are generally perpendicular to the ground plane104, which is coincident with a bottom face116of the 3D model100. A planar surface, generally parallel to the ground plane104(and to the bottom face116), forms a top face118of the structure102. It is worth noting the ground plane104need not necessarily correspond to the bottom face116of the 3D model100, but could instead correspond to the top face118of the 3D model100, one of the sides108,110,112, and114of the 3D model100and, in fact, need not correspond to any side or face of the 3D model100. However, throughout this specification the 3D model100is described as modeling an above-ground section of a real-world building and, thus, for ease, the ground plane104will be described as corresponding to the bottom face116of the 3D model100.

The system10may allow a user to select a surface of the 3D model100to texture. As used in this specification, the word “texture” refers to application to a surface of detail conveying texture. For example, a surface of a 3D model generally may be textured by applying to the selected surface a brick pattern, a concrete pattern, a glass façade pattern, a wood siding pattern, etc. The 3D modeling application34and, in particular, a video photo texturing routine35(stored, for example, in the non-volatile memory27of the memory sub-system26on the workstation18, and executed by the processor24) described in this specification, textures a surface by applying to the selected surface of the modeled structure102an image of a corresponding surface of a real-world structure (not shown). Throughout this specification, this is referred to as photo-texturizing (or photo texturing). For example, after photo texturing, the side110of the 3D model100depicted inFIG. 2Amay appear as depicted inFIG. 2B. Generally, photo texturing requires acquiring an image (e.g., from a digital photograph or a frame of a video) of the surface to which the photo texture is to be applied (and, possibly, manipulating an apparent viewing angle of the image to match that of the 3D model100), and selecting the portion of the acquired image to apply to the selected surface. This is often a tedious, time-consuming process.

The database42may, in some instances, store geotagged photographic or video imagery that may be used by a 3D modeling application34for photo texturing. For instance, the database42may store photographic or video imagery with associated geotagging metadata that indicates the location at which each image, video, segment of video, and/or frame of video was captured. In some instances, the geotagging metadata may be stored and/or searchable by latitude and longitude, by address, by reference to a particular point or feature (e.g., “100 feet north of the intersection of Main Street and First Avenue”), or by any other desirable reference system. This may allow a user to search for a previously captured photographic or video image of the structure modeled by the 3D model100, rather than requiring the user to acquire the photographic or video image personally. The user may then select the portion of the photographic image or video frame to apply to the selected surface as a texture, manipulate the perceived viewing angle of the selected portion, and apply the selected portion of the image to the selected surface. This remains a tedious (if slightly less time-consuming) process.

If geotagged photographic or video imagery of the 3D model100exists in the database42, the video photo texturing routine35may, in some instances, be able to apply one of the images as texture to the 3D model100. To do this, the video photo texturing routine35must have information about the location of the 3D model100in the 3D program space106of the modeling application34. That is, the 3D model100must be geolocated within the modeling application34. Once the 3D model100is geolocated such that the modeled structure's real-world location is known, the video photo texturing routine35may access the database42, determine whether appropriate imagery exists for the modeled structure, select, or allow a user to select, an appropriate photographic or video image or portion of an image to apply as texture to a selected surface of the 3D model100, and apply the selected image or portion of an image to the selected surface. This process is described in greater detail below.

Referring again toFIG. 2A, the 3D program space106may be depicted on the display20as having an origin120at the intersection of X, Y, and Z-axes122,124, and126, respectively. The 3D model100may be disposed within the 3D program space106such that an intersection of three planes (e.g., the sides114and112and the bottom face116) is situated at the origin120, as depicted inFIG. 2A. Of course, while it may be preferable to some users for the 3D-model100to have a surface (e.g., the bottom face116) coincident with the plane defined by the X- and Y-axes122and124, respectively, such alignment is not required.

The 3D program space106may be geolocated, before or after creating the 3D model100, to establish the real-world location of the modeled structure. For example, a user may operate a user-interface of the modeling application34to cause the modeling application34to commence a geolocation routine37(seeFIGS. 1A,1B) (e.g., by using an input device to “click” a “button” in the user-interface, to select a “geolocate” or “add location” menu item, etc.). The geolocation routine37may cause the display20to display a dialog box, to display a new “window,” to change to display a different screen, etc. The geolocation routine37may be part of or separate from the modeling application34, may be executed locally on the workstation18or remotely on the server40, and may use data stored on the workstation18or data stored in the server40or the database42. In any event, the geolocation routine37allows the user to geolocate the 3D model100in the 3D program space106of the modeling application34. Specifically, the geolocation routine37accesses a database (such as, for example, the database42) of geotagged imagery and/or maps, and displays the imagery and/or maps on the display20.

FIG. 3, depicts one embodiment of a dialog box130that may be displayed on the display20in response to activation of the geolocation routine37. The dialog box130may include a map or imagery display area132, navigation bar134, and a title bar136. The map or imagery display area132may, for example, default to displaying an initial view, such as the view of North America, depicted inFIG. 3. Alternatively, the map or imagery display area132may default to any other view, such as a view of a continent, country, or city specified by a user, or to a view of the Earth. The default view displayed in the map or imagery display area132may be a graphical map, a satellite image, or some combination of a map and a satellite image. The map or image display area132may also include a pan control138and a zoom control140. The pan control138allows the user to select one of four directional indictors142on the pan control138. Each directional indicator138pans the map in the indicated direction. The zoom control140allows the user to zoom in and/or out on the location currently at the center of the display area132. In some embodiments, a user may also manipulate the display area132with an input device by, for example, “dragging” the map with a mouse to pan, or “clicking” on the map to zoom in on a particular point.

For any particular image and/or map displayed in the display area132, each pixel may be mapped to a set of coordinates to indicate the location of the pixel. The data for given pixels may be stored in the database42or may be calculated with reference to one or more reference points upon accessing the image and/or map. For instance, the image depicted in the display area132ofFIG. 3may have associated with it in the database42metadata indicating the coordinates of one or more reference points (e.g., a center point, a corner, multiple corners) (not shown), the resolution of the image (e.g., 800 pixels×350 pixels), the change per pixel in each coordinate value, etc. In this manner, the workstation18and/or the server40may provide to the geolocation routine37the geolocation of each pixel. The coordinates may be selected from any coordinate system desirable for a mapping the available area (e.g., the surface of the Earth) and, in at least some embodiments, the set of coordinates comprises a latitude and a longitude. As the image and/or map is zoomed in (i.e., as the resolution of the displayed area is increased, for example, by manipulating the zoom control140), the accuracy and/or the precision of the set of coordinates associated with each pixel in the display area132may increase. At some minimum zoom level, the metadata for a particular image and/or map may include multiple sets of coordinates. For example, when the zoom control140is adjusted to “street level” (i.e., when individual streets are visible in the display area132), each pixel may be associated with a first set of coordinates (e.g., a latitude and a longitude) and a second set of coordinates (e.g., a nearest street address).

The navigation bar134on the dialog box130may include a search field144, into which a user may type a search term. For example, the search term could be an address, an intersection, a state, a country, a landmark name, a business name, a latitude and longitude, etc. After entering a search term into the search field144, the user may cause a search to be executed by, for instance, pressing the “Enter” key on a keyboard, or using a pointing device (e.g., a mouse) to “depress” a “Search” button146.FIG. 4depicts the dialog box130displaying an exemplary search result. The search field144has in it an address150. The display area132inFIG. 4depicts the area around the address150entered into the search field144. A building152having the same address as the address150entered into the search field144is marked with a star154.

Once the desired area is displayed in the display area132, a user may geolocate one or more points of the 3D model100in the 3D program space106. In some embodiments, the user may select in the dialog box a particular point, such as a corner156of the building152, and may associate the point with a point in the 3D program space106. For example, the user may associate the corner156with the origin120, with a point on the 3D model100, or with any point in the 3D program space106. By doing so, the set of coordinates of the selected point in the display132becomes associated with the selected point (e.g., the origin120) in the 3D program space106. In some embodiments, the user may instead activate a user interface control148(e.g., using an input device to select a button) and select a region of the display area132to import into the 3D program space106. For example, and with reference toFIG. 5, activation of the control148may cause the dialog box130to display, in the same dialog box130or in a separate dialog box (not shown) a selection control158. The selection control158may be, by way of example, a bounded area160adjustable by moving one or more of a plurality of vertices162, each of which vertices may have associated with it a set of coordinates. Alternatively, the selection control158may be a fixed-shape (e.g., square, rectangular, circular, etc.) selection window having an adjustable size or even having a fixed size. In any event, controls164and166may, respectively, allow the user to import into the 3D program space106the region indicated (e.g., by the bounded area160) or to cancel the selection and to return, for example, to the display depicted inFIG. 4.

Activation of the control164may import the bounded area160into the 3D program space106of the modeling application34, as illustrated inFIG. 6.FIG. 6depicts an application window170that may be displayed on the display20during execution of the modeling application34. The application window170depicts the 3D program space106with the X-, Y-, and Z-axes,122,124, and126, respectively, meeting at the origin120. The bounded area160is copied into the 3D program space106and may be displayed as a two-dimensional image (i.e., a plane). By default, the plane defined by the two-dimensional image of the bounded area160may be placed within the 3D program space106such that the plane is coincident with the plane defined by the X- and Y-axes122and124, and may optionally be placed such that a center (not shown) of the bounded area160is coincident with the origin120. In some embodiments, the user may be able to realign the origin120and/or the bounded area160to facilitate easier creation (or manipulation) of the 3D model100. InFIG. 6, for example, the bounded area160and/or the origin120are aligned such that the corner156of the building152is aligned with the origin120.

In some embodiments, every point within the bounded area160may have associated with it metadata indicating an altitude. The altitude may be relative to mean sea level, relative to the points next to it, relative to the center of the Earth, etc. Regardless of how the altitudes of various points within the bounded area are indicated, the bounded area160may be displayed selectively as either a 2D plane (as depicted inFIG. 6) or as 3D topography. Displaying the bounded area160as a 3D topography may allow the user to create the 3D model100with greater accuracy. In any event, the 3D model100may be constructed such that a base164of the 3D model100is coincident with the depiction of the building152and with the 2D planar depiction or 3D topography of the bounded area160. Alternatively, the already-constructed 3D model100could be aligned with the depiction of the building152and with the 2D planar depiction or 3D topography of the bounded area160.

Of course, with even a single point in the 3D program space106associated with a set of coordinates, every other point in the 3D program space106may be located referentially and assigned a corresponding set of coordinates. Thus, if the bounded area160is a square measuring 1000 feet per side, and the origin120is located at [X, Y, Z]=[150, 300, 0], one can determine that the vertices162of the bounded area160will be located at [−150, −300, 0], [850, −300, 0], [850, 700, 0], and [−150, 700, 0]. Likewise, every point within the 3D program space106may be determined relative to a point having known coordinates.

It should be understood that multiple coordinate systems may be in use within the system10. For instance, the set of coordinates associated with the bounded area160in the database42and/or with each pixel within the display area132, may comprise a latitude value and a longitude value (and possibly an altitude value), while the 3D program space106in the application34may associate each point therein with a set of X, Y, and Z coordinates. Nevertheless, so long as a single point within the 3D program space106is associated with a corresponding latitude and longitude, the corresponding latitude and longitude of every point within the 3D program space106may be calculated. Accordingly, the location of every point on any surface of the 3D model100may be specified by a first set of coordinates in the 3D program space106and by a second set of coordinates indicating a real-world position.

FIG. 7depicts a geolocated image174. The geolocated image174corresponds to the bounded area160and has at least one geolocated point. The geolocated point may be a point176corresponding to the origin120, a point178corresponding to a vertex of the geolocated image174, or any other point in the geolocated image174. From that geolocated point, any point in the 3D program space106may likewise be geolocated. In some embodiments, the geolocated image174represents a ground plane in the 3D program space106and corresponds to the plane defined by the X- and Y-axes122and124(i.e., the Z=0 plane in the 3D program space106). In some embodiments, the geolocated image174may, by default, be oriented in the 3D program space106such that the positive X-axis122is east of the origin120, the positive Y-axis124is north f the origin120, and the positive Z-axis126is above the ground (i.e., “up”). Additionally, in some embodiments, the geolocated image174may, by default, be placed in the 3D program space106such that the Z-axis126passes through the center of the geolocated image174. Optionally, a user may adjust the origin120and, specifically, may adjust the origin120to correspond to a corner of the 3D model100, as depicted inFIG. 7.

Once a geolocated, 3D model100is created in the 3D program space106of the modeling application34, the user may select (e.g., using an input device such as a mouse, a touch screen, a stylus, etc.) a desired surface172to photo-texture as depicted inFIG. 7. It is worth noting, at this point, that texturing a surface does not necessarily require the geolocation of the 3D program space. However, in instances where the 3D program space and the 3D model are geolocated, the photo texturing process may be automated to a greater extent, as described below. In any event, the application34(which may be executing on the processor24or on a processor of the server40) receives the selection of the surface to photo texture. The selected surface172may have vertices176,180,182, and184(if the selected surface172is a rectangle, for example) and a center point186, each of which may be specified by a set of coordinates in the 3D space106. Having selected the surface172, the user may activate a control causing the application34to execute the video photo texturing routine35and, in particular, may activate a control for causing the application34to execute a video photo texturing routine. For example, inFIG. 7, the user is depicted selecting a control175from a series of pull-down menu items173, as one of ordinary skill in the art would readily understand. Of course, the modeling application34could, additionally or alternatively, include a button control (not shown) that would cause execution of the video photo texturing routine35. In some embodiments, the modeling application34may also give the user the option to execute a manual version of the video photo texturing routine35or an automatic version of the video photo texturing routine35, either of which may acquire the photo texture from a photograph instead of a video.

Upon activating the control175, the application34may execute the video photo texturing routine35. In some embodiments, the photo texturing routine may be configured to texture the selected surface172using a frame of a video file specified by the user. For example, the video photo texturing routine35called by activation of the control175may allow the user to select a video file hosted by a web service (e.g., in a database such as the database42) or a video stored locally on the workstation18, for example in the non-volatile memory27. Accordingly, execution in the processor24of the video photo texturing routine35may cause the workstation18display20to display a dialog box350, such as that illustrated inFIG. 8. The dialog box350may operate, as generally understood, to allow the user to select a local file352stored in the non-volatile memory27, or to input the location (e.g., by specifying a uniform resource identifier (URI)354) of a video resource stored on a network (e.g., on the Internet). Of course, the video resource need not be a specific file or file type and, in fact, in some embodiments, the URI354may indicate a streaming video resource from which a user may select a particular frame for application to the selected surface172by the video photo texturing routine35.

As discussed above, in some embodiments, the video resource/file may include or otherwise be associated with metadata indicating one or both of GPS data and/or compass data.FIGS. 9A and 9B, for example, depict representations of two different video files (or video resources)360and370, respectively, that include GPS and/or compass metadata. For simplicity, the video files and/or video resources360and370will be described throughout the remainder of this description as video files, though it should be understood that a file, per se, is not required. Instead, as described above, the video resource could, for example, be a video stream, etc. The file360includes initial metadata362A associated with a position364A at the beginning of the video file360. The file360may additionally have metadata associated with each of a number of other positions within the video file360. For example, a video capture program having access to GPS and/or compass data may acquire current GPS and/or compass data at regular intervals while recording video imagery. In the video file360, for instance, metadata362B-J is associated with positions364B-J in the video. A plurality of markers or bookmarks associated with the file may indicate positions within the video for which metadata are available.

In some embodiments, the video photo texturing routine35may search metadata associated with the selected video file360to determine a section of the video file360likely to include a video frame including the selected surface172. For example, a frame of video at a point366in the video file360may include the best image of the selected surface172. The video photo texturing routine35may, where the selected surface172is geolocated within the 3D program space106, search for and/or select a bookmark having metadata indicating a position closest to the location of the selected surface172. Alternately or additionally, the video photo texturing routine35may search for and/or select a bookmark having metadata indicating that the camera was pointed in the direction of the selected surface172. In the video file360depicted inFIG. 9A, the video photo texturing routine35may, for example, determine that the metadata362D indicates a position closest to the selected surface172and/or a compass heading indicating that the camera was pointed at the selected surface172. Thus, the photo texturing routine360may display, by default, (e.g., in a viewer appropriate to the video file type) a first frame at or after the position364D corresponding to the metadata362D.

Of course, in some embodiments, a selected video file may include only a single set of metadata for the file (e.g., metadata associated with the beginning or end of the video file). For example,FIG. 9Bdepicts the video file370as having only metadata372associated with a starting point374of the video file370. Thus, a user would be left to find an appropriate frame376to use as an image for photo texturing.

FIG. 10illustrates an exemplary display380for viewing a video and selecting a frame to use to texture the selected surface172. For example, using the display380, a user may select a frame of a video that does not have any metadata, or may refine a default selection made by the video photo texturing routine35using metadata to select an appropriate bookmark. The display380may have a video display area381in which the video imagery is displayed. A video control bar382may include a number of controls383-387, for controlling the display380and, in particular, may have: a control383for playing and/or pausing the motion of the video imagery; a control384for displaying the previous frame; a control385for displaying the next frame; a control386for activating further volume controls, and controls387for adjusting the size of the display window or performing other viewing-related tasks. A position indication bar (also known as a “scrubber bar”)388may indicate the position of the currently displayed frame relative to the entirety of the video and, as generally understood, a position control389may allow a user to quickly move to a different point in the video. A select button390and a cancel button391may, respectively, allow the user to select the currently displayed frame to use as an image to use for photo texturing the selected surface172, or to cancel the photo texturing routine.

Once the user has selected a video frame for the video photo texturing routine35to use to photo texture the selected surface172, the video photo texturing routine35may display a dialog box392(seeFIG. 11) for selecting an area of the selected frame to apply as a photo texture to the selected surface172. The dialog box392may include a display area393that displays the selected video frame. Within the frame, a number of selectable controls394may allow the user to select an area of the selected frame that corresponds to the selected surface172. For example, in the dialog box392illustrated inFIG. 11, four controls394A-394D allow the user to select a tetragonal area397, which may be indicated, for example, by shading. A control395may allow the user to select the area bounded by the controls394, while a control396may allow the user to cancel the execution of the routine or to return to the frame selection dialog box380.

FIGS. 12 and 13are flow charts illustrating, respectively, a method200for commencing the process of photo texturing a surface (e.g., the surface172) of a 3D model (e.g., the 3D model100) from video, and a method220for geolocating the 3D program space106. Referring first toFIG. 12, as described above, the process of photo texturing the 3D model100from video imagery may commence, in some embodiments, by geolocating the 3D program space106in which the 3D model100is to be created (block202) and creating the 3D model100in the geolocated 3D program space106(block204). In some embodiments, the process of photo texturing the 3D model100from video imagery may instead commence by creating the 3D model100in the 3D program space106(block201), geolocating the 3D program space106(block202), and, if necessary, adjusting the 3D model100in the geolocated 3D program space106(block203). In still other embodiments, geolocation of the 3D program space106(and, therefore, the steps indicated at blocks202,203) is omitted. In any event, a surface (e.g., the surface172) to be photo textured may be selected (block206) and the video photo texturing routine35may be executed (block208).

The geolocation process (block202; method220) commences when the application34receives a geolocation request (block222) and, in response to receiving the request executes the geolocation routine37and displays the geolocation dialog box130(block224). As described above, the user may navigate to, or search for, an image (which may be an aerial image) of the desired location (i.e., the location of the real-world structure represented by the 3D model100) (block226). This may involve sending a search term from the routine37to the server40, and receiving in response to the search term a corresponding image from the server40, which image the server40may retrieve from the database42. The user, having located in the geolocation dialog box130the desired region, may manipulate a control in the geolocation routine37to indicate a desire to select a region (block228). In response to receiving the user input (block228), the geolocation routine37may display a selection control (e.g., the bounded area160and the plurality of vertices162) (block230) and the user may manipulate the selection control to specify a desired area. The geolocation routine37may next receive a selection of a region (e.g. the bounded area160) to copy into the 3D program space106of the application34(block232). This may be accomplished, for example, when the user, having adjusted the vertices162, indicates that the bounded area160is adjusted according to the user's desire by manipulating a control (e.g., clicking on a button) in the geolocation dialog box130(block234). Having received the selection of a region, the geolocation routine37may copy the selected region of the geolocated image into the 3D program space106of the modeling application34.

In embodiments, the geolocated image copied into the 3D program space106includes at least one geolocated point specified by a first coordinate system such as latitude and longitude (and, in some embodiments, altitude). Of course, while the geolocated point(s) are described herein as designated by latitude and longitude, the geolocated point(s) need not be specified by a latitude and longitude, but could instead be specified in any available coordinate system.

To automatically apply a photo texture to the selected surface172from video, the application34and/or the video photo texturing routine35may, perhaps cooperatively: (1) find one or more videos likely to include one or more frames capturing an image of the selected surface of the real-world structure modeled by the 3D model100and/or receive a selection of a video from which to select a frame that includes an image of the selected surface; (2) select from the selected frame a portion of the image corresponding to the selected surface; (4) map the selected portion to the selected surface; and (5) apply the selected portion to the selected surface according to the mapping. Accomplishing these tasks may require that the application34and/or the video photo texturing routine35transform one or more sets of coordinates through several coordinate spaces. That is, the application34and/or the video photo texturing routine35may operate or cooperate to change a point or a set of points from one coordinate system to another coordinate system.

Turning now toFIG. 14, a flow chart depicts an exemplary method250, executed by the video photo texturing routine35, for applying a photo texture to a selected surface (e.g., the selected surface172) from video.

In some embodiments in which the 3D program space100has been geolocated or in which location data for the selected surface172has been otherwise provided, the video photo texturing routine35may determine or select a likely location190of a video camera viewing the selected surface (block251). To determine and select a likely location of a video camera viewing the selected surface172(block251), the video photo texturing routine35may execute a method280depicted inFIG. 15. In some embodiments, the first step in calculating the likely viewing point is calculating, determining, or retrieving a set of coordinates corresponding to the center point186(seeFIG. 7) of the selected surface172(block284). In various embodiments, the set of coordinates corresponding to the center point186of the selected surface172may be calculated or determined by the modeling application34and stored for later retrieval and use by the video photo texturing routine35. However, in some embodiments, the set of coordinates corresponding to the center point186of the selected surface172may be calculated or otherwise determined by the video photo texturing routine35, and may include first determining for each of the vertices176,180,182, and184a set of coordinates corresponding to the location in the 3D space106of the vertex (block283). Further, in some embodiments, the selected surface172may be represented by a set of vertices (e.g., a set of four vertices) even if the selected surface172is not rectangular. For example, the selected surface172could be a circular surface, specified by four vertices defining a square circumscribing the circular surface.

Referring now toFIGS. 15 and 16, to determine a likely location of a video camera viewing the selected surface172, the video photo texturing routine35may next translate the center point186a predefined distance (e.g. 15 meters) along the normal188of the selected surface172(block286). The video photo texturing routine35may then project this translated point190onto the ground plane104(block288) to determine a position192from which an image would likely be captured. A set of coordinates in the 3D program space106corresponds to the position192. The video photo texturing routine35may then transform this set of coordinates from the 3D program space106to a set of coordinates used to geotag video imagery (e.g., latitude, longitude, and altitude) (block290).

Turning again toFIG. 14, using that information, the video photo texturing routine35may, in some embodiments, initiate a search for geotagged videos located near the determined likely location190(block252). Of course, the video photo texturing routine35may initiate a search of web hosted videos (e.g., videos on YouTube.com), in some embodiments, may initiate a search of locally stored videos (e.g., stored on the non-volatile memory27, in some embodiments, and/or may initiate a search of both web hosted videos and locally stored videos. Thereafter, the video photo texturing routine35may select from the search results the best candidate video (block253) or, alternatively, may display a list of search results (block254).

In some embodiments, however, instead of searching for videos near the determined likely location190, the video photo texturing routine35may display the video selection dialog350(seeFIG. 8) (block255).

Regardless of whether the video photo texturing routine35searches for videos according to the determined likely location190or displays the video selection dialog350, the video photo texturing routine35next receives a selection of a video (block256).

Once the video photo texturing routine35has received a selection of a video (block256) from which a frame will be selected to use to texture the selected surface172, it is necessary for a frame of the selected video to be determined or selected to be used. In some embodiments, the video photo texturing routine35may search for the presence of bookmark metadata in the selected video (block258). If bookmark metadata is not present in the selected video file, or in embodiments in which bookmark metadata is not supported, the video photo texturing routine35may proceed to display the first frame of the selected video file (block261A) after receiving the selection of the video file (block256). Alternately, if bookmark metadata is found to be present in the selected video file, the video photo texturing routine35may determine the bookmark within the video file that indicating a position closest to the likely location190determined at block251(block260), and display the first frame after the determined bookmark (block261B).

The video photo texturing routine35next receives a selection of video frame from which an area will be selected to apply as a photo texture to the selected surface172(block262). Next, the video photo texturing routine35receives a selection of the area of the selected video frame (block263). Having received the selection of the area of the selected video frame, the video photo texturing routine35captures the selected area (block264) and uses standard UV mapping techniques to apply the captured area as a texture to the selected surface (block266).

In some embodiments, the 3D modeling application34may call (i.e., instantiate) the video photo texturing routine35multiple times, successively or concurrently, to photo texture multiple surfaces of the 3D model100. For example, prior to initiating the video photo texturing routine35(FIG. 12at block208), a user may select multiple surfaces of the 3D model100(FIG. 12at block206).

For example, the network16may include but is not limited to any combination of a LAN, a MAN, a WAN, a mobile, a wired or wireless network, a private network, or a virtual private network. Moreover, while only one workstation18is illustrated inFIG. 1to simplify and clarify the description, it is understood that any number of workstations18are supported and can be in communication with the server40.

Still further, the figures depict preferred embodiments of a photo texturing system for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.