A three-dimensional asset (3D) reconstruction technique for generating a 3D asset representing an object from images of the object. The images are captured from different viewpoints in a darkroom using one or more light sources having known locations. The system estimates camera poses for each of the captured images and then constructs a 3D surface mesh made up of surfaces using the captured images and their respective estimated camera poses. Texture properties for each of the surfaces of the 3D surface mesh are then refined to generate the 3D asset.

TECHNICAL FIELD

Examples set forth in the present disclosure relate to generating a three-dimensional (3D) asset. More particularly, but not by way of limitation, the present disclosure describes an asset reconstruction technique that generates virtual representations (3D assets) from physical objects.

BACKGROUND

Augmented reality (AR) shopping and try-on allow brands to enhance user experience by bringing an asset directly to users and allowing them to seamlessly interact with the asset. According to consumer tests, interacting with 3D assets brings more conversion and engagement compared to conventional catalog-based shopping. Currently on the market, there are 3D asset generation engines that allow users to render and try-on assets fast even on mobile and edge devices (e.g., LensCore by Snap, Filament by Google, etc.). One of the remaining issues is that brands typically do not produce 3D assets as they release new goods. This is because creating such 3D assets requires time consuming manual work by 3D artists, which can take forty (40) or more hours per 3D asset reconstruction and many iterations with the brand owner. Furthermore, conventional automated solutions cannot achieve production quality without requiring time-consuming post-processing and refinement.

DETAILED DESCRIPTION

A 3D asset reconstruction technique for generating a 3D asset representing a physical object from images of the object. The images are captured from different viewpoints in a darkroom using one or more light sources having known locations. The technique estimates camera poses for each of the captured images and then constructs a 3D surface mesh made up of surfaces using the captured images and their respective estimated camera poses. Texture properties (e.g., one or more of reflectance, color, or roughness) for each of the surfaces of the 3D surface mesh are then refined to generate the 3D asset.

The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products illustrative of examples set forth in the disclosure. Numerous details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its relevant teachings. Those skilled in the relevant art, however, may understand how to apply the relevant teachings without such details. Aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described because the relevant teachings can be applied or practiced in a variety of ways. The terminology and nomenclature used herein is for the purpose of describing particular aspects only and is not intended to be limiting. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

The term “connect,” “connected,” “couple,” and “coupled” as used herein refers to any logical, optical, physical, or electrical connection, including a link or the like by which the electrical or magnetic signals produced or supplied by one system element are imparted to another coupled or connected system element. Unless described otherwise, coupled, or connected elements or devices are not necessarily directly connected to one another and may be separated by intermediate components, elements, or communication media, one or more of which may modify, manipulate, or carry the electrical signals. The term “on” means directly supported by an element or indirectly supported by the element through another element integrated into or supported by the element.

Reference now is made in detail to the examples illustrated in the accompanying drawings and discussed below. An example 3D asset reconstruction system and method will be described with respect toFIGS.1,2A,2B,3,4B, and5.

FIG.1depicts an example 3D asset reconstruction system100. The system100includes a computing system102configured to obtain images of an object104in a darkroom106with a camera system108. The camera system108includes one or more cameras that capture images (raw images) of the object104from different viewpoints (e.g., a few hundred viewpoints). The camera system108may include one or more cameras that is/are movable to different locations within the darkroom106to capture different viewpoints, multiple cameras that are statically positioned within the darkroom106to capture different viewpoints, or a combination thereof. Each camera may be a conventional camera capable of capturing high resolution digital images. The computing system102can include any of the components of the machine500(FIG.5), such as one or more processors.

A light source110illuminates the object and has a known location relative to each camera (or camera location) of the camera system108. In one example, a respective light source110is coupled to each camera of the camera system108, which facilitates determining the location of the light source110with respect to a captured image as the light source will have a known relationship to the camera. As used herein, the term darkroom refers to a room where substantially all of the light present in the room originates from a known source location(s).

FIGS.2A and2Bdepict an example pipeline200for 3D asset reconstruction, withFIG.2Aincluding textual blocks corresponding to functions performed during the reconstruction andFIG.2Bincluding images corresponding to those textual blocks that visually depict the functionality. Although the pipeline200is described with reference to the components of the 3D asset reconstruction system100, it will be apparent to one of skill in the art how to implement the pipeline using other systems from the description herein.

The camera system108captures raw photos202of the object104. The camera system captures a plurality of raw photos202(e.g., a few hundred) from different viewpoints surrounding the object104. Camera poses204are determined for each of the raw photos202. Camera poses may be determined using conventional structure from motion (SfM) algorithms. The computing system102then generates a surface mesh206for the object104using the raw photos202and determined camera poses204. The surface mesh206comprises a plurality of surfaces (which are typically triangular) and may be generated using an SfM algorithms (e.g., the SfM used to determine camera poses or another SfM). After generating the initial surface mesh206, the computing system102applies a differential renderer210to the surface mesh206, camera poses204, and raw photos202. The differential renderer210may be initialized with, for example, predefined texture values (e.g., uniform texture values208where all texture properties are set to the same value such as “0” or “1” or non-uniform texture values based on likely materials), pseudo-random texture values, texture values including noise, or a combination thereof that are then refined by the differential renderer210. In one example, the differential renderer210applies a conventional gradient descent algorithm to reduce the difference between the textures of the raw photos202and interim texture properties (starting from the initialized texture properties) associated with surfaces making up the surface mesh to produce learned texture properties212for respective surfaces of the surface mesh206. In one example, the learned texture properties212are stored as metadata associated with the respective surfaces of the surface mesh206to produce the 3D asset.

FIGS.3A-3Cdepict flowcharts300/320/340of example steps for generating a 3D reconstruction asset for an object. The steps are described with reference to the system100and pipeline200, however, implementation using other systems/pipelines will be understood by one of skill in the art from the description herein. Additionally, it will be understood that one or more steps depicted in the flowcharts may be performed sequentially as shown, out of order, at least substantially simultaneously, or may be omitted depending on the implementation.

Flowchart300depicts steps for reconstructing a three-dimensional (3D) asset representing an object. At block302, the camera system108captures images202of the object104(e.g., a shoe) from different viewpoints (e.g.,400viewpoints). The camera system108may store the images202(along with metadata) in a memory (e.g., memory504;FIG.5). The computing system102may obtain the images202directly from the camera system or via the memory. In an example, the images202are captured in a darkroom106. Capturing images202in a darkroom106where substantially all the light present for illuminating the object104originates from a known source location(s) enhances the ability of the differential renderer210in determining surface texture properties for the surfaces in the surface mesh206.

At block304, the computing system102estimates camera poses204for each of the captured images202. In an example, the computing system102estimates camera poses204by applying an SfM algorithm to the images202. The SfM algorithm matches each image to adjacent images in an iterative manner to estimate the camera poses204for each captured image202. For example, the SfM algorithm may extract features in the particular image, match the extracted features to extracted features in at least one other image to obtain image correspondence (e.g., a set of points in one image that can be identified as the same points in another image), and estimate the camera pose from the matches and the associated geometric relationships of a combination of features in one image to at least one other image including those features. Images including the same (matching) set of features captured from different viewpoints will have different geometric relationships among that set of features in each image due to their respective viewpoints. The SfM algorithm is able to use these geometric relationships to precisely determine the camera pose for each image by comparing the geometric relationship of a set of features in an image to the geometric relationship of the set of features in at least one other image. The camera pose204for each image may be stored in the metadata of that image.

At block306, the computing system102constructs a 3D surface mesh206. The 3D surface mesh206includes surfaces (e.g., triangular surfaces). The computing system102constructs the 3D mesh206using the captured images202and their respective estimated camera poses204. In an example, the computing system102constructs the 3D surface mesh206by applying a second SfM algorithm to the images202and their respective estimated camera poses204. The SfM algorithm matches each image to adjacent images in an iterative manner to construct the 3D surface mesh206using the captured images202and their respective camera poses204.

In an example, the computing system102may additionally receive manual input from a user during or after block306. For example, if upon visual inspection it is apparent to a user that the surface mesh206is not properly modeling the object104, manual input supplied by the user may be received by the computing system102for adjusting vertices of one or more surfaces (e.g., triangles) to model the object104more accurately. The computing system may then correct the 3D surface mesh206responsive to the manual input.

At block308, the computing system102refines texture properties of the plurality of surfaces of the 3D surface mesh206to generate the 3D asset including learned texture properties212for the surfaces of the 3D surface mesh206. In an example, each texture property is stored as a separate parameter for each surface of the 3D surface mesh206.

In an example, the differential renderer210is used to reconstruct texture properties from the captured images202to refine the texture properties. The differential renderer210enhances scene parameters (e.g., mesh vertices or textures) by minimizing differences between artificially rendered and real-world images. In an example, the difference is modeled by Mean Squared Error (MSE) between real and generated images, which is minimized via gradient descent. This is an iterative process, where the system (1) generates an image/scene from a known camera position, (2) computes MSE between the generated image/scene and a corresponding captured image, (3) computes gradients of MSE by scene parameters, and (4) updates scene parameters (param) in a way that can be roughly described as: param=param−alpha*gradient. Iterations are performed until the loss is minimized and the rendered images start looking the same to the captured images.

Flowchart320depicts example steps for a texture determination method for use in the asset reconstruction and rendering method ofFIG.3A. At block322, the computing system102models a scene based on captured images202. In an example, the computing system102models a scene including an object104using adjacent captured images202to produce a 3D surface mesh206. At block324, the computing system102initializes texture parameters for each surface of the 3D surface mesh206. In an example, the computing system102initializes the texture parameters by setting the parameter(s) of every surface to predefined values, pseudo-random values, texture values including noise, or a combination thereof. At block326, the computing system102refines the texture parameters, e.g., using the differential renderer210.

Flowchart340depicts example steps for a texture refinement method for use in the texture determination method ofFIG.3B. At block342, the computing system102generates a scene (e.g., an image of one or more surfaces) including an object using the captured images202. At block344, the computing system102generates an error value for the texture parameters. In an example, the computing system102generates texture parameter error values by (1) comparing each parameter of each pixel of a scene/surface(s) to the same parameter in a portion of a region of the captured image202corresponding to that pixel/surface and (2) computing a difference therebetween (e.g., a difference for each parameter or a difference between an average of multiple parameters). For example, the computing system102may compute the Mean Squared Error (MSE) for the parameters between the pixels of a generated image and a corresponding area of the captured image(s). At block346, the computing system102computes a gradient value representing the generated error values (i.e., gradients of the MSE values) using scene parameters. At block348, the computing system102updates the texture properties for the surface, i.e., to reduce the gradient value and, at block350, determines the difference between the error/gradient values using the updated parameters and prior to updating the error/gradient values.

At block352, the computing system102checks if the current iteration of the refinement has improved the surface mesh206with learned texture properties212. If the texture properties have not resulted in an improvement (i.e., a determined difference between a texture property/value (or an average of multiple texture properties) in an area of the captured image202and a corresponding surface of the 3D surface mesh206is greater than the previous iteration; or if a value representing the determined difference is at or below a threshold value), the refinement ends at block354. Alternatively, if the texture refinement has resulted in an improvement (i.e., the determined difference is less than the previous iteration; or if a value representing the determined difference is above the threshold value), another iteration is performed starting at block342.

Something about determining that the difference hasn't improved is determined by comparing a determined difference to a threshold difference and maybe how the difference is determined as expressed as a value.

Referring back toFIG.3A, at block310, the computing system102renders the 3D asset including the learned texture properties212. The 3D asset may be stored in a memory of a computing system. In an example, the 3D asset is rendered by retrieving the 3D asset from memory, providing a 3D rendering algorithm with the 3D asset and parameters affecting the appearance of textures (e.g., light intensity, direction of light, color/hue of light). In accordance with this example, the computing system102, executing instructions for implementing the 3D rendering algorithm, applies the parameters to the 3D asset and generates instructions for presenting an image of the 3D asset (as modified based on the parameters) to a user, e.g., on a display visible by the user.

FIG.4Adepicts a conventional 3D reconstruction asset400produced from a shoe andFIG.4Bdepicts a 3D reconstruction asset410produced from the same shoe in accordance with the new techniques described herein. The 3D reconstruction asset410includes greater and more realistic detail with respect to features of the shoe, e.g., the laces412of the 3D reconstruction asset410have greater detail than the laces402of the conventional 3D reconstruction asset400. Additionally, the 3D reconstruction asset410includes details (such as stitching414) not present in the conventional 3D reconstruction asset.

Techniques described herein may be used with one or more of the computing systems described herein or with one or more other systems. For example, the various procedures described herein may be implemented with hardware or software, or a combination of both. For example, at least one of the processor, memory, storage, output device(s), input device(s), or communication connections discussed below can each be at least a portion of one or more hardware components. Dedicated hardware logic components can be constructed to implement at least a portion of one or more of the techniques described herein. For example, and without limitation, such hardware logic components may include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc. Applications that may include the apparatus and systems of various aspects can broadly include a variety of electronic and computing systems. Techniques may be implemented using two or more specific interconnected hardware modules or devices with related control and data signals that can be communicated between and through the modules, or as portions of an application-specific integrated circuit. Additionally, the techniques described herein may be implemented by software programs executable by a computing system. As an example, implementations can include distributed processing, component/object distributed processing, and parallel processing. Moreover, virtual computing system processing can be constructed to implement one or more of the techniques or functionalities, as described herein.

FIG.5illustrates an example configuration of a machine500including components that may be incorporated into the computing system102adapted to manage the 3D asset reconstruction.

In particular,FIG.5illustrates a block diagram of an example of a machine500upon which one or more configurations may be implemented. In alternative configurations, the machine500may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine500may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine500may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. In sample configurations, the machine500may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a smart phone, a web appliance, a server, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. For example, machine500may serve as a workstation, a front-end server, or a back-end server of a communication system. Machine500may implement the methods described herein by running the software used to implement the features described herein. Further, while only a single machine500is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

Machine (e.g., computing system)500may include a hardware processor502(e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory504and a static memory506, some or all of which may communicate with each other via an interlink (e.g., bus)508. The machine500may further include a display unit510(shown as a video display), an alphanumeric input device512(e.g., a keyboard), and a user interface (UI) navigation device514(e.g., a mouse). In an example, the display unit510, input device512and UI navigation device514may be a touch screen display. The machine500may additionally include a mass storage device (e.g., drive unit)516, a signal generation device518(e.g., a speaker), a network interface device520, and one or more sensors522. Example sensors522include one or more of a global positioning system (GPS) sensor, compass, accelerometer, temperature, light, camera, video camera, sensors of physical states or positions, pressure sensors, fingerprint sensors, retina scanners, or other sensors. The machine500may include an output controller524, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

The mass storage device516may include a machine readable medium526on which is stored one or more sets of data structures or instructions528(e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions528may also reside, completely or at least partially, within the main memory504, within static memory506, or within the hardware processor502during execution thereof by the machine500. In an example, one or any combination of the hardware processor502, the main memory504, the static memory506, or the mass storage device516may constitute machine readable media.

The features and flowcharts described herein can be embodied in one or more methods as method steps or in one or more applications as described previously. According to some configurations, an “application” or “applications” are program(s) that execute functions defined in the programs. Various programming languages can be employed to generate one or more of the applications, structured in a variety of manners, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application developed using the ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of the particular platform) may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or another mobile operating system. In this example, the third-party application can invoke API calls provided by the operating system to facilitate the functionality described herein. The applications can be stored in any type of computer readable medium or computer storage device and be executed by one or more general purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in specialized computer hardware or an application specific integrated circuit (ASIC), field programmable gate array (FPGA) or a complex programmable logic device (CPLD).

Program aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of at least one of executable code or associated data that is carried on or embodied in a type of machine-readable medium. For example, programming code could include code for the touch sensor or other functions described herein. “Storage” type media include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from the server system or host computer of a service provider into the computer platforms of the smartwatch or other portable electronic devices. Thus, another type of media that may bear the programming, media content or metadata files includes optical, electrical, and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links, or the like, also may be considered as media bearing the software. As used herein, unless restricted to “non-transitory,” “tangible,” or “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions or data to a processor for execution.

Hence, a machine-readable medium may take many forms of tangible storage medium. Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, such as may be used to implement the client device, media gateway, transcoder, etc. shown in the drawings. Volatile storage media include dynamic memory, such as main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computing system. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read at least one of programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

While the foregoing has described what are considered to be the best mode and other examples, it is understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples, and that they may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all modifications and variations that fall within the true scope of the present concepts.