Patent ID: 12254570

DETAILED DESCRIPTION

One or more embodiments described herein include an object representation system that generates accurate three-dimensional representations of real-world objects utilizing mesh-guided thin volumes. For instance, in one or more embodiments, the object representation system constructs a mesh for an object portrayed in a set of multi-view images using multi-view stereo. The object representation system further builds a thin volume around the mesh. For example, in some implementations, the object representation system utilizes a learnable feature map to identify neural textures for points near the mesh and utilizes the neural textures to determine the volume densities and radiances of those points. Further, in some embodiments, the object representation system registers the volume of a generated representation with the mesh so that editing or animation of the representation is performed via modification of the mesh.

As mentioned in one or more embodiments, the object representation system generates a mesh for a digital object. In particular, in some embodiments, the object representation system generates the mesh from a set of digital images that portray the digital object (e.g., in different views). For instance, in some cases, the object representation system utilizes a multi-view stereo model to generate the mesh based on the set of digital images.

As further mentioned, in one or more embodiments, the object representation system utilizes the mesh to generate a three-dimensional hybrid mesh-volumetric representation for the digital object. In particular, in some embodiments, the object representation system builds a thin volume around the mesh to create the three-dimensional hybrid mesh-volumetric representation.

For example, in some implementations, the object representation system determines a set of sample points for the thin volume. To illustrate, in some instances, the object representation system generates a ray from a source point to the mesh and samples a plurality of points along the ray. The object representation system further selects one or more of the sample points for the thin volume. For instance, in some cases, the object representation system selects one or more sample points that fall within a threshold distance of the mesh.

Additionally, in some embodiments, the object representation system determines neural textures for the thin volume. In particular, the object representation system determines neural textures for the set of sample points. For instance, in some cases, the object representation system generates (e.g., using the multi-view stereo model) a UV map for the digital object that corresponds to a learned feature map that includes feature vectors representing neural textures. Accordingly, the object representation system utilizes the UV map to identify and extract feature vectors from the learned feature map for the set of sample points. To illustrate, in at least one implementation, the object representation system extracts a feature vector for a sample point by identifying a point on the mesh that corresponds to the sample point, determining coordinates for the point within the UV map, and extracting a feature vector from the feature map that corresponds to the coordinates for that point.

In one or more embodiments, the object representation system generates the three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the neural textures. In some cases, the object representation system further generates the three-dimensional hybrid mesh-volumetric representation utilizing the offset of the sample points and the direction of the one or more rays used to determine the one or more sample points. In some implementations, the object representation system generates the three-dimensional hybrid mesh-volumetric representation utilizing a neural network, such as a multi-layer perceptron. In some instances, the object representation system further renders the three-dimensional hybrid mesh-volumetric representation for display on a computing device (or causes the computing device to render the three-dimensional hybrid mesh-volumetric representation for display).

In some embodiments, the object representation system provides various additional features and applications related to digital objects. For instance, in some cases, the object representation system edits the three-dimensional hybrid mesh-volumetric representation of a digital object by editing the underlying mesh created for the digital object. Likewise, in some embodiments, the three-dimensional hybrid mesh-volumetric representation animates the three-dimensional hybrid mesh-volumetric representation by animating (e.g., deforming) the underlying mesh. In some cases, the object representation system further supports mipmaps. For example, in some instances, the object representation system generates a mipmap using a pyramid of neural textures associated with the digital object.

As mentioned above, conventional object representation systems suffer from several technological shortcomings that result in inaccurate, inefficient, and inflexible operation. For instance, many conventional systems fail to generate accurate representations for objects having a complex appearance. To illustrate, some existing systems utilize meshes to model real-world objects; however, meshes often fail to capture the complex details of many objects, such as objects with thin structures (e.g., fur or hair) or objects with heavy self-occlusion. Further, conventional systems typically fail to effectively integrate meshes into a learning pipeline, hindering their ability to improve the captured detail.

Further, conventional object representation systems often suffer from inefficiencies. For example, some conventional systems generate volumetric representations of objects using a computationally expensive process that involves over sampling empty regions around an object. Thus, many conventional systems utilize a significant amount of computing resources, such as memory and processing power, to generate a representation of an object as well as to train the models implemented in generating the representation.

In addition to inaccuracy and inefficiency problems, conventional object representation systems can also operate inflexibly. In particular, volumetric representations of digital objects are inherently difficult to edit, animate, or otherwise manipulate. Accordingly, conventional systems relying on volumetric representations typically fail to provide flexible processes for editing or animating their generated representations.

The object representation system provides several advantages over conventional systems. For example, the object representation system improves the accuracy of implementing computing devices when compared to conventional systems. In particular, the object representation system improves the accuracy with which the implementing computing devices generate representations for complex digital objects. For example, by building a thin volume around a mesh generated for a digital object, the object representation system can mitigate errors present in a mesh to improve the resulting representation of complex features, such as thin structures or portions associated with self-occlusion. Further, by utilizing a neural network in building the thin volume (e.g., generating the three-dimensional hybrid mesh-volumetric representation) based on the mesh, the object representation system integrates the mesh into a learning pipeline that can improve how the details of a digital object are represented.

Additionally, the object representation system can improve the efficiency of implementing computing devices when compared to conventional systems. For instance, by generating the thin volume using sample points that are near the mesh, the object representation system reduces the number of samples typically used by conventional systems in generating a volumetric representation. Accordingly, the object representation system utilizes fewer computing resources to generate volumetric representations. Further, as fewer samples are used, the object representation system can train the models implemented to generate the volumetric representation with better efficiency.

Further, the object representation system can improve the flexibility of implementing computing device. In particular, the object representation system provides flexibility in editing and/or animating volumetric representations of digital objects. Indeed, by modifying the three-dimensional hybrid mesh-volumetric representation of a digital object via the underlying mesh, the object representation system provides more flexible processes to implement edits and/or animations.

As illustrated by the foregoing discussion, the present disclosure utilizes a variety of terms to describe features and benefits of the object representation system. Additional detail is now provided regarding the meaning of these terms. For example, as used herein, the term “digital object” refers to a digital representation of an item or environment (e.g., person, place, or thing). In particular, in some embodiments, a digital object refers to a portrayal of a real-world object in a digital image. To illustrate, in some embodiments, a digital object includes, but is not limited to, a digital portrayal of a person, an animal, a building, a plant, a vehicle, a chair, or a handheld item.

Additionally, as used herein, the term “thin volume” refers to digital data that is utilized to represent volumetric detail of a digital object. In particular, in some embodiments, a thin volume refers to digital data that represents features of a digital object around a mesh for digital object. For instance, in some implementations, a thin volume includes digital data representing features within a threshold distance of the mesh. In one or more embodiments, a thin volume includes various levels of detail. For example, in some cases, a thin volume includes sample points within a threshold distance of the mesh. In some embodiments, a thin volume includes neural textures associated with such sample points. Further, in some implementations, a thin volume includes color values and/or volumetric densities associated with such sample points.

Further, as used herein, the term “three-dimensional hybrid mesh-volumetric representation” refers to a digital three-dimensional representation of a digital object created from a mesh and a thin volume that correspond to the digital object. In particular, in some embodiments, a three-dimensional hybrid mesh-volumetric representation refers to a digital three-dimensional model of a digital object generated using features determined from a mesh and a thin volume associated with a digital object. To illustrate, in some implementations, a three-dimensional hybrid mesh-volumetric representation includes a thin volume built around a mesh, the thin volume having color values and/or volumetric densities. Indeed, as will be described in more detail below, in some implementations, a three-dimensional hybrid mesh-volumetric representation includes a thin volume that has been modified with color values and/or volumetric densities.

As used herein, the term “neural texture” refers to one or more learned features for a digital object. In particular, in some embodiments, a neural texture refers to one or more high-level features learned for a digital object. For instance, in some cases, a neural texture includes on more high-level features learned from a set of digital images that portray a digital object. In some implementations, neural textures are represented by feature vectors of a feature map. Relatedly, as used herein, the term “pyramid of neural textures” refers to a collection of various levels of neural textures. For example, in one or more embodiments, a pyramid of neural textures includes neural textures associated with different resolutions of features learned for a digital object.

Additionally, as used herein, the term “multi-view stereo model” includes a computer-implemented model or algorithm for generating a mesh for a digital object portrayed in a set of digital images. In particular, in some embodiments, a multi-view stereo model includes a computer-implemented model or algorithm that analyzes a set of digital images that portray a digital object in various views (e.g., perspectives, orientations, etc.) and generates a mesh for the digital object based on the analysis. In one or more embodiments, a multi-view stereo model includes a pipeline or framework of multiple computer-implemented models and/or algorithms. For example, in some embodiments, a multi-view stereo model includes one or more machine learning models. In some implementations, a multi-view stereo model further generates a UV map and/or a feature map for the digital object portrayed in the set of digital images.

As used herein, the term “machine learning model” refers to a computer representation that can be tuned (e.g., trained) based on inputs to approximate unknown functions. In particular, in some embodiments, a machine learning model refers to a model that utilizes algorithms to learn from, and make predictions on, known data by analyzing the known data to learn to generate outputs that reflect patterns and attributes of the known data. For instance, in some instances, a machine learning model includes, but is not limited to a neural network (e.g., a convolutional neural network, recurrent neural network or other deep learning network), a decision tree (e.g., a gradient boosted decision tree), association rule learning, inductive logic programming, support vector learning, Bayesian network, regression-based model (e.g., censored regression), principal component analysis, or a combination thereof.

Further, as used herein, the term “neural network” refers to a model of interconnected artificial neurons (e.g., organized in layers) that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. In some instances, a neural network includes one or more machine learning algorithms. Further, in some cases, a neural network includes an algorithm (or set of algorithms) that implements deep learning techniques that utilize a set of algorithms to model high-level abstractions in data. To illustrate, in some embodiments, a neural network includes a convolutional neural network, a recurrent neural network (e.g., a long short-term memory neural network), a generative adversarial neural network, a graph neural network, or a multi-layer perceptron. In some embodiments, a neural network includes a combination of neural networks or neural network components.

As used herein, the term “ray” refers to a straight line representation in three-dimensional space (e.g., a line having a source and direction). In particular, in some embodiments, a ray refers to a vector. For example, in some implementations, a ray includes vector that extends outward from a source point in a straight line. Rays can include vectors of different dimensions in various embodiments.

Additionally, as used herein, the term “color value” refers to a representation of color (e.g., a red, blue, and green (RGB) value or another color value such as a CMYK value). In particular, in some embodiments, a color value refers to a RGB value (or radiance value) associated with a portion of a three-dimensional hybrid mesh-volumetric representation. For example, in some cases, a color value includes a RGB value (or radiance value) associated with a sample point determined from a mesh for a thin volume built around the mesh. In some cases, a color value includes a view-dependent color value. In other words, in some implementations, a color value associated with a portion of a three-dimensional hybrid mesh-volumetric representation changes depending on how the sample point is viewed.

Further, as used herein, the term “volume density” refers to a representation of density. In particular, in some embodiments, a volume density refers to a density value associated with a portion of a three-dimensional hybrid mesh-volumetric representation. For example, in some cases, a volume density includes a density value associated with a sample point determined from a mesh for a thin volume built around the mesh.

Additional detail regarding the object representation system will now be provided with reference to the figures. For example,FIG.1illustrates a schematic diagram of an exemplary system environment (“environment”)100in which an object representation system106operates. As illustrated inFIG.1, the environment100includes a server(s)102, a network108, and client devices110a-110n.

Although the environment100ofFIG.1is depicted as having a particular number of components, the environment100is capable of having any number of additional or alternative components (e.g., any number of servers, client devices, or other components in communication with the object representation system106via the network108). Similarly, althoughFIG.1illustrates a particular arrangement of the server(s)102, the network108, and the client devices110a-110n, various additional arrangements are possible.

The server(s)102, the network108, and the client devices110a-110nare communicatively coupled with each other either directly or indirectly (e.g., through the network108discussed in greater detail below in relation toFIG.11). Moreover, the server(s)102and the client devices110a-110ninclude one of a variety of computing devices (including one or more computing devices as discussed in greater detail with relation toFIG.11).

As mentioned above, the environment100includes the server(s)102. In one or more embodiments, the server(s)102generates, stores, receives, and/or transmits data including neural networks, digital images, and three-dimensional hybrid mesh-volumetric representations. In one or more embodiments, the server(s)102comprises a data server. In some implementations, the server(s)102comprises a communication server or a web-hosting server.

In one or more embodiments, the three-dimensional modeling system104generates and/or edits digital three-dimensional models. For example, in some embodiments, the three-dimensional modeling system104generates a digital three-dimensional model (e.g., in response to user input for designing the three-dimensional model). In some cases, the three-dimensional modeling system104further provides tools for editing, deforming, or otherwise interacting with a generated digital three-dimensional model.

Additionally, the server(s)102includes the object representation system106. In one or more embodiments, via the server(s)102, the object representation system106generates three-dimensional hybrid mesh-volumetric representations for digital objects. For example, in one or more embodiments, the object representation system106, via the server(s)102, generates a mesh114for a digital object portrayed in a set of digital images. In some cases, via the server(s)102, the object representation system106further determines a thin volume116around the mesh114. In some embodiments, via the server(s)102, the object representation system106generates a three-dimensional hybrid mesh-volumetric representation118(e.g., a modified thin volume) for the digital object from the mesh114and the thin volume116. Example components of the object representation system106will be described below with regard toFIG.9.

In one or more embodiments, the client devices110a-110ninclude computing devices that can access, edit, modify, store, and/or provide, for display, digital images and/or three-dimensional hybrid mesh-volumetric representations. For example, the client devices110a-110ninclude smartphones, tablets, desktop computers, laptop computers, head-mounted-display devices, or other electronic devices. The client devices110a-110ninclude one or more applications (e.g., the client application112) that can access, edit, modify, store, and/or provide, for display, digital images and/or three-dimensional hybrid mesh-volumetric representations. For example, in some embodiments, the client application112includes a software application installed on the client devices110a-110n. In other cases, however, the client application112includes a web browser or other application that accesses a software application hosted on the server(s)102.

The object representation system106can be implemented in whole, or in part, by the individual elements of the environment100. Indeed, as shown inFIG.1the object representation system106can be implemented with regard to the server(s)102and/or at the client devices110a-110n. In particular embodiments, the object representation system106on the client devices110a-110ncomprises a web application, a native application installed on the client devices110a-110n(e.g., a mobile application, a desktop application, a plug-in application, etc.), or a cloud-based application where part of the functionality is performed by the server(s)102.

In additional or alternative embodiments, the object representation system106on the client devices110a-110nrepresents and/or provides the same or similar functionality as described herein in connection with the object representation system106on the server(s)102. In some implementations, the object representation system106on the server(s)102supports the object representation system106on the client devices110a-110n.

For example, in some embodiments, the server(s)102train one or more machine learning models described herein (e.g., a multi-view stereo model and/or a neural network). The object representation system106on the server(s)102provides the one or more trained machine learning models to the object representation system106on the client devices110a-110nfor implementation. Accordingly, although not illustrated, in one or more embodiments the client devices110a-110nutilize the one or more trained machine learning models to generate three-dimensional hybrid mesh-volumetric representations.

In some embodiments, the object representation system106includes a web hosting application that allows the client devices110a-110nto interact with content and services hosted on the server(s)102. To illustrate, in one or more implementations, the client devices110a-110naccesses a web page or computing application supported by the server(s)102. The client devices110a-110nprovides input to the server(s)102(e.g., a set of digital images portraying a digital object). In response, the object representation system106on the server(s)102utilizes the trained machine learning models to generate a three-dimensional hybrid mesh-volumetric representation for the digital object. The server(s)102then provides the three-dimensional hybrid mesh-volumetric representation to the client devices110a-110n.

In some embodiments, though not illustrated inFIG.1, the environment100has a different arrangement of components and/or has a different number or set of components altogether. For example, in certain embodiments, the client devices110a-110ncommunicate directly with the server(s)102, bypassing the network108. As another example, the environment100includes a third-party server comprising a content server and/or a data collection server.

As mentioned above, the object representation system106generates a three-dimensional hybrid mesh-volumetric representation for a digital object.FIG.2illustrates an overview diagram of the object representation system106generating a three-dimensional hybrid mesh-volumetric representation for a digital object in accordance with one or more embodiments.

As shown inFIG.2, the object representation system106generates a three-dimensional hybrid mesh-volumetric representation208for a digital object210portrayed in a set of digital images202. As illustrated, in one or more embodiments, the set of digital images202includes a multi-view collection of images. In other words, in some embodiments, each digital image from the set of digital images202portrays the digital object210with a different view (e.g., perspective, orientation, etc.) than the other digital images.

Indeed, in one or more embodiments, the object representation system106analyzes the set of digital images202and generates the three-dimensional hybrid mesh-volumetric representation208based on the analysis. For instance, as shown inFIG.2, the object representation system106generates the three-dimensional hybrid mesh-volumetric representation208from the set of digital images202using a multi-view stereo model204and a neural network206.

As further shown inFIG.2, the object representation system106generates the three-dimensional hybrid mesh-volumetric representation208utilizing a mesh212(shown as the outline or contour of a projection of the mesh212) and a thin volume214(shown as a volume surrounding the projected contour of the mesh212). In particular, as will be described in more detail below, in some implementations, the object representation system106generates the mesh212for the digital object210. Further, the object representation system106builds the thin volume214around the mesh212. For example, in some implementations, the object representation system106determines some initial details (e.g., sample points) for the thin volume214and further modifies the thin volume214to generate the three-dimensional hybrid mesh-volumetric representation208.

Further, as shown inFIG.2, the object representation system106provides the three-dimensional hybrid mesh-volumetric representation208for display on a client device216. In one or more embodiments, the object representation system106further renders the three-dimensional hybrid mesh-volumetric representation208for display on the client device216. To illustrate, in one or more embodiments, the object representation system106operating on the server(s)102renders the three-dimensional hybrid mesh-volumetric representation208and provides the rendering to the client device216for display. In some cases, the object representation system106transmits the three-dimensional hybrid mesh-volumetric representation208from the server(s)102to the client device216and causes the client device216to render the three-dimensional hybrid mesh-volumetric representation208. In some implementations, the object representation system106operates on the client device216to generate and/or render the three-dimensional hybrid mesh-volumetric representation208for display.

As mentioned, the object representation system106generates a mesh for a digital object.FIG.3illustrates a diagram for generating a mesh for a digital object in accordance with one or more embodiments.

As shown inFIG.3, the object representation system106generates a mesh302for a digital object304portrayed in a set of digital images306(e.g., multi-view images). In particular, as shown, the object representation system106generates the mesh302from the set of digital images306. In other words, in some implementations, the object representation system106analyzes the set of digital images306and generates the mesh302based on the analysis (e.g., based on the portrayal of the digital object304in the set of digital images306). Thus, in some implementations, the object representation system106utilizes the different views provided by the set of digital images306in constructing the mesh302.

As illustrated byFIG.3, the object representation system106constructs the mesh302from the set of digital images306using a multi-view stereo model308. In one or more embodiments, the object representation system106utilizes, as the multi-view stereo model308, the open source COLMAP model described by J. L. Schönberger et al.,Pixelwise View Selection for Unstructured Multi-view Stereo, Proceedings of the European Conference on Computer Vision, pp. 501-518, 2016, which is incorporated herein by reference in its entirety. It should be noted, however, that the object representation system106, however, can utilize one of various alternatives in other embodiments.

As further mentioned above, the object representation system106builds a thin volume around a mesh generated for a digital object. For example, in some instances, the object representation system106determines a set of sample points for the thin volume. In some embodiments, the object representation system106further determines feature vectors representing neural textures for the set of sample points and utilizes the feature vectors to determine color values and volume densities for the sample points.

FIG.4illustrates a diagram for determining a set of sample points for a thin volume around a mesh in accordance with one or more embodiments. In particular,FIG.4illustrates the object representation system106determining one or more sample points for the set of sample points by generating a ray402from a source point404to the mesh406(e.g., having a direction represented by (θ, ϕ)). As illustrated, the ray402intersects the mesh406at a location408on the mesh406. In some implementations, the ray402intersects the mesh406at a plurality of locations.

As shown inFIG.4, the object representation system106further samples a plurality of points410a-410falong the ray. In one or more embodiments, the object representation system106samples the plurality of points410a-410frandomly or semi-randomly along the ray402. For instance, in some cases, the object representation system106samples a pre-determined number of points at random or semi-random locations along the ray402. In some implementations, the object representation system106randomly or semi-randomly samples a first point on the ray402and then samples the other points at regular distances along the ray402on either side of the first sampled point. In some embodiments, the object representation system106samples the plurality of points410a-410fat regular intervals based on the location408at which the ray402intersects the mesh406.

Additionally, in one or more embodiments, the object representation system106selects one or more points from the plurality of points410a-410fto include in the set of sample points (e.g., to use as sample points) for the thin volume around the mesh406. For instance, in some embodiments, the object representation system106selects a point to include in the set of sample points based on a distance between the point and the mesh406. Indeed, in some implementations, the object representation system106determines the distance between each point from the plurality of points410a-410fand the mesh406. For example, in some cases, the object representation system106determines the distance between a point and a closest point on the mesh (e.g., a closest distance).

To illustrate, in one or more embodiments, the object representation system106determines the distance between point410fand the mesh406to include the distance represented by the dashed line412. Accordingly, the object representation system106identifies the closest point on the mesh406and determines the distance between the point410fand that point on the mesh406.

In one or more embodiments, the object representation system106determines to include a point within the set of sample points for the thin volume if the point is within a threshold distance of the mesh406. For example, in some embodiments, the object representation system106compares the determined distance between the point and the mesh406to a threshold distance. In some cases, upon determining that the point is within the threshold distance based on the comparison, the object representation system106includes the point as a sample point within the set of sample points. Likewise, in some embodiments, the object representation system106excludes the point from the set of sample points upon determining that the point is located outside the threshold distance based on the comparison. In one or more embodiments, the threshold distance is configurable. In other words, the object representation system106utilizes a parameter for the threshold distance and modifies the parameter based on one or more factors (e.g., user input or features of the digital object).

Thus, as shown inFIG.4, the object representation system106selects the points410b-410efor inclusion within the set of sample points. In other words, the object representation system106determines that the points410b-410eare within a threshold distance to the mesh406while the point410aand the point410fare located outside the threshold distance to the mesh406. Based on selecting the points410b-410efor inclusion within the set of sample points, the object representation system106determines a thin volume around the corresponding portion of the mesh406having a thickness ε.

In one or more embodiments, the object representation system106similarly generates a plurality of additional rays from the source point404(or other source points) to the mesh406, where the additional rays have different directions than the ray402. The object representation system106samples points along the rays, compares the distances between the points and the mesh406to a threshold distance, and selects the points that fall within the threshold distance. Accordingly, the object representation system106determines a set of sample points for the thin volume414around the mesh406.

In some cases, the object representation system106utilizes a pre-determined number of rays for determining the set of sample points. In some cases, the object representation system106generates rays using a set of pre-determined directions or at least using pre-determined direction intervals. For instance, in some embodiments, for each pixel in the images, the object representation system106generates a ray that corresponds to the pixel using extrinsic and intrinsic information of the source point404(e.g., the camera). In one or more embodiments, the number of rays and their directions or direction intervals are configurable. Indeed, in some cases, the object representation system106utilizes parameters for the number of rays and their directions or direction intervals and modifies the parameter based on one or more factors (e.g., user input or features of the digital object).

As further discussed above, the object representation system106utilizes the set of sample points determined using the rays to generate a three-dimensional hybrid mesh-volumetric representation for the digital object. In particular, in some embodiments, the object representation system106modifies the thin volume determined using the set of sample points to generate the three-dimensional hybrid mesh-volumetric representation.FIG.5illustrates a diagram for generating a three-dimensional hybrid mesh-volumetric representation using sample points around a mesh in accordance with one or more embodiments.

As shown inFIG.5, the object representation system106determines a set of sample points for a thin volume504around a mesh506generated for a digital object as discussed above with regard toFIG.4. In one or more embodiments, the object representation system106further identifies neural textures represented by feature vectors for the set of sample points. In one or more embodiments, the object representation system106modifies the thin volume504to include the neural textures of the feature vectors.

Indeed, as shown inFIG.5, the object representation system106utilizes a multi-view stereo model508(e.g., the multi-view stereo model used to generate the mesh506for the digital object) to generate a UV map510for the digital object. In one or more embodiments, the UV map510includes UV coordinates of the mesh506. Further, in some embodiments, the UV map510corresponds to a feature map that includes a plurality of feature vectors representing neural textures. In other words, in one or more embodiments, the UV map510corresponds to a feature map of learned neural textures (e.g., neural textures generated from the multi-view stereo model508having parameters learned during the training process). Accordingly, the neural textures are represented by feature vectors in UV space.

In one or more embodiments, the object representation system106determines, for each sample point from the set of sample points, a feature vector using the UV map510. In particular, in some embodiments, the object representation system106determines a feature vector for a corresponding point on the mesh506(the closest point on the mesh506). For example, the UV map510can include a two-dimensional representation where coordinates along the two-dimensional map represent locations on a mesh (similar to a two-dimensional map of a globe). The object representation system106determines feature vectors corresponding to the UV map510that correspond to a matching point on the mesh506.

For example, as illustrated byFIG.5, the object representation system106identifies a point512on the mesh506that corresponds to the sample point502. Additionally, as shown, the object representation system106extracts, for the sample point502, a feature vector514corresponding to the point512on the mesh506using the UV map510. For instance, in some cases, the object representation system106utilizes the UV map510to determine the UV coordinates of the point512on the mesh506and extracts the feature vector514from the feature map using the UV coordinates.

As shown inFIG.5, the object representation system106utilizes the feature vector514extracted for the sample point502to determine a color value524(represented by RGB) and a volume density526(represented by σ) for the sample point502. As further shown, the object representation system106also utilizes a direction518(represented by (θ, ϕ)) of the ray516of the sample point502and a distance520(represented by δ) between the sample point502and the mesh506(e.g., the closest distance) to determine the color value524and the volume density526. In one or more embodiments, the object representation system106combines (e.g., concatenates) the feature vector514, the direction518, and the distance520and determines the color value524and the volume density526using the combination. Further, in some implementations, the distance520corresponds to a signed distance.

As indicated inFIG.5, the object representation system106utilizes a neural network522to determine the color value524and the volume density526from the feature vector514, the direction518, and the distance520. In one or more embodiments, the object representation system106trains the neural network522to generate color values and volume densities. For instance, in some implementations, the object representation system106utilizes the neural network522to generate predicted color values and/or predicted volume densities for training sample points. The object representation system106further compares the predictions to corresponding ground truths (e.g., via a loss function) and modifies parameters of the neural network522based on the comparisons to minimize the error of the neural network522. For instance, in some cases, the object representation system106weightedly accumulates the colors of the sample points along a ray as the prediction for the ray color and compares that prediction to the ground truth for modifying the network parameters.

In one or more embodiments, the object representation system106similarly determines a color value and volume density for each sample point from the set of sample points. Indeed, in some cases, the object representation system106determines a point on the mesh506that corresponds to the sample point, determines UV coordinates for the point on the mesh506using the UV map510, extracts a feature vector for that point on the mesh506from the feature map using the UV coordinates, and uses the neural network522to determine the color value and volume density based on the feature vector, the distance for the sample point, and the direction of the corresponding ray for that sample point. Accordingly, even where some sample points are similar in some values (e.g., direction and feature vector), the object representation system106can identify a different combination of color value and volume density for those sample points.

Thus, in one or more embodiments, the object representation system106generates a three-dimensional hybrid mesh-volumetric representation528for the digital image that includes a plurality of color values and volume densities corresponding to the sample points. In one or more embodiments, the object representation system106further modifies the thin volume504around the mesh506to generate the three-dimensional hybrid mesh-volumetric representation528. Indeed, in some implementations, the object representation system106generates the three-dimensional hybrid mesh-volumetric representation528by progressively building and modifying the thin volume504around the mesh506(e.g., using sample points, then neural textures, then color values and volume densities).

As further shown inFIG.5, the object representation system106renders the three-dimensional hybrid mesh-volumetric representation528for display on a client device530(or causes the client device530to render the three-dimensional hybrid mesh-volumetric representation528for display). For example, in one or more embodiments, the object representation system106utilizes raymarching to render the three-dimensional hybrid mesh-volumetric representation528.

Thus, in one or more embodiments, the object representation system106generates a representation for a digital object by building a thin volume around a mesh. In particular, the object representation system106utilizes the mesh to build the thin volume for the final representation. Accordingly, in some embodiments, the algorithm and acts described with reference toFIGS.4-5comprise the corresponding structure for performing a step for generating, from a set of digital images, a three-dimensional hybrid mesh-volumetric representation for a digital object.

By generating a three-dimensional hybrid mesh-volumetric representation for a digital object, the object representation system106offers improved accuracy when compared to conventional systems. In particular, the object representation system106generates more accurate representations of digital objects, particularly when compared to those that rely solely on mesh representations. Indeed, by building a thin volume around a mesh generated for a digital object, the object representation system106mitigates potential errors present within the mesh. Accordingly, the resulting thin volume provides a more accurate representation of the digital object than the underlying mesh.

Further, the object representation system106operates with improved efficiency when compared to conventional systems. In particular, by building the thin volume utilizing sample points within a threshold distance of the underlying mesh, the object representation system106efficiently generates a volumetric representation without processing an excessive number of samples of the empty areas around a digital object as is done by many conventional systems. Accordingly, the object representation system106can generate a volumetric representation for a digital object utilizing fewer computing resources than conventional systems.

As previously mentioned, the object representation system106provides additional features and applications via the three-dimensional hybrid mesh-volumetric representation generated for a digital object. For example,FIGS.6-8illustrate various features and applications provided via a three-dimensional hybrid mesh-volumetric representation for a digital object in accordance with one or more embodiments.

In particular,FIG.6illustrates modifying a three-dimensional hybrid mesh-volumetric representation for a digital object in accordance with one or more embodiments. As shown inFIG.6, the object representation system106generates a mesh602for a digital object and a three-dimensional hybrid mesh-volumetric representation604corresponding to the mesh602.

As shown inFIG.6, the object representation system106edits the three-dimensional hybrid mesh-volumetric representation604by editing the mesh602. Indeed, as shown, the object representation system106generates a modified mesh606by removing a portion610of the mesh602. Accordingly, the object representation system106generates a modified three-dimensional hybrid mesh-volumetric representation608using the modified mesh606. As illustrated, the modified three-dimensional hybrid mesh-volumetric representation608does not include the portion610that was removed from the mesh602.

In one or more embodiments, the object representation system106facilitates edits by providing one or more options to edit the three-dimensional hybrid mesh-volumetric representation604for display on a client device (e.g., a client device that is displaying the three-dimensional hybrid mesh-volumetric representation604). In some cases, upon selection of one of the options, the object representation system106enables direct user interaction with the three-dimensional hybrid mesh-volumetric representation604. For example, in some cases, the object representation system106receives one or more user interactions for editing the three-dimensional hybrid mesh-volumetric representation604, modifies the mesh602in accordance with the user interaction(s), and then generates the modified three-dimensional hybrid mesh-volumetric representation608.

In some implementations, however, the object representation system106enables user interaction with the mesh602itself. For instance, in some embodiments, in response to receiving a selection of an option to edit the three-dimensional hybrid mesh-volumetric representation604, the object representation system106provides the mesh602for display on the client device. The object representation system106further receives one or more user interactions for editing the mesh602and generates the modified three-dimensional hybrid mesh-volumetric representation608in accordance with the user interaction(s).

Thus, the object representation system106implements more flexible editing of volumetric representations when compared to conventional systems. Indeed, by modifying a three-dimensional hybrid mesh-volumetric representation via modifications to the underlying mesh, the object representation system106provides an intuitive, flexible process for changing the appearance of the three-dimensional hybrid mesh-volumetric representation.

FIG.7illustrates animating a three-dimensional hybrid mesh-volumetric representation for a digital object in accordance with one or more embodiments. As shown inFIG.7, the object representation system106generates a mesh702for a digital object and a three-dimensional hybrid mesh-volumetric representation704corresponding to the mesh702.

As shown inFIG.7, the object representation system106animates the three-dimensional hybrid mesh-volumetric representation704by animating the mesh702. Indeed, as shown, the object representation system106deforms the mesh702(e.g., the using side-to-side movement706), which results in a corresponding deformation of the three-dimensional hybrid mesh-volumetric representation704. In some cases, the object representation system106deforms the mesh702in response to receiving one or more user interactions with the three-dimensional hybrid mesh-volumetric representation704. In some implementations, however, the object representation system106provides the mesh702itself for display on a client device and enables direct user interaction with the mesh702.

In one or more embodiments, the object representation system106deforms the mesh702over a plurality of animation frames. For each animation frame, the deformation causes at least some points on the mesh702to move to a different location in comparison to the previous animation frame. Accordingly, in one or more embodiments, the object representation system106modifies the three-dimensional hybrid mesh-volumetric representation704(e.g., generates a new three-dimensional hybrid mesh-volumetric representation) for each animation frame based on the movement of mesh702.

To illustrate, in one or more embodiments, the object representation system106modifies the three-dimensional hybrid mesh-volumetric representation704for an animation frame by determining new color values and volume densities for the three-dimensional hybrid mesh-volumetric representation704based on the deformation of the mesh702. For example, in some embodiments, the object representation system106generates a new set of rays that intersect the deformed mesh, determines a new set of sample points utilizing the rays, extracts feature vectors for the sample points using UV coordinates determined from the corresponding UV map, and generates color values and volume densities for the sample points from the feature vectors, the ray directions, and the distances between the sample points and the mesh.

Thus, the object representation system106further implements more flexible animation of volumetric representations when compared to conventional systems. Indeed, as the three-dimensional hybrid mesh-volumetric representation is registered to the underlying mesh, the object representation system106allows for flexible manipulation of the three-dimensional hybrid mesh-volumetric representation via manipulation of the mesh.

FIG.8illustrates utilizing a pyramid of neural textures to support mipmaps in accordance with one or more embodiments. Indeed, as shown inFIG.8, the object representation system106generates a pyramid of neural textures802. For example, in one or more embodiments, the object representation system106utilizes a multi-view stereo model to generate at least one UV map and at least one feature map having feature vectors that correspond to the pyramid of neural textures802. For instance, in some cases, the object representation system106utilizes the multi-view stereo model to generate the at least one UV map and the at least one feature map from a set of digital images that portray the corresponding digital object as discussed above with reference toFIG.3.

As further shown inFIG.8, the object representation system106utilizes the pyramid of neural textures802to generate a mipmap804. In one or more embodiments, the object representation system106generates the mipmap804by generating a three-dimensional hybrid mesh-volumetric representation of the corresponding digital object that encodes different levels of detail corresponding to the neural textures802. Indeed, in some cases, the object representation system106generates the three-dimensional hybrid mesh-volumetric representation using a corresponding pyramid of color values and volume densities. In some implementations, however, the object representation system106generates multiple three-dimensional hybrid mesh-volumetric representations corresponding to the neural textures802. For example, in some cases, the object representation system106generates a three-dimensional hybrid mesh-volumetric representation for each level of the pyramid of neural textures802, using corresponding color values and volume densities. Thus, in one or more embodiments, the object representation system106can provide a representation of a digital object for multiple resolutions.

Turning now toFIG.9, additional detail will now be provided regarding various components and capabilities of the object representation system106. In particular,FIG.9illustrates the object representation system106implemented by the computing device900(e.g., the server(s)102and/or one of the client devices110a-110ndiscussed above with reference toFIG.1). Additionally, the object representation system106is also part of the three-dimensional modeling system104. As shown inFIG.9, the object representation system106includes, but is not limited to, a mesh/UV map generator902, a thin-volume sampler904, a neural textures manager906, an object representation generator908, a representation rendering engine910, and data storage912(which includes a multi-view stereo model914, a neural network916, UV maps918, and meshes920).

As just mentioned, and as illustrated inFIG.9, the object representation system106includes the mesh/UV map generator902. In one or more embodiments, the mesh/UV map generator902generates a mesh for a digital object from a set of digital images that portray the digital object. For example, in some cases, the mesh/UV map generator902utilizes a multi-view stereo model to generate the mesh from the set of digital images. Further, in some cases, the mesh/UV map generator902generates a UV map and/or a corresponding feature map for the digital object from the set of digital images using the multi-view stereo model.

Additionally, as shown inFIG.9, the object representation system106includes the thin-volume sampler904. In one or more embodiments, the thin-volume sampler904determines a set of sample points for a thin volume around the mesh created for a digital object. For example, in some cases, the thin-volume sampler904generates a plurality of rays from a source point to the mesh, samples points along the rays, and identifies the points along the rays that are within a threshold distance of the mesh. In some cases, the thin-volume sampler904utilizes rays that intersect the mesh. In some implementations, however, the thin-volume sampler904also utilizes rays that do not intersect the mesh (e.g., rays that are proximate to the mesh but don't intersect).

As further shown inFIG.9, the object representation system106includes the neural textures manager906. In one or more embodiments, the neural textures manager906extracts neural textures for a set of sample points corresponding to a mesh. For instance, in some implementations, the neural textures manager906extracts feature vectors that represent neural textures using the UV map and corresponding feature map generated for the corresponding digital object. In particular, the neural textures manager906extracts the feature vectors for the set of sample points determined for the thin volume around the mesh. In one or more embodiments, the neural textures manager906modifies the thin volume associated with the sample points using the neural textures.

As illustrated inFIG.9, the object representation system106also includes the object representation generator908. In one or more embodiments, the object representation generator908generates a three-dimensional hybrid mesh-volumetric representation of a digital object. For instance, in some cases, the object representation generator908generates the three-dimensional hybrid mesh-volumetric representation using neural textures extracted for the digital object. To illustrate, in some cases, the object representation generator908utilizes a neural network to generate color values and volume densities for the three-dimensional hybrid mesh-volumetric representation from the feature vectors representing the neural textures, the directions of the rays corresponding to the sample points, and the distances between the sample points and the mesh.

As shown inFIG.9, the object representation system106further includes the representation rendering engine910. In one or more embodiments, the object representation system106renders a three-dimensional hybrid mesh-volumetric representation generated for a digital object for display on a computing device. In some implementations, the representation rendering engine910causes another computing device to render the three-dimensional hybrid mesh-volumetric representation for display.

Additionally, as shown, the object representation system106includes data storage912. In particular, data storage912(implemented by one or more memory devices) includes the multi-view stereo model914, the neural network916, the UV maps918, and the meshes920. In one or more embodiments, the multi-view stereo model914stores the multi-view stereo model implemented by the mesh/UV map generator902generate a mesh and a UV map from a set of digital images. In some embodiments, the neural network916stores the neural network implemented by the object representation generator908to generate color values and volume densities in creating a three-dimensional hybrid mesh-volumetric representation for a digital object. In some cases, the UV maps918store UV maps and corresponding feature maps generated for digital objects. Likewise, in some instances, the meshes920stores the meshes generated for digital objects.

Each of the components902-920of the object representation system106can include software, hardware, or both. For example, the components902-920can include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device. When executed by the one or more processors, the computer-executable instructions of the object representation system106can cause the computing device(s) to perform the methods described herein. Alternatively, the components902-920can include hardware, such as a special-purpose processing device to perform a certain function or group of functions. Alternatively, the components902-920of the object representation system106can include a combination of computer-executable instructions and hardware.

Furthermore, the components902-920of the object representation system106may, for example, be implemented as one or more operating systems, as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components902-920of the object representation system106may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components902-920of the object representation system106may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components902-920of the object representation system106may be implemented in a suite of mobile device applications or “apps.” For example, in one or more embodiments, the object representation system106can comprise or operate in connection with digital software applications such as ADOBE® CAPTURE or ADOBE® AERO®. The foregoing are either registered trademarks or trademarks of Adobe Inc. in the United States and/or other countries.

FIGS.1-9, the corresponding text, and the examples provide a number of different methods, systems, devices, and non-transitory computer-readable media of the object representation system106. In addition to the foregoing, one or more embodiments can also be described in terms of flowcharts comprising acts for accomplishing the particular result, as shown inFIG.10.FIG.10may be performed with more or fewer acts. Further, the acts may be performed in different orders. Additionally, the acts described herein may be repeated or performed in parallel with one another or in parallel with different instances of the same or similar acts.

FIG.10illustrates a flowchart of a series of acts1000for generating three-dimensional hybrid mesh-volumetric representation for a digital object in accordance with one or more embodiments.FIG.10illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown inFIG.10. In some implementations, the acts ofFIG.10are performed as part of a computer-implemented method. Alternatively, a non-transitory computer-readable medium can store instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising the acts ofFIG.10. In some embodiments, a system performs the acts ofFIG.10. For example, in one or more embodiments, a system includes one or more memory devices comprising a set of digital images that portray a digital object. The system further includes one or more processors configured to cause the system to perform the acts ofFIG.10.

The series of acts1000includes an act1002for generating a mesh for a digital object portrayed in a set of digital images. For example, in one or more embodiments, the act1002involves generating, from a set of digital images portraying a digital object, a mesh for the digital object utilizing a multi-view stereo model.

Additionally, the series of acts1000includes an act1004for determining a thin volume around the mesh. To illustrate, in one or more embodiments, the act1004involves determining a set of sample points for a thin volume around the mesh.

In one or more embodiments, determining the set of sample points for the thin volume around the mesh comprises generating a ray from a source point to the mesh, the ray intersecting the mesh at one or more locations; and determining a sample point for the set of sample points utilizing the ray. In some implementations, the object representation system106determines the sample point for the set of sample points utilizing the ray by: sampling a plurality of points along the ray intersecting the mesh; and determining a point from the plurality of points along the ray that is within a threshold distance of the mesh.

Further, the series of acts1000includes an act1006for generating a representation for the digital object using the thin volume and the mesh. For instance, in some cases, the act1006involves generating, utilizing a neural network, a three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the set of sample points for the thin volume and the mesh. In one or more embodiments, generating, utilizing the neural network, the three-dimensional hybrid mesh-volumetric representation for the digital object comprises determining, for a sample point from the set of sample points, at least one of a color value or a volume density utilizing the neural network.

In one or more embodiments, the object representation system106further determines a direction of the ray and a distance between the sample point and the mesh for the digital object. Accordingly, in some implementations, the object representation system106generates the three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the set of sample points for the thin volume and the mesh by generating the three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the direction of the ray and the distance between the sample point and the mesh.

In some implementations, the object representation system106further generates, from the set of digital images portraying the digital object, a UV map for the digital object utilizing the multi-view stereo model, the UV map corresponding to a plurality of feature vectors representing neural textures. Accordingly, in some embodiments, the object representation system106generates, utilizing the neural network, the three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the set of sample points for the thin volume and the mesh by generating the three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the set of sample points for the thin volume, the mesh, and the UV map. To illustrate, in at least one embodiment, the object representation system106determines, for a sample point from the set of sample points, a feature vector utilizing the UV map; and generates the three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the set of sample points for the thin volume, the mesh, and the UV map by generating the three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the feature vector for the sample point and the mesh.

In some embodiments, the series of acts1000further includes acts for modifying the three-dimensional hybrid mesh-volumetric representation of a digital object. For instance, in some cases, the acts include modifying the mesh for the digital object by removing a portion of the mesh; and generating a modified three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the set of sample points for the thin volume and the modified mesh. In some cases, the acts include generating an animation of the three-dimensional hybrid mesh-volumetric representation using a manipulation of one or more portions of the mesh.

To provide an illustration, in one or more embodiments, the object representation system106generates, utilizing a multi-view stereo model, a mesh and a UV map for the digital object utilizing a set of digital images; determines a sample point on a ray within a threshold distance of the mesh; determines, for the sample point, a feature vector using the UV map; and generates, utilizing a neural network, a three-dimensional hybrid mesh-volumetric representation of the digital object from the feature vector, a direction of the ray, and a distance between the sample point and the mesh. In some cases, the object representation system106further renders the three-dimensional hybrid mesh-volumetric representation of the digital object for display on a client device.

In one or more embodiments, the object representation system106determines the sample point on the ray within the threshold distance of the mesh by: sampling a plurality of points along the ray; and determining a set of sample points for the ray by including, within the set of sample points, one or more points from the plurality of points that are within the threshold distance and excluding, from the set of sample points, at least one point from the plurality of points that is located outside of the threshold distance.

In some cases, the object representation system106determines, from the sample point, the feature vector using the UV map by: determining a point on the mesh that corresponds to the sample point; and identifying a feature vector that corresponds to the point on the mesh using the UV map. In some embodiments, identifying the feature vector that corresponds to the point on the mesh using the UV map comprises: determining UV coordinates for the point on the mesh; and determining the feature vector from a feature map that is located at the UV coordinates.

In some instances, the object representation system106generates the three-dimensional hybrid mesh-volumetric representation of the digital object from the feature vector, the direction of the ray, and the distance between the sample point and the mesh by determining, for the sample point, a view-dependent color value and a volume density from the feature vector, the direction of the ray, and the distance between the sample point and the mesh.

In some implementations, the object representation system106determines an additional sample point on the ray within the threshold distance of the mesh, wherein the sample point is on a first side of the mesh and the additional sample point is on a second side of the mesh; determines, for the additional sample point, an additional feature vector using the UV map; and generates the three-dimensional hybrid mesh-volumetric representation from the feature vector, the additional feature vector, the direction of the ray, and the distance between the sample point and the mesh.

In some embodiments, the object representation system106further determines, for the sample point, one or more additional feature vectors using at least one UV map for the digital object, wherein the feature vector and the one or more additional feature vectors correspond to a pyramid of neural textures; and generates a mipmap utilizing the feature vector and the one or more additional feature vectors for the sample point.

To provide another illustration, in one or more embodiments, the object representation system106receives a set of digital images that portray a digital object in a plurality of views. Further, the object representation system106generates a three-dimensional hybrid mesh-volumetric representation for the digital object using the set of digital images. The object representation system106further renders the three-dimensional hybrid mesh-volumetric representation for the digital object for display on a computing device.

In some cases, the object representation system106receives, via the computing device, one or more user interactions for manipulating a portion of the three-dimensional hybrid mesh-volumetric representation for the digital object; and modifies the three-dimensional hybrid mesh-volumetric representation in accordance with the one or more user interactions by modifying a portion of a mesh for the digital object that corresponds to the portion of the three-dimensional hybrid mesh-volumetric representation. In some implementations, the object representation system106generates a mipmap utilizing a pyramid of neural textures associated with the three-dimensional hybrid mesh-volumetric representation.

Embodiments of the present disclosure may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. In particular, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices (e.g., any of the media content access devices described herein). In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.

Computer-readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the disclosure can comprise at least two distinctly different kinds of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM, ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM), Flash memory, phase-change memory (“PCM”), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media.

Further, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to non-transitory computer-readable storage media (devices) (or vice versa). For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a “NIC”), and then eventually transferred to computer system RAM and/or to less volatile computer storage media (devices) at a computer system. Thus, it should be understood that non-transitory computer-readable storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions and data which, when executed by a processor, cause a general-purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. In some embodiments, computer-executable instructions are executed on a general-purpose computer to turn the general-purpose computer into a special purpose computer implementing elements of the disclosure. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloud computing environments. In this description, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer ubiquitous and convenient on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud-computing model can also expose various service models, such as, for example, Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computing model can also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth. In this description and in the claims, a “cloud-computing environment” is an environment in which cloud computing is employed.

FIG.11illustrates a block diagram of an example computing device1100that may be configured to perform one or more of the processes described above. One will appreciate that one or more computing devices, such as the computing device1100may represent the computing devices described above (e.g., the server(s)102and/or the client devices110a-110n). In one or more embodiments, the computing device1100may be a mobile device (e.g., a mobile telephone, a smartphone, a PDA, a tablet, a laptop, a camera, a tracker, a watch, a wearable device). In some embodiments, the computing device1100may be a non-mobile device (e.g., a desktop computer or another type of client device). Further, the computing device1100may be a server device that includes cloud-based processing and storage capabilities.

As shown inFIG.11, the computing device1100can include one or more processor(s)1102, memory1104, a storage device1106, input/output interfaces1108(or “I/O interfaces1108”), and a communication interface1110, which may be communicatively coupled by way of a communication infrastructure (e.g., bus1112). While the computing device1100is shown inFIG.11, the components illustrated inFIG.11are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in certain embodiments, the computing device1100includes fewer components than those shown inFIG.11. Components of the computing device1100shown inFIG.11will now be described in additional detail.

In particular embodiments, the processor(s)1102includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, the processor(s)1102may retrieve (or fetch) the instructions from an internal register, an internal cache, memory1104, or a storage device1106and decode and execute them.

The computing device1100includes memory1104, which is coupled to the processor(s)1102. The memory1104may be used for storing data, metadata, and programs for execution by the processor(s). The memory1104may include one or more of volatile and non-volatile memories, such as Random-Access Memory (“RAM”), Read-Only Memory (“ROM”), a solid-state disk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage. The memory1104may be internal or distributed memory.

The computing device1100includes a storage device1106including storage for storing data or instructions. As an example, and not by way of limitation, the storage device1106can include a non-transitory storage medium described above. The storage device1106may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive or a combination these or other storage devices.

As shown, the computing device1100includes one or more I/O interfaces1108, which are provided to allow a user to provide input to (such as user strokes), receive output from, and otherwise transfer data to and from the computing device1100. These I/O interfaces1108may include a mouse, keypad or a keyboard, a touch screen, camera, optical scanner, network interface, modem, other known I/O devices or a combination of such I/O interfaces1108. The touch screen may be activated with a stylus or a finger.

The I/O interfaces1108may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In certain embodiments, I/O interfaces1108are configured to provide graphical data to a display for presentation to a user. The graphical data may be representative of one or more graphical user interfaces and/or any other graphical content as may serve a particular implementation.

The computing device1100can further include a communication interface1110. The communication interface1110can include hardware, software, or both. The communication interface1110provides one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices or one or more networks. As an example, and not by way of limitation, communication interface1110may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI. The computing device1100can further include a bus1112. The bus1112can include hardware, software, or both that connects components of computing device1100to each other.

In the foregoing specification, the invention has been described with reference to specific example embodiments thereof. Various embodiments and aspects of the invention(s) are described with reference to details discussed herein, and the accompanying drawings illustrate the various embodiments. The description above and drawings are illustrative of the invention and are not to be construed as limiting the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the present invention.

The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. For example, the methods described herein may be performed with less or more steps/acts or the steps/acts may be performed in differing orders. Additionally, the steps/acts described herein may be repeated or performed in parallel to one another or in parallel to different instances of the same or similar steps/acts. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.