Patent Description:
The present disclosure relates generally to acquisition of data assets. More particularly, the present disclosure relates to the acquisition of data assets from a third party to provide an augmented reality experience to users.

Augmented reality (AR) can refer to the creating and execution of interactive experiences of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information. As one example, an AR experience can include augmenting a scene captured by a user's camera by inserting virtual objects into the scene and/or modifying the appearance of real world objects included in the scene.

Certain entities may provide the ability for a user to engage in an AR experience relative to objects manufactured or distributed by the entity (e.g., an augmented reality "try-ons"). However, most entities provide these AR experiences on their own website only. The limited accessibility for augmented reality product "try-ons" can require consumers to have to go from one website to another website to engage in different AR experiences for different objects (e.g., to try-on different products and compare the products). This problem can cause the consumer to spend much more computing resources navigating among different websites to facilitate engaging in different AR sessions.

Moreover, web-based implementations of augmented reality experiences can lead to lower frame rates and less realistic renderings compared to an augmented reality experience provided in a native application.

Another issue with engaging in different augmented reality experiences is that different AR applications may be coded in different coding languages. Certain coding languages can lead to security issues, which can be particularly concerning when the application is processing an image or video of a user's face.

<CIT>describes that augmented reality e-commerce may be useful for home improvement store chains that offer products and services. Initially, environmental data that includes spatial data or image data may be received from scanning sensors. The environmental data may be analyzed to detect recognizable patterns that represent environmental features. The environmental features may include surfaces or objects. Subsequently, a user selection of a surface or an object as a feature that is related to a desired product may be received. The feature is then compared to existing data to determine a specific product that fits the environment, in which the existing data includes at least one of virtual image data or product data. Further, an augmented reality environment that corresponds to the environment may be generated for presentation via an augmented reality device, in which the augmented reality environment may present a three-dimensional (3D) virtual representation of the specific product.

<CIT>describes a server for content creation. A content creation tool of the server receives, from a first device, a content identifier of a physical object, a virtual object content, and a selection of a template corresponding to an interactive feature for the virtual object content. The content creation tool generates a content dataset based on the content identifier of the physical object, the virtual object content, and the selected template. The content creation tool provides the content dataset to a second device, the second device configured to display the interactive feature corresponding to the selected template.

<CIT> describes a method for generating an augmented reality module by an apparatus. Processing circuitry of the apparatus obtains preset third party software development interface information, the third party software development interface information being uniformly encapsulated with an AR core engine and an AR rendering engine, and a system parameter and pose information that are associated with the AR core engine being passed into the AR rendering engine. The processing circuitry generates, according to the third party software development interface information and a document configured corresponding to the third party software development interface information, an AR module of a mobile client, the correspondingly configured document comprising interface use information of the AR core engine and the AR rendering engine.

The invention is however defined by the appended set of claims.

One example aspect of the present disclosure is directed to a computer-implemented method for providing third party data assets to clients. The method can include sending, by a computing device, a software development kit. The software development kit can include a template for building one or more rendering effect shaders. The method can include receiving, by the computing device, data assets. In some implementations, the data assets can include one or more rendering effect shaders built using the software development kit. The method can include storing, by the computing device, the data assets and providing, by the computing device, an augmented reality rendering experience. In some implementations, augmented reality renderings can be based at least in part on the data assets.

Another example aspect of the present disclosure is directed to a computing system. The computing system can include one or more processors and one or more non-transitory computer readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include receiving a software development kit comprising a template for generating an augmented reality experience. The operations can include receiving one or more inputs to modify the template. In some implementations, modifying the template can generate a product-specific augmented reality experience. In some implementations, the operations can include generating data assets based at least in part on the one or more inputs and sending the data assets to a second computing device.

Another example aspect of the present disclosure is directed to a computer-readable medium comprising instructions which, when executed by the computing system, cause the computing system to perform operations. The operations can include sending, by a computing device, a software development kit. In some implementations, the software development kit can include a template for building one or more rendering effect shaders. The operations can include receiving, by the computing device, data assets. The data assets can include one or more rendering effect shaders built using the software development kit. The operations can include storing, by the computing device, the data assets and providing, by the computing device, an augmented reality rendering experience. In some implementations, augmented reality renderings can be based at least in part on the data assets.

Generally, the present disclosure is directed to a platform that facilitates the collection of data assets from third parties to provide an augmented reality rendering experience to users. For example, the platform can provide an interface for third parties to build and/or submit rendering effects to be provided to users. The platform may provide an augmented reality rendering experience to users, in which the users can input user data and receive augmented user data as an output. In addition, according to another example aspect, the platform can perform various obfuscation techniques on the received third party data assets to protect third party code.

Thus, example aspects of the present disclosure can provide a system for acquiring third party data assets for augmented reality rendering and obfuscating the acquired code to protect third party proprietary information. In some implementations, the systems and methods can include sending a software development kit to a third party. The software development kit may include a template for building a rendering effect shader. As one example, AR effects can be organized into a shared template and multiple per-product presets. The template can contain shaders and other resources that are common to all products in a particular family (e.g., all lipsticks from a particular vendor). The presets contain parameters corresponding to these shaders, which can include uniform values and textures. The third party can provide data assets back to the platform (e.g., packaged within the SDK). As examples, the data assets can include one or more rendering effect shaders built using the software development kit. The received data assets can then be stored. The data assets can then be used to facilitate an augmented reality rendering experience, in which the data assets can aid in generating an augmented reality try-on experience.

The systems and methods for data asset acquisition can involve one or more systems or devices. The first computing device may be a server, a facilitating computing device, or an intermediary computing device. The second computing device may be a third party computing device. The third party can be a video game company, a product manufacturer, or a product brand. The first computing device and the second computing device can exchange data to generate an augmented reality rendering experience for users. The augmented reality rendering experience can include rendering an augmented reality view that includes one or more products or items. The product may be cosmetics (e.g., lipstick, eye shadow, etc.), furniture or other home goods (e.g., electronic equipment, cookware, glassware, decorations, plants, etc.), clothing, paint colors, automobiles, various electronics, or any other item.

The data asset acquisition can include the first computing device sending a software development kit to the second computing device. The software development kit can include a template for building rendering effect shaders. The software development kit can include example effects, tools to build a rendering effect, and a preview mode to help build an augmented reality rendering. The second computing device can be used to build the rendering effect, and once the rendering effect is built, the second computing device can export the built rendering effect data into a renderable compressed file (e.g., a. ZIP file), which can include data assets needed to recreate the rendering effect. The data assets can then be sent to the first computing device. The first computing device, upon receiving the data assets, can store the data assets for use in an augmented reality rendering experience provided to users. The provided augmented reality rendering experience can be provided to users, where users can input their user data for processing, and the output can be augmented user data that includes the rendering effect built on the second computing device. The user data can be image data or video data captured by a user device. In some implementations, the user data can be a live camera feed.

In some implementations, the one or more rendering effect shaders can include data descriptive of rendering features. One or more rendering effect shaders may be used for rendering a rendering effect for an augmented reality experience. For example, rendering lipstick on a face with augmented reality can include the utilization of a lipstick shader, a texture, uniforms, and a filtering shader. In some implementations, the textures and the uniforms can be input into shaders to aid in generating the rendering effect. In some implementations, the data assets can include the rendering effect shaders. In some implementations, the rendering effect shaders can be processed by a rendering calculator along with the user data to generate a rendered image.

In some implementations, the received data assets can be used to facilitate the generation of augmented reality renderings. A mesh model may be able to process the input data to generate meshes and segmentation masks. An augmentation model may then receive the meshes and segmentation masks and can use the data assets to determine where and how to render a certain rendering. For example, the data assets can include anchoring landmark data. Therefore, the data assets can be used to determine where the rendering needs to occur. In some implementations, the data assets can include shader data. The shader data may be descriptive of lipstick color or eyeshadow color to aid in generation of a make-up augmented reality rendering for the user. Moreover, the shader data may be descriptive of a product's color, glossiness, opacity, and/or various other characteristics.

The augmented reality rendering experience can include a rendering calculator that generates renderings based at least in part on the data assets. In some implementations, the augmented reality rendering experience can include a rendering engine, and the rendering engine can include a rendering calculator.

The augmented reality rendering experience may include a perception subgraph and a rendering subgraph. The perception subgraph can be uniform throughout the system. The perception subgraph may be used with a variety of different rendering subgraphs. The rendering subgraph can be built by a third party to generate a rendering effect to provide to a user. The rendering subgraph may be built then used by the augmented reality rendering experience platform that stores the perception subgraph. The rendering subgraph may vary depending on the rendering effect and the third party. In some implementations, a single perception subgraph can be used with multiple rendering subgraphs to render multiple renderings in an augmented user image or video. For example, a picture or video of a user's face can be processed to generate an augmented reality rendering of lipstick, eyeshadow, and mascara on the face. The processing may include a singular perception subgraph but a rendering subgraph for each respective product (i.e., lipstick, eye shadow, and mascara).

In some implementations, the data assets can include product data descriptive of a product sold by a third party. In some implementations, the systems and methods disclosed herein can be used to compile product augmented reality rendering experiences for retailers to allow consumers to have virtual try-ons of different products from a variety of different brands or providers. The retailer can be an online retailer, in which the consumer can virtually try-on products in the comfort of their own home. In some implementations, the retailer can be a physical store retailer, in which the augmented reality experience can be enabled through a mobile application or a computing device found inside the store. In some implementations, the systems and methods disclosed herein can enable an augmented reality rendering experience inside a search engine application, in which a consumer may search for a brand of a product, a type of product, a color of a product, etc., and the search engine can provide results based on the search query, in which the results include an option to try-on a determined result using augmented reality.

In some implementations, the received data assets can allow for the generation of a variety of augmented reality experiences. For example, the data assets can include data assets generated by a manufacturer, distributor, seller, etc. of furniture or other home goods to aid in rendering one or more of their products inside of a home. In this implementation, the third party may modify and fine-tune the software development kit to be able to generate a furniture or other home goods augmented reality experience. When the third party is satisfied with generated experience, they may export and send the data assets to the facilitating system/platform to store. The facilitating system can then use the data assets to enable the augmented reality rendering experience accessible to users.

The augmented reality rendering is generated by receiving user data, processing the user data with an encoder model to generate a user mesh, and processing the user mesh with an augmentation model to generate the augmented reality rendering. In some implementations, the mesh can be a polygonal mesh. The augmentation model includes shaders based at least in part on the data assets. In some implementations, the augmentation model can generate the augmented reality rendering by rendering the polygonal mesh using custom shaders, superimposed over the camera feed image.

In some implementations, the systems and methods can include a third party computing device intaking and modifying a software development kit. The method can include receiving a software development kit configured to be modified to generate an augmented reality experience. Furthermore, the method can include receiving one or more inputs to modify the software development kit, in which modifying the software development kit generates a product-specific augmented reality experience. The method can include generating data assets based at least in part on the one or more input and sending the data assets to a second computing device.

In some implementations, the third party computing device may test the augmented reality experience for fine-tuning. Testing the augmented reality experience can include comparing rendered images and pre-rendered reference ("golden") images. Alternatively, in some implementations, testing can include obtaining training data, processing the training data with the augmented reality experience to generate augmented reality media, and comparing the augmented reality media and reference data. One or more parameters of the data assets can be adjusted (e.g., automatically using a machine learning algorithm) based at least in part on the comparison (e.g., based on a loss function that compares the augmented reality media and the reference data).

In some implementations, the systems and methods may process the received data assets to obfuscate the code. The obfuscation can include removing whitespace from the code, removing one or more comments from the code, and renaming terms and symbols in the code. The one or more comments may be removed, because the one or more comments may include text descriptive of code semantics. Renaming one or more terms in the data asset's code can include uniform renaming across files. The uniform renaming process can include indexing terms to provide a reference for future renaming uniformity. In some implementations, the renaming can include using hashing functions. Hashing may be indexed in a global registry or an index table. New symbols may be actively added to the index as received. The renamed terms and symbols may be deterministic terms. In some implementations, the system can rename symbols such as function names that are shared among multiple files.

Renaming can include parsing the shader code and isolating the specific declarations for replacement. Stripping whitespace can remove the organization of the code to remove some logic included in formatting code in certain ways. Obfuscation can include removing semantic content that could include shader information (e.g., proprietary information found in OpenGL code, GLSL, or other native sources) the third party does not want to make public.

In some implementations, the platform can be used for generating a directory of product renderings. The directory of augmented reality renderings can be implemented to provide the renderings for advertisements, video web applications, or mobile apps. In some implementations, shared code may be integrated into the directory as a library dependency.

Receiving or ingesting the data assets can occur through a variety of methods. In some implementations, the ingestion can occur through a built in export function in the platform. In other implementations, the data assets can be delivered through emailing a single effect template, parameterized by shader uniforms defined in a single top-level file. For example, for a lipstick rendering, the third party can send shader uniform values for each lipstick product they want provided to a user. The systems and methods can combine the effect template and per-product shader uniforms, producing an AR effect (e.g., a beauty effect) per product. In some implementations, the data assets can be ingested through an internet-based data feed instead of email. Data assets can be ingested individually or en masse.

In some implementations, the software development kit can be configured for different product types. For example, a software development kit for building data assets for rendering a lipstick can include different templates and presets compared to a software development kit for building data assets for rendering a couch. Moreover, the organization of beauty effects into a shared "template" and multiple per-product presets can aid third parties in building data assets for rendering certain products. The template can contain shaders and other resources that are common to all products in a particular family (e.g., all lipsticks from vendor A). The presets can contain parameters corresponding to these shaders, which can include uniform values and textures.

In some implementations, the systems and methods disclosed herein can be implemented as a native application. The native application can provide the client with an augmented reality rendering experience that includes third party product renderings for selection by a client.

The systems and methods disclosed herein may also be applicable to other technologies, including mixed reality. In some implementations, third parties can use the software development kit to build interactive renderings. For example, a furniture brand may build various rendering effects for their various recliners and various expandable tables. The rendering effect shaders can be transferred to the augmented reality rendering experience platform to provide a mixed reality rendering experience, in which the user can render the furniture in their home. The user can then interact with the renderings to see a rendering of the furniture in their default position and in their alternative position. Therefore, a user can test out how a recliner may fit in their home in the up-right position and in the reclined position. The user can use the mixed reality experience to determine if the expanded table fits in a given room.

Furthermore, the platform of data asset acquisition and obfuscation can be applied to a variety of other platforms for generating supported user experiences. The platform for data acquisition can be utilized for supported application creation, embedded feature creation, and widget creation.

Moreover, in some implementations, the systems and methods may be used as a visual compatibility calculator. For example, the systems and methods can be used to ensure a certain product or part will fit in the desired space or location. The systems and methods can be used to virtually test-out the measurements/size of a product using virtual reality. The third party can provide data assets that includes data descriptive of the measurements of a product. The data assets can then be used to provide an augmented reality rendering experience to a user in which a product is rendered according to the measurements provided by the third party. This aspect can allow consumers to "try-on" products to visualize the space the product may take-up.

The systems and methods of the present disclosure provide a number of technical effects and benefits. As one example, the system and methods can receive parameters for augmented reality from third parties to allow clients to virtually see a product in application. The systems and methods can further be used to ensure the security of the provided data from third parties. Furthermore, the systems and methods can enable the centralized collection of augmented reality data sets for virtual try-ons from a variety of vendors to allow a user to try on products from multiple vendors without having to navigate from website to website or from proprietary application to proprietary application.

Furthermore, in some implementations, the disclosed systems and methods can be implemented into a native application. The implementation of the systems and methods in a native application can provide a higher frame rates and more realistic renderings compared to the web-application alternative.

Another technical benefit of the systems and methods of the present disclosure is the ability to obfuscate code (e.g., shader code). The systems and methods can cause the code to become clustered without comments or symbols that divulge proprietary information found in the originally provided source code. As the systems and methods can also obfuscate the received data assets, the third parties can provide their data without divulging certain proprietary information.

<FIG> depicts a block diagram of an example computing system <NUM> that performs data asset acquisition according to example embodiments of the present disclosure. The system <NUM> includes a client computing system <NUM>, an augmented reality platform <NUM>, and a third party computing system.

As illustrated in <FIG>, the augmented reality platform <NUM> can communicate with the third party computing system <NUM> to generate a third party-augmented reality (AR) asset library <NUM>. The third party AR asset library <NUM> can be leveraged to provide an augmented reality (AR) experience to a client computing system <NUM> via a client interface <NUM>.

For example, the augmented reality platform <NUM> can provide a software development kit (SDK) with templates to the third party computing system <NUM>. The third party computing system <NUM> can use the SDK with templates to build augmented reality rendering effects descriptive of products sold by the third party. The completed rendering effects can be provided back to the augmented reality platform <NUM> as completed data assets. Each completed data asset can be stored in the augmented reality platform's <NUM> third party AR asset library <NUM>. The stored product assets in the third party AR asset library <NUM> can be from a singular third party or from a plethora of third parties. For example, Product <NUM> Assets <NUM>, Product <NUM> Assets <NUM>, all the way to Product N Assets <NUM> can be from a singular third party computing system and can include a variety of products provided by the third party. Alternatively, Product <NUM> Assets <NUM> and Product <NUM> Assets <NUM> can be provided by different third parties and can include data assets descriptive of different products from different brands.

The augmented reality platform <NUM> can intake client data via the client interface <NUM> to be processed by the rendering engine <NUM> to provide an AR experience to the client computing system <NUM>. The rendering engine <NUM> can process the client data with a perception model and an augmentation model. The perception model can output one or more meshes and one or more segmentation masks that can be input into the augmentation model. The augmentation model can process the client data, the one or more meshes, and the one or more segmentation masks to output an augmented reality client image or video, which can be sent to the client computing system <NUM> via the client interface <NUM>.

In some implementations, the AR experience can include the client computing system <NUM> sending a selection to the augmented reality platform <NUM> to indicate a desired product to "try-on. " The augmented reality platform can use the stored data asset from the third party AR asset library <NUM> to render the product in a client provided image or video by parameterizing the augmentation model of the rendering engine <NUM> using the data assets. For example, the client computing system <NUM> may use the client interface <NUM> to select Product <NUM> to virtually try-on. Product <NUM> Assets <NUM> can be provided to the rendering engine <NUM> along with a set of client data. The rendering engine <NUM> can process the set of client data and the Product <NUM> Assets <NUM> to generate images or video of Product <NUM> in the images or video provided by the client computing system <NUM>.

<FIG> depicts a block diagram of an example computing system <NUM> that performs data asset acquisition and obfuscation according to example embodiments of the present disclosure. The system <NUM> includes a user computing device <NUM>, a server computing system <NUM>, and a training computing system <NUM> that are communicatively coupled over a network <NUM>.

The user computing device <NUM> can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

The user computing device <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the user computing device <NUM> to perform operations.

In some implementations, the user computing device <NUM> can store or include one or more augmented reality rendering models <NUM>. For example, the augmented reality rendering models <NUM> can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Example augmented reality rendering models <NUM> are discussed with reference to <FIG> and <FIG>.

In some implementations, the one or more augmented reality rendering models <NUM> can include data assets received from the training computing system <NUM> over network <NUM>, stored in the user computing device memory <NUM>, and then used or otherwise implemented by the one or more processors <NUM>. In some implementations, the user computing device <NUM> can implement multiple parallel instances of a single augmented reality rendering model <NUM> (e.g., to perform parallel renderings of effects).

More particularly, the server computing system <NUM> and the training computing system <NUM> can exchange data to generate data assets that can enable the augmented reality rendering models to process image or video data and output augmented image data or augmented video data.

Additionally or alternatively, one or more augmented reality rendering models <NUM> can be included in or otherwise stored and implemented by the server computing system <NUM> that communicates with the user computing device <NUM> according to a client-server relationship. For example, the augmented reality rendering models <NUM> can be implemented by the server computing system <NUM> as a portion of a web service (e.g., a "live try-on" service for make-up, clothing, electronics, automobiles, or furniture or other home goods). Thus, one or more models <NUM> can be stored and implemented at the user computing device <NUM> and/or one or more models <NUM> can be stored and implemented at the server computing system <NUM>.

The user computing device <NUM> can also include one or more user input component <NUM> that receives user input. For example, the user input component <NUM> can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

The server computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the server computing system <NUM> to perform operations.

As described above, the server computing system <NUM> can store or otherwise include one or more machine-learned augmented reality rendering models <NUM>. For example, the models <NUM> can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Example models <NUM> are discussed with reference to <FIG> and <FIG>.

The user computing device <NUM> and/or the server computing system <NUM> can train the models <NUM> and/or <NUM> via interaction with the training computing system <NUM> that is communicatively coupled over the network <NUM>. The training computing system <NUM> can be separate from the server computing system <NUM> or can be a portion of the server computing system <NUM>.

The training computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the training computing system <NUM> to perform operations. In some implementations, the training computing system <NUM> includes or is otherwise implemented by one or more server computing devices.

The training computing system <NUM> can include a model trainer <NUM> that trains the machine-learned models <NUM> and/or <NUM> stored at the user computing device <NUM> and/or the server computing system <NUM> using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and/or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer <NUM> can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

In particular, the model trainer <NUM> can train the augmented reality rendering models <NUM> and/or <NUM> based on a set of training data <NUM>. The training data <NUM> can include, for example, shaders built by a third party with a software development kit, in which the third party received the software development kit from a facilitating computing device or the server computing system <NUM>. The third party may have generated the shaders and the data assets by building and testing augmented reality experiences with the software development kit.

In some implementations, if the user has provided consent, the training examples can be provided by the user computing device <NUM>. Thus, in such implementations, the model <NUM> provided to the user computing device <NUM> can be trained by the training computing system <NUM> on user-specific data received from the user computing device <NUM>. In some instances, this process can be referred to as personalizing the model.

The model trainer <NUM> includes computer logic utilized to provide desired functionality. The model trainer <NUM> can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer <NUM> includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer <NUM> includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media.

The machine-learned models described in this specification may be used in a variety of tasks, applications, and/or use cases.

In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.

In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.

In some implementations, the input to the machine-learned model(s) of the present disclosure can be latent encoding data (e.g., a latent space representation of an input, etc.). The machine-learned model(s) can process the latent encoding data to generate an output. As an example, the machine-learned model(s) can process the latent encoding data to generate a recognition output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reconstruction output. As another example, the machine-learned model(s) can process the latent encoding data to generate a search output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reclustering output. As another example, the machine-learned model(s) can process the latent encoding data to generate a prediction output.

In some implementations, the input to the machine-learned model(s) of the present disclosure can be sensor data. The machine-learned model(s) can process the sensor data to generate an output. As an example, the machine-learned model(s) can process the sensor data to generate a recognition output. As another example, the machine-learned model(s) can process the sensor data to generate a prediction output. As another example, the machine-learned model(s) can process the sensor data to generate a classification output. As another example, the machine-learned model(s) can process the sensor data to generate a segmentation output. As another example, the machine-learned model(s) can process the sensor data to generate a segmentation output. As another example, the machine-learned model(s) can process the sensor data to generate a visualization output. As another example, the machine-learned model(s) can process the sensor data to generate a diagnostic output. As another example, the machine-learned model(s) can process the sensor data to generate a detection output.

In some cases, the machine-learned model(s) can be configured to perform a task that includes encoding input data for reliable and/or efficient transmission or storage (and/or corresponding decoding). In another example, the input includes visual data (e.g., one or more images or videos), the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding for input data (e.g., visual data).

In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest. As another example, the image processing task can be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, a motion of the scene depicted at the pixel between the images in the network input.

<FIG> illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device <NUM> can include the model trainer <NUM> and the training dataset <NUM>. In such implementations, the models <NUM> can be both trained and used locally at the user computing device <NUM>. In some of such implementations, the user computing device <NUM> can implement the model trainer <NUM> to personalize the models <NUM> based on user-specific data.

<FIG> depicts a block diagram of an example computing device <NUM> that performs according to example embodiments of the present disclosure. The computing device <NUM> can be a user computing device or a server computing device.

The central intelligence layer includes a number of machine-learned models. For example, as illustrated in <FIG>, a respective machine-learned model (e.g., a model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device <NUM>.

In some implementations, the systems and methods can be used as a rendering pipeline. The pipeline can include a software development kit (SDK) that can include all the tools needed to build a renderable compressed file (e.g., a ZIP file). In some implementations, the software development kit with the compressed file (e.g., a ZIP file) can be tested with viewers on various platforms. The source assets generated while building with the software development kit can be used to augment still images or video.

The software development kit can be sent to a third party. The third party can build and preview inward-facing rendering pipelines. The pipelines can be used for a variety of uses including, but not limited to, beauty product try-ons with a computer or a mobile device.

The software development kit can include a script with associated binaries, a set of preview tools, documentation, and a set of sample effects. The script and associated binaries can be used for compiling product effect sources into renderable compressed files. The set of preview tools can be used for visualizing the rendered results. Moreover, the preview tools can provide a joint interface for editing and preview, or a quick interface switch between the two. Documentation can include raw HTML or other forms of documentation for review to aid in the building process. The set of sample effects can aid the third parties in understanding the software development kit, while providing baselines.

The software development kit can be designed to be self-sufficient, easy to port, able to run on stock hardware, be fast, and be secure. The software development kit can be designed to run without dependencies to allow for third parties or other creators to build renderings using the software development kit without having to rely on other applications. The back-end can mirror the back-end used by the system interfacing with the consumers. Moreover, the easy porting can allow for third parties to use their existing shaders with minimal to no modification. Furthermore, the software development kit can be designed to run on a variety of operating systems without requiring software installation outside of the software development kit. In some implementations, the software development kit can include features that allow for opening up render graphs and GPU shaders for customization. The interface can eliminate the need for implementing third party GPU code into the system interfacing with the consumer, which can maintain security for the user. The software development kit can use a rendering calculator to transform incoming data into outgoing data.

Inward-facing augmented reality effect generation can involve two components. The first component can involve perception. The perception component can compute and output pixel coordinates of landmarks in an image (e.g., lips on a face). The second component can involve a rendering component. The rendering component can include rendering the augmented reality effect on the original received frame with the computed landmarks facilitating the location. The result can then be output.

Graphs for augmented reality rendering can be divided into a third party subgraph and a facilitator subgraph. The facilitator subgraph can be a perception subgraph, while the third party subgraph can be a rendering subgraph that can be edited by both the third party and the facilitator. The separation can allow for modification of the perception subgraph by the facilitator without affecting the rendering subgraph. Moreover, the separation can allow for a single perception subgraph to be used even if multiple augmented reality effects are being rendered. Therefore, multiple rendering subgraphs may be layered on a single perception subgraph to produce multiple renderings with the computation of a single perception model process.

The aggregation of a perception subgraph and a rendering subgraph can produce a complete graph or augmented reality media.

A third party effect source can contain the rendering subgraph file and one or more directories of assets consumed by the rendering calculators in the subgraph.

In some implementations, the software development kit may include a perception model. The software development kit can invoke bundled binaries to convert each type of source file into a processed format, that can preserve the input's directory structure in the generated renderable compressed file, while emitting any errors encountered along the way. Asset files such as graph protos, shaders, and blueprints may reference other files that can be included in the software development kit or are part of the effect source. Textures can be converted into webp format based on a per-effect image_conversion. txt file that customizes the conversions.

In some implementations, product effects built with the software development kit can contain GLSL shaders that can be served onto users' devices for runtime compilation. These shaders can represent valuable intellectual property from third party tech providers. The systems and methods disclosed herein can obfuscate the shaders to aid in protecting the information. Obfuscation can include stripping comments and syntactically superfluous whitespace, and then systematically renaming most non-reserved symbols so as to obscure semantics. In some implementations, the obfuscated symbols can be obfuscated uniformly amongst files. The obfuscation can occur after inspecting and testing the original shaders obtained from third party partners.

The systems and methods can be applied to a variety of augmented reality renderings including, but not limited to, make-up renderings, furniture renderings, apparel renderings, video game renderings, and building structure renderings.

<FIG> depicts a block diagram of an example data asset acquisition system <NUM> according to example embodiments of the present disclosure. In some implementations, the data asset acquisition system <NUM> is trained to send a software development kit <NUM> for building an augmented reality rendering experience and, as a result of sending the software development kit <NUM>, receive templates and presets <NUM> from a third party. Thus, in some implementations, the data acquisition system <NUM> can include a facilitator <NUM>, a third party <NUM>, and a user <NUM>.

In particular, <FIG> depicts a system for data asset acquisition for use in generating an augmented reality rendering experience. A facilitator <NUM> can be a facilitating system for compiling data assets for augmented reality rendering by communicating with third parties <NUM>. The facilitator <NUM> can be a server, a web application, or a facilitating computing system. When the facilitator <NUM> has received a data asset, the facilitating system can provide an augmented reality rendering experience to a user <NUM>.

The data asset acquisition can include a facilitator built software development kit (SDK) <NUM> being sent to a third party <NUM>. The third party can use the software development kit <NUM> to build rendering experiences. The software development kit <NUM> can compile the data assets <NUM> and can allow for rendering previews <NUM> of the generated rendering experiences.

The third party <NUM> may use the rendering preview to determine what, if any, source modifications <NUM> need to be made to the templates and presets <NUM> of their data assets. When building, testing, and fine-tuning is completed, the third party can send their data assets to the facilitating system, including the templates and presets <NUM> along with third party metadata <NUM>. The facilitating system <NUM> can ingest the effects <NUM> and store them for later providing an augmented reality rendering experience that can include the rendering effects built by the third party <NUM>. The facilitator can intake user data <NUM> from the user <NUM> and can output augmented user data with the rendering effect included in the augmented user data.

<FIG> depicts a block diagram of an example obfuscation <NUM> according to example embodiments of the present disclosure. The obfuscation <NUM> can be included in the data asset acquisition system <NUM> of <FIG> to obfuscate the data assets.

More particularly, <FIG> depicts a three pronged approach to obfuscation. In this implementation, the original code <NUM> can be obfuscated with an obfuscation system <NUM> to generate obfuscated code <NUM>. In some implementations, the original code <NUM> can be data asset code for augmented reality renderings.

Furthermore, in this implementation, the obfuscation system <NUM> can include renaming <NUM> symbols or terms, removing whitespace <NUM>, and removing comments <NUM> pertaining to semantics.

Renaming <NUM> can include indexing symbols and terms from the original code and replacing the original symbols and terms with assigned symbols and terms. In some implementations, renaming <NUM> can include the utilization of hashing functions. The renaming can be uniform throughout different files.

Removing whitespace <NUM> can include removing indentations, blank lines, and returns. The removal of whitespace can obscure the logical format of code making the code harder to read.

Removing comments <NUM> can include removing one or more comments that relate to code semantics. The obfuscation system may process the code and determine if comments are semantic comments. If a comment serves to explain code semantics, that comment can be removed.

These three components can decrease readability of the code while also hiding third party symbols and terms.

<FIG> depicts a block diagram of an example data asset acquisition system <NUM> according to example embodiments of the present disclosure. The data asset acquisition system <NUM> is similar to data asset acquisition system <NUM> of <FIG> except that data asset acquisition system <NUM> can be specifically configured for beauty effect renderings.

In this implementation, the beauty effect <NUM> can include a make-up rendering. The beauty effect can include different data sets. The data sets can include a render entity <NUM> data sets with blueprints, GLSL Shaders <NUM>, textures, and geometry files. The beauty effect <NUM> can further include rendering subgraphs, shader uniforms, and image conversions. These data sets can be input into the software development kit <NUM> to build an augmented reality rendering experience. The software development kit <NUM> can allow the builder to preview and test the newly built experience. In particular the GLSL shaders <NUM> of the beauty effect can be converted into a shader pipeline inside the software development kit. The built augmented reality rendering experience can parallel the previously existing beauty effect <NUM> rendering experience outside of the software development kit <NUM>. When the build is complete the software development kit can generate a renderable zip file <NUM> for use in recreating the augmented reality rendering experience on another device.

<FIG> depicts a block diagram of an example augmented reality rendering experience model <NUM> according to example embodiments of the present disclosure. The augmented reality rendering experience model <NUM> is similar to data asset acquisition system <NUM> of <FIG> except that augmented reality rendering experience model <NUM> further includes the processing of a camera feed with a mesh model and an augmentation model with the augmentation model using the generated data assets.

In particular, <FIG> depicts an example perception subgraph <NUM> and an example rendering subgraph <NUM> being used to process a camera feed <NUM> to generate a rendered image <NUM>. The perception subgraph <NUM> can process the camera feed <NUM> with a mesh model <NUM> to generate a mesh and a segmentation mask. The mesh and the segmentation mask can be processed by rendering calculators <NUM>. The rendering calculators <NUM> can be included in the rendering subgraph <NUM> and can be influenced by shaders. The mesh, the segmentation mask, and the camera feed <NUM> can be processed by the rendering calculators <NUM> to generate a rendered image. The rendered image can include an augmented reality rendering. The rendering can be a rendering generated by a third party using a software development kit. Moreover, in some implementations, the rendering can be a make-up rendering, in which the mesh model can be a face tracker, the shaders can include a lipstick shader, a texture shader, a uniform shader, and/or a filtering shader.

<FIG> depicts a flow chart diagram of an example method to perform according to example embodiments of the present disclosure. Although <FIG> depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method <NUM> can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the present disclosure.

At <NUM>, a computing system can send a software development kit. In some implementations, the software development kit can include a template for building one or more rendering effect shaders. The software development kit may include a script and associated binaries for compiling source effects into renderable files. The software development kit may further include preview tools for the configuring third party to preview their render build during the build for fine-tuning and testing. In some implementations, the software development kit can include documentation and sample effects for aiding the third party in how to build and fine-tune effects with the software development kit.

At <NUM>, the computing system can receive data assets. In some implementations, the data assets can include one or more rendering effect shaders built using the software development kit. The data assets can be received in the form of a renderable file (e.g., a. The data assets can include shader data for rendering a product or item sold or provided by a third party. The product may be cosmetics (e.g., lipstick, eye shadow, etc.) furniture or other home goods (e.g., electronic equipment, cookware, glassware, decorations, plants, etc.), clothing, paint colors, automobiles, or any other item.

At <NUM>, the computing system can store the data assets. The data assets can be stored on a server, a user computing device, or a facilitator computing device.

At <NUM>, the computing system can provide an augmented reality rendering experience. In some implementations, augmented reality renderings can be based at least in part on the data assets. The augmented reality renderings can include product renderings of products sold by a third party.

At <NUM>, a computing system can receive a software development kit. The software development kit can include a template for generating an augmented reality experience. The software development kit may be sent by a facilitating computing system or an intermediary system to compile rendering experiences to provide to a user.

At <NUM>, the computing system can receive inputs to modify the software development kit. Modifying the software development kit can modify the template and can generate a product-specific augmented reality experience. For example, the software development kit can be built and configured to generate a make-up rendering effect. The make-up rendering effect can be implemented into an augmented reality rendering experience for users to "try-on" different lipstick colors or types provided using their personal computing device.

At <NUM>, the computing system can generate data assets. In some implementations, the data assets can be generated based at least in part on the received inputs. The data assets can be converted into a renderable compressed file for sending.

At <NUM>, the computing system can send the data assets to a second computing device. The second computing device may be the sender of the software development kit. In some implementations, the second computing device can be a facilitating computing device or an intermediary computing device that interacts with users to provide the users an augmented reality rendering experience.

At <NUM>, a computing system cand send a software development kit. In some implementations, the software development kit can include a template for building one or more rendering effect shaders. The base application can be an augmented reality rendering application with example effects and directions for building new renderings or translating previously built rendering effects.

At <NUM>, the computing system can receive data assets. In some implementations, the data assets can include one or more rendering effect shaders built using the software development kit. The data assets can be received as a renderable file.

At <NUM>, the computing system can obfuscate the data assets. Obfuscating the data assets can include removing whitespace from the code, removing comments regarding code semantics, and renaming symbols and terms. The obfuscation can decrease code readability and remove possible proprietary information included in terminology or comments. The obfuscation can help protect the third party builders.

At <NUM>, the computing system can store the data assets. The data assets can be stored locally or via a server. The data assets can be stored to be readily accessible for use with a web application or a mobile application.

At <NUM>, the computing system can provide an augmented reality rendering experience. In some implementations, augmented reality renderings can be based at least in part on the data assets. The augmented reality rendering experience can include providing an experience that augments user images and videos to include rendering effects built by a third party. The rendering effect can be part of a video game or a "live try-on" experience. Moreover, the augmented reality rendering experience can be provided via a web application, a mobile application, or an in store kiosk.

Claim 1:
A computer-implemented method for providing third party data assets to clients, the method comprising:
sending, by a computing device (<NUM>), a software development kit, wherein the software development kit comprises a template for building one or more rendering effect shaders;
receiving, by the computing device (<NUM>), data assets, wherein the data assets comprise one or more rendering effect shaders built using the software development kit;
storing, by the computing device (<NUM>), the data assets; and
providing, by the computing device (<NUM>), an augmented reality rendering experience, wherein augmented reality renderings are based at least in part on the data assets, wherein the augmented reality rendering is generated by:
receiving, by the computing device (<NUM>), user data including image data or video data captured by a user device;
processing, by the computing device (<NUM>), the user data with an encoder model to generate a user mesh; and
processing, by the computing device (<NUM>), the user mesh with an augmentation model to generate the augmented reality rendering, wherein the augmentation model comprises shaders based at least in part on the data assets.