Patent Description:
<CIT> discloses a system in which a real-time image server provides a mobile device with visualisation data. The visualization functionality is provided in the real-time image server.

The disclosed systems and methods can enable a webpage to provide interactive visualization of products in images selected or provided by users. More specifically, the disclosed systems and methods can enable a webpage to provide interactive visualization of an image of a product in images selected or provided by a user. In some embodiments, the webpage can provide (or be modified to provide) single-page application functionality, allowing the user to select products for which visualizations are available and interact with the visualizations in the selected or provided images, without refreshing the webpage, opening a new webpage, or navigating away from the webpage.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments or the scope of the inventions as claimed. The concepts in this application may be employed in other embodiments without departing from the scope of the inventions.

The drawings are not necessarily to scale or exhaustive. Instead, emphasis is generally placed upon illustrating the principles of the inventions described herein. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. In the drawings:.

Reference will now be made in detail to exemplary embodiments, discussed with regards to the accompanying drawings. In some instances, the same reference numbers will be used throughout the drawings and the following description to refer to the same or like parts. Unless otherwise defined, technical or scientific terms have the meaning commonly understood by one of ordinary skill in the art. The disclosed embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosed embodiments. It is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the disclosed embodiments. Thus, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting. It is to be understood in the ensuing description, that reference to providing visualization of a product, refers to providing visualization of an image of the product. The image may be a two-dimensional or a three-dimensional image.

The disclosed embodiments can enable a webpage to provide interactive visualization of products in images selected or provided by users. The products can include physical goods. For example, the products can include furniture (e.g., a table, chair, or the like), decorative objects or decorations (e.g., a vase, picture, or the like), functional objects (e.g., a lamp, consumer electronic device, or the like), construction or finishing materials (e.g., flooring, paneling or molding, a countertop material, paint, wallpaper, or the like), or like physical goods. The interactive visualizations can benefit users by affording them the opportunity to view a product in a particular location before obtaining the product. The disclosed embodiments can be structured as single-page applications, with the webpage modified as-needed to support interactive visualization, without reloading the webpage or navigating to a new webpage. Such a single-page application framework can improve the user experience, as reloading the webpage or navigating to a new webpage can disrupt the user experience, and may also reduce bandwidth.

A visualization application can be integrated into the webpage to provide the interactive visualizations. Integration of the visualization application into the webpage can be handled using a script, which can be customized for the webpage. The script can configure the web browser displaying the webpage to perform a variety of useful functions. Such functions can include determining whether the browser can support the visualization application; determining whether products described or referenced in the product page are available for visualization; determining an appropriate language for providing the visualization; filtering out bots and crawlers that might otherwise pollute usage statistics; placing first-party cookies that can be used to track user sessions, as well as conversions (e.g., purchases or the like); and integrating the visualization application with an ordering application, so that users can initiate orders from within the visualization application. The script can be embedded into the webpage using a tag. Thus the functionality provided by the disclosed embodiments can be added to a webpage without requiring extensive modifications to the webpage. Furthermore, the visualization application may run independently of the webpage, such that user manipulation of an image displayed in the visualization application may occur without requiring reloading of the webpage.

<FIG> depicts an exemplary system <NUM> for providing interactive visualizations using a single-page application, in accordance with the disclosed embodiments. As depicted in <FIG>, system <NUM> can include client system <NUM>, host system <NUM>, and application system <NUM>. Client system <NUM>, host system <NUM>, and application system <NUM> can be configured to communicate using network <NUM>. Client system <NUM> can be configured to retrieve the webpage from host system <NUM> and retrieve the script and visualization application from application system <NUM>. The script can be used to integrate the webpage and the visualization application, removing from the developer of the webpage the burden of integrating the webpage and visualization application. The visualization application can allow the user to modify an image to display products, and more specifically images of products, depicted or referenced in the webpage.

Client system <NUM> can be configured, according to a script, to display a webpage including a visualization of a product in an image. Client system <NUM> can be or include an interactive computing device with a display. For example client system <NUM> can be or include a desktop, a laptop, smart phone, a tablet, or a wearable device. Client system <NUM> can be configured to obtain the webpage from host system <NUM>. For example, client system <NUM> can be configured with a web browser and a user of client system <NUM> can interact with the web browser (e.g., by entering a URL into an address bar or selecting a reference to the webpage in another webpage, or the like) to cause client system <NUM> to request a webpage from host system <NUM>. Client system <NUM> can use the webpage to implement a single-page application, in which the webpage is repeatedly modified (e.g., in response to user interactions or data or instructions received from host system <NUM> or application system <NUM>). The webpage can include instructions that, when processed by client system <NUM>, cause client system <NUM> to obtain a script from application system <NUM>. Client system <NUM> can be configured to execute the script.

Client system <NUM> can be configured, according to the script, to determine whether the webpage can be modified to display an image modified to depict a product. In some embodiments, client system <NUM> can determine the products displayed or referenced in the webpage. In various embodiments, client system <NUM> can determine whether application system <NUM> can obtain visualization information enabling modification of an image to display a product depicted or referenced in the webpage. When such visualization information is available, client system <NUM> can be configured to modify the webpage to display a control.

Client system <NUM> can be configured, according to the script and in response to selection of the control, to modify the webpage to display a visualization application. Client system <NUM> can use the visualization application, in accordance with the script, to display the modified image. The visualization application can enable a user of client system <NUM> to select the image (e.g., by uploading the image or selecting a saved or predetermined image). Client system <NUM> can transmit a request to display the modified image to application system <NUM>. Client system <NUM> can receive instructions for displaying the product in the image from application system <NUM>. In some embodiments, the instructions can include a version of the image, modified to display the product in the image. In various embodiments, the instructions can include a model of the product and instructions or data enabling the visualization application to display the model of the product in the image. For example, the instructions and data can specify at least one of a scaling, perspective, or lighting of the product in the image. Client system <NUM> can then display the image in accordance with the instructions received from application system <NUM>. For example, when the instructions include a model of the product and information describing the scale and perspective of the object at a location in the image, client system <NUM> can be configured to place an image of the product, based on the model, scale, and perspective information, at the location in the image. In some embodiments, client system <NUM> can be configured to subsequently update the image based on user interaction. For example, a user can interact with the visualization application to change the position or orientation of the product within the image. The client can then recalculate how the model of the product is displayed, based on the changed position or orientation information and the instructions previously received from application system <NUM>.

In some embodiments, the script can be configured to manage interactions or control communications between the visualization application and the webpage. For example, the visualization application can create events in response to user interactions. These events can be handled by the script. For example, the user can interact with the visualization application to indicate an intention to purchase a product. In response, the visualization application can generate an event, which can be handled by the script. The script can configure client system <NUM>, in response to the event, to initiate a purchase or add the product to a shopping cart through the webpage. Similarly the script can configure client system <NUM> to set cookies in response to events generated by the visualization application. These cookies can then be read by host system <NUM>.

Host system <NUM> can be configured to provide the webpage to client system <NUM>. Host system <NUM> can be or include a computing system, such as a desktop, workstation, server, server cluster, cloud computing environment (e.g., Amazon Web Services™ (AWS), Microsoft Azure™, IBM Cloud™, Google Cloud Platform™, Cisco Metapod™, Joyent™, vmWare™, or the like), or other suitable computing system. Host system <NUM> can be configured as a web server. For example, host system <NUM> can provide one or more webpages to web browsers in response to requests (e.g., HTTP requests or the like). The webpages can be configured to provide single-page application functionality. For example, the webpage can include data and instructions enabling a web browser of a client to modify the webpage without reloading a new webpage from host system <NUM>. In some embodiments, host system <NUM> can be associated with a provider of the products displayed or referenced by the webpage. For example, the host system can provide a catalog of the products available for purchase from the provider (e.g., host system <NUM> can provide webpages for a consumer goods store such as Home Depot, Walmart, or the like).

Application system <NUM> can be configured to enable client system <NUM> to modify an image to include a product. Application system <NUM> can be or include a computing system, such as a desktop, workstation, server, server cluster, cloud computing environment (e.g., Amazon Web Services™ (AWS), Microsoft Azure™, IBM Cloud™, Google Cloud Platform™, Cisco Metapod™, Joyent™, vmWare™, or the like), or other suitable computing system.

Application system <NUM> can be configured to provide a script to client system <NUM>. Application system <NUM> can provide the script in response to a request transmitted by client system <NUM>. As shown in the following example, the webpage can include an element that causes client system <NUM> to request the script from application system <NUM>:.

<script type="text/javascript" src="https://host address/scriptname" async></script>.

In this example, the script tag can be included within the webpage (e.g., within a <head> tag or <body> of the webpage). The host address can be a URL pointing to a location hosted by application system <NUM> (e.g., a directory), while scriptname can be a name of the script stored by application system <NUM> at that location. In this example, the scriptname can be or include a codename associated with host system <NUM> or a product provider associated with host system <NUM>. In this example, the location hosted by application system <NUM> can include multiple differing scripts having different script names. When client system <NUM> processes the webpage, the script tag can cause client system <NUM> to request the script scriptname located at host address from application system <NUM>. Client system <NUM> can then execute the script to provide functionality as described herein.

Application system <NUM> can be configured to communicate with client system <NUM> after provision of the script. In some embodiments, application system <NUM> can receive, from client system <NUM>, an indication of one or more products displayed or referenced by the webpage. Application system <NUM> can determine whether information enabling display of each of the one or more products is available. When such information is available for a product, application system <NUM> can provide an indication of such availability to client system <NUM>.

In various embodiments, application system <NUM> can be configured to generate instructions regarding the display of a modified image including the product. These instructions can be generated in response to a request to display the modified image including the product. The request can include the image in which the product is to be displayed or a selection of an image (e.g., a previously stored or predetermined image). When an image is received (e.g., an image uploaded by the user), application system <NUM> can be configured to process the image to determine image characteristics such as a perspective of the image, to identify surfaces in the image (e.g., floors, walls, countertops, tables, or the like), to determine distances to surfaces in the image, or to obtain similar information. As described herein, such processing can be performed using a machine learning model (e.g. neural network model, or the like). When a previously stored or predetermined image is selected, application system <NUM> can be configured to obtain the image (e.g., receive the image from another system, retrieve the image from a database, computer memory, or the like, accessible to application system <NUM>, or the like).

In some embodiments, application system <NUM> can be configured to provide, to client system <NUM>, the generated instructions. In some embodiments, the instructions can include the image, or a modified version of the image. For example, the instructions can include the image modified to display the product. Additionally or alternatively, the instructions can include a model of the product (e.g., a three-dimensional model, or the like). When client system <NUM> does not have the image, for example when the request includes a selection of a stored or predetermined image, application system <NUM> can be configured to provide the image, in addition to the model of the product.

In some embodiments, application system <NUM> can be configured to provide, to client system <NUM>, updated instructions in response to user interactions with the product. For example, a user can interact with client system <NUM> to request repositioning or reorientation of the product in the image. Client system <NUM> can be configured to provide the request to application system <NUM>, which can generate instructions to reposition or reorient the product in the image. For example, the instructions can include an updated version of the image with the product repositioned or reoriented in accordance with the request provided by client system <NUM>.

Network <NUM> can facilitate communication and sharing of information between client system <NUM>, host system <NUM>, and application system <NUM>. Network <NUM> may be any type of network that provides communications, exchanges information, and/or facilitates the exchange of information. For example, network <NUM> may be the Internet, a Local Area Network, a cellular network, a public switched telephone network ("PSTN"), or other suitable connection(s) that enables transmission of information between the components of system <NUM>. Network <NUM> may support a variety of electronic messaging formats and may further support a variety of services and applications for communicating between client system <NUM> and host system <NUM>, or client system <NUM> and application system <NUM>.

<FIG> depicts a method <NUM> for initializing a visualization application, in accordance with disclosed embodiments. A webpage received from host system <NUM> and displayed by client system <NUM> can be modified using data and instructions received from application system <NUM> to display the visualization application. In this manner, the visualization application can be integrated into webpage provided by a host website, allowing the user to visualize products without having to leave or reload the website. In this manner, the visualization application can modify a webpage (which need not originally support single-page application functionality) to provide single-page application functionality. Furthermore, method <NUM> can use a script to automate integration of the visualization application into the webpage. By using such a script, maintenance and development of the visualization functionality can be separated from maintenance and development of the webpage. In some embodiments, integration of the visualization functionality into the webpage may then require minimal modification of the webpage (e.g., adding a script tag, as described herein).

In step <NUM>, client system <NUM> can obtain a webpage from host system <NUM>. Client system <NUM> can obtain the webpage through a network connection established with host system <NUM> according to network protocol (e.g., using an HTTP request, or the like). Host system <NUM> can provide the webpage in response to a request received from client system <NUM>. The disclosed embodiments are not limited to a particular method of obtaining the webpage from host system <NUM>.

The webpage can include code and instructions (e.g., Hypertext Markup Language or the like) for displaying content in a browser of client system <NUM>. In some embodiments, the webpage can include product information for at least one product. For example, the webpage can be a product page providing detailed product information about a single product, or a catalog page including product information about each of multiple products. As a non-limiting example, the product information can include product name, product brand, product category, model numbers, serial numbers, stock keeping units (SKUs), product color options, pricing information, product dimensions, product availability and inventory information, product installation information, or similar information about the product.

The webpage can include vendor product code information usable to associate a product with visualization information for the product. In some embodiments, vendor product code information can be scrapable from the webpage. In various embodiments, the visualization information can be associated with an identifier for the product. The identifier may be unique for a combination of product and host system <NUM> (or product provider associated with host system <NUM>), or unique for a combination of product, product type, and host system <NUM> (or product provider associated with host system <NUM>).

Vendor product code information included in the webpage can be used to determine, create, or recover the identifier, enabling retrieval of the visualization information. For example, a stock keeping unit (SKU) number or Universal Product Code (UPC) for a product can serve as an identifier, associating the product with visualization information. The webpage can include the SKU number for the product. As an additional example, an identifier can be created (e.g., using one or more string operations) from two or more items of product information for a product (e.g., by concatenating the product name, product brand, and product color; or the like). The webpage can include the two or more items of product information for a product, enabling the identifier to be recreated by scraping the two or more items of product information from the webpage.

In step <NUM>, client system <NUM> can obtain a script from application system <NUM>. Client system <NUM> can obtain the script through a network connection established with application system <NUM> according to network protocol (e.g., using an HTTP request, or the like). The disclosed embodiments are not limited to a particular method of obtaining the webpage from application system <NUM>.

In some embodiments, application system <NUM> can provide the script in response to a request received from client system <NUM>. Client system <NUM> can generate the request while processing the webpage. For example, the webpage can include instructions (e.g., an HTML <script> tag) that, when parsed or executed by client system <NUM>, cause client system <NUM> to request the script from application system <NUM>. In some embodiments, in addition to the location of the script, the instructions can indicate a type, language, or version of the script (e.g., JavaScript or another suitable scripting language), or other information used in parsing or executing the script by client system <NUM>. The disclosed embodiments are not limited to a particular scripting language or script format.

In some embodiments, the script can be specific to at least one of the webpage, host system <NUM>, a product provider associated with host system <NUM>, or a product type or category. In various embodiments, application system <NUM> can provide a previously generated script. In some embodiments, application system <NUM> can provide a script generated in response to the request. A script generated in response to the request can be specifically tailored to each request by client system <NUM>.

Although depicted in <FIG> as two separate steps, in some embodiments, steps <NUM> and <NUM> can be performed as one step. For example, host system <NUM> can store a cached copy of the script, mirroring the script stored in the application system <NUM>, and transmit the copy of the script to client system <NUM> with the webpage or after the website is transmitted.

In step <NUM>, according to the script obtained in step <NUM>, client system <NUM> can obtain identifiers for each of any products described or referenced in the webpage. In some embodiments, the script can configure client system <NUM> to scrape or parse the webpage for vendor product code information. For example, client system <NUM> can be configured to extract vendor product code information, as described herein, from a Document Object Model (DOM) of the webpage. In some embodiments, the vendor product code information can be stored in a predetermined location in the webpage (e.g., at a known location within a particular document element, within a hidden or displayed form element, within an element with a known name or id, or the like). In some embodiments, the vendor product code information can be extracted from the URL of the webpage. The script, which can be specific to the webpage, can cause client system <NUM> to extract the vendor product code information from the predetermined location in the webpage. In various embodiments, the script can extract the product information by searching the source code of the webpage for text with predetermined characteristics (e.g., product names, descriptions, colors, serial numbers, model numbers, or the like). In some embodiments, such searching can be performed using regular expressions or the like. The disclosed embodiments are not limited to embodiments in which client system <NUM> obtains all identifiers for all products described or referenced in the webpage. As would be appreciated by those of skill in the art, client system <NUM> may only attempt or be able to obtain identifiers for each of a subset of the products described or referenced in the webpage.

In some embodiments, client system <NUM> can be configured by the script to generate the identifier from the vendor product code information. For example, the identifier can be generated by applying one or more string operations to items of the vendor product code information (e.g., concatenating product name, brand, model number, and color to generate an identifier for the product).

In some embodiments, the product information can be included in the URL requesting the script as a parameter or part of the address. In these instances, the product information may have been transmitted to application system <NUM> prior to receipt of the script by client system <NUM>. Application system <NUM> can then transmit a script that has been tailored to match the product.

In some embodiments, client system <NUM> can be configured by the script to manage user identity or session information. For example, in some embodiments, client system <NUM> can be configured by the script to place or check cookies identifying the user or establishing a session. To continue this example, when client system <NUM> lacks a current user session cookie, client system <NUM> can be configured by the script to generate such a cookie (e.g., by requesting a session or user identifier from application system <NUM> and storing the obtained information in the cookie). This cookie may be, or may be treated as, a first party cookie by client system <NUM>. For example, the cookie can be associated with a domain of host system <NUM>. Accordingly, this cookie may not be blocked by ad blockers or third-party cookie blockers. If a current user session cookie does exist in client system <NUM>, but application system <NUM> does not recognize a corresponding session, client system <NUM> can be configured by the script to request application system <NUM> create a new user session associated with the user or with client system <NUM>.

In step <NUM>, according to the script obtained in step <NUM>, client system <NUM> can query application system <NUM> for visualization information corresponding to the identifier(s) obtained in step <NUM>. The query, and the response to the query, may not cause client system <NUM> to reload the webpage. For example, the query can be transmitted to application system <NUM> using Asynchronous JavaScript + XML (AJAX) techniques, or the like, and the data transmitted with the query can be organized in the form of JavaScript Object Notation (JSON), or the like. As an additional example, the query can be as simple as a URL pointing to a predetermined location in application system <NUM>. The visualization information may be present at that location.

In response to the query, application system <NUM> can determine whether visualization information corresponding to the identifier(s) is available. In some embodiments, application system <NUM> can compare the identifier(s) to a list or database of products having visualization information. Application system <NUM> can transmit to client system <NUM> an indication of whether visualization information corresponding to the identifier(s) is available.

In step <NUM>, when visualization information is available for a product and according to the script obtained in step <NUM>, client system <NUM> can modify the webpage to display a control. The control can be an interactive element in the webpage, for example, a button, a toggle, or the like. In some embodiments, the control may have been included in the webpage received from host system <NUM>, but may be inactive (e.g., disabled, hidden, or the like). When the query indicates that visualization information is available for the product, client system <NUM> can modify the webpage to activate the control (e.g., permitting the control to be interacted with by a user). In some embodiments, the control may not have been included in the webpage received from host system <NUM>. The control may then be added to the webpage. For example, client system <NUM> can be configured by the script to modify the webpage to include a selectable button at a location associated with the product. As the script may be specific to the webpage or to host system <NUM>, the modification can preserve the look and feel of the webpage. Thus the control can be displayed in a manner that appears natural to the user and without disrupting the user experience.

In step <NUM>, when visualization information is available for a product and according to the script obtained in step <NUM>, client system <NUM> can retrieve a visualization application from application system <NUM>. In some embodiments, client system <NUM> can retrieve a visualization application in response to selection of the control (e.g., by a user interaction) in the modified webpage. For example, selection of the control can trigger an event. The script can handle the event, causing client system <NUM> to retrieve the visualization application from application system <NUM>. For example, when the webpage is modified to display a button control, a selection of the button by a user can cause client system <NUM> to retrieve the visualization application.

In step <NUM>, when visualization information is available for a product and according to the script obtained in step <NUM>, client system <NUM> can modify the webpage to display the visualization application. The visualization application can enable the user to select an image and modify the image to display the product, as described herein. In some embodiments, client system <NUM> can be configured by the script to modify the webpage to include a container, such as an iframe, or the like. Client system <NUM> can then load the retrieved visualization application into the container. The disclosed embodiments are not limited to use of a container. Other modifications resulting in display of the visualization application are also envisioned, including addition of a layer over the existing webpage, opening a new tab or a new window to display the visualization application, or removing or rearranging one or more webpage elements to create space for displaying the visualization application.

The operations of method <NUM> have been labeled and described sequentially, for convenient description. As would be appreciated by those of skill in the art, the indicated sequence is not intended to be limiting. Steps may be rearranged or combined, or additional steps may be added. For example, in some embodiments, a single operation can include querying product availability (step <NUM>) and obtaining the script (step <NUM>). For example, the webpage obtained in step <NUM> can contain an URL pointing to the location of the script and at least one parameter can be combined with the URL, in effect querying availability of visualization information for the product. Similarly, the visualization application can be retrieved prior to selection of the control in the modified webpage. The visualization application may not be integrated into the webpage or may be integrated into the webpage but set to "invisible" or "inactive," until the control is selected.

<FIG> depicts a method <NUM> for interacting with a visualization application, in accordance with disclosed embodiments. Such interactions can occur after modification of the webpage to display the visualization application. In some embodiments, a user can interact with the visualization application (e.g., through a graphical user interface of client system <NUM>) to provide an image. The image can then be modified to include a product. The user may then further interact with the visualization application to modify the display of the product in the image (e.g., by modifying the object's placement, orientation, or the like).

In step <NUM>, a user can interact with client system <NUM> to select an image. The image can be of a location (indoor or outdoor) where the product can be placed. As a non-limiting example, the image can be of a room. The user can interact with a graphical user interface provided by the visualization application to select the image. In some embodiments, the graphical user interface may provide options for uploading an image. For example, the visualization application may provide the option to upload images from a memory accessible to client system <NUM>. As a further example, the visualization application may provide the option to acquire an image and upload the acquired image (e.g., visualization application may provide an option to access a camera of client system <NUM> or communicatively connected to client system <NUM>, to capture an image for uploading). To continue this example, when client system <NUM> is a smartphone, the visualization application can be used to obtain an image using the camera of the smartphone. In various embodiments, the graphical user interface of the visualization application may provide options for selecting an existing image (e.g., an image previously saved or uploaded by the user, or a default image). In some embodiments, client system <NUM> can be configured to communicate with application system <NUM> to obtain the image options for selection. For example, application system <NUM> can retrieve the image options and provide them to client system <NUM> for display. As an additional example, application system <NUM> can provide information enabling client system <NUM> to retrieve image options from another location. The image can be a JPG, GIF, Bitmap, PNG, or other suitable format. The disclosed embodiments are not limited to any particular standard or format of the image. An exemplary graphical user interface is described herein with regards to <FIG>.

As described above, in some embodiments the user can cause client system <NUM> to display the visualization application by selecting a control associated with a product. The visualization application can be configured to display this product in the image. However, in some embodiments, the user can interact with the visualization application to select a product for placement in the image in addition to selecting the image. For example, the visualization application can display products available for visualization to the user and the user can interact with the visualization application to select a product. In such embodiments, the visualization application can display a thumbnail of each product and other information that may help the user to determine which product to select, (e.g. price, material, rate, etc.).

In step <NUM>, client system <NUM> can provide a request to display the selected product in the selected image to application system <NUM>. The request can be provided using network <NUM>. The disclosed embodiments are not limited to any particular method or protocol for providing the request. In some embodiments, when the image is uploaded by client system <NUM>, the request can include the image. In various embodiments, when an image option is selected, the request may include the image or an indication of the image. In some embodiments, client system <NUM> can convert the selected image into a predetermined format before transmitting it to application system <NUM>, while in some other embodiments, client system <NUM> can transmit the selected image to application system <NUM> without converting the image. The disclosed embodiments are not limited to a particular format for transferring the image. In some embodiments, the request can indicate the selected product. In various embodiments, the request can include metadata concerning the image. For example, depth information can be associated with the image. In some embodiments, client system <NUM> can capture the depth information using one or more images from multiple cameras or multiple images from the same camera. The disclosed embodiments are not limited to a particular method of acquiring such depth information.

In step <NUM>, application system <NUM> can generate instructions for modifying the selected image to display the selected product. In some embodiments, application system <NUM> can generate instructions enabling client system <NUM> to update the display of the product in the selected image in response to user input. In such embodiments, the instructions can include a product model and information enabling display of the product in the selected image. For example, when the product is a lamp, the instructions can include a three-dimensional model (3D) of the lamp and information specifying the perspective of the image and the distance from a viewpoint of the image to one or more surfaces depicted in the image. Such information could enable client system <NUM> to scale and rotate the 3D model of the lamp for correct placement on the one or more surfaces depicted in the image. This may help to ensure that the perspective and/or scaling of the product displayed in the selected image is consistent with the perspective and scaling of the selected image.

In various embodiments, the instructions received from application system <NUM> may not enable client system <NUM> to update the display of the product in the selected image in response to user input. Instead, client system <NUM> may request updates to the modified image from application system <NUM>. In some embodiments, such instructions can include a version of the selected image modified to display the product in a particular position and orientation.

In some embodiments, generating the instructions can include creating a 3D model from the image. The 3D model provides a model of the scene captured in the image, including objects present in the image. This information enables the perspective and scaling of the image and all objects therein to be varied. Application system <NUM> can be configured to generate the 3D model using a machine learning model, such as a convolutional neural network. The machine learning model can be configured to estimate depth and perspective information for the image. For example, the machine learning model can be configured to estimate a viewpoint for the image and a distance from the viewpoint to a depicted surface in the image or pixel in the image. In various embodiments, when the image depicts a room, the machine learning model can be configured to determine a shape of the room (e.g., the location of the walls, floor, or ceiling in the image, with respect to the viewpoint or to each other, or the like). In some embodiments, when the request received from client system <NUM> includes depth information, Application system <NUM> can be configured to use such depth information when creating the 3D model.

In some embodiments, application system <NUM> can be configured to perform semantic segmentation of the image. Such semantic segmentation can identify portions of the image relevant to placement of a product in the image. Semantic segmentation may be used to identify image pixels associated with a shared object represented in the image, thereby enabling objects represented in the image to be identified. As a non-limiting example, semantic segmentation can be used to identify floors, walls, countertops, items of furniture (e.g., tables, bookcases, shelving), or the like in the image. On the basis of the identified objects represented in the image, portions of the image may be designated for placement of the product in the image. In some embodiments, the semantic segmentation can be performed using, in addition to the image, the type of the product or the 3D model generated from the image. For example, when the product is a type of flooring, this product type can inform the semantic segmentation (e.g., through selection of a machine learning model for performing the segmentation or as an input to the machine learning model that performs the segmentation). Similarly, the 3D model can inform the semantic segmentation (e.g., the 3D model can indicate surfaces that may potentially be part of a floor). This may help to reduce processing overhead, enabling a targeted semantic segmentation of the image to be carried out, in which the image is analyzed for the presence of specific objects. The type of product selected for visualization may be used to identify one or more objects present in the image required by the product to provide its function. For example, where the selected product for visualization relates to a type of flooring, the semantic segmentation may be restricted to identifying image objects corresponding to a floor in the image. The floor being essential to the flooring's function.

Application system <NUM> can use one or more machine learning models to perform the semantic segmentation of the image. For example, Application system <NUM> can perform semantic segmentation using a convolutional neural network built on an AlexNet, VGG-<NUM>, GoogLeNet, ResNet, or other suitable architecture. The convolutional neural network can be trained to perform region-based semantic segmentation, fully convolutional network-based semantic segmentation, or to perform another suitable method of semantic segmentation.

In some embodiments, application system <NUM> can be configured to use multiple machine learning models to perform semantic segmentation of the image. The multiple machine learning models can be trained to identify differing portions of the image. For example, one machine learning model can be trained to identify floors in the image, while another machine learning model can be trained to identify walls in the image. Application system <NUM> can be configured to determine which of the multiple machine learning models to use based on a type of the product. For example, when the product is a wall clock, application system <NUM> can perform semantic segmentation on the image using a machine learning model trained to identify walls in the image. As an additional example, when the product is flooring, application system <NUM> can perform semantic segmentation on the image using a machine learning model trained to identify floors in the image.

The disclosed embodiments are not limited to embodiments using machine learning models trained to identify a single semantic class. For example, the disclosed embodiments can be configured to use a machine learning model trained to identify walls, floors, and counters in the image, or more generally identify surfaces in the image.

In some embodiments, application system <NUM> can be configured to store the image and the 3D model of the image. Should the user later want to place another product in the stored image, the stored image and model can be retrieved and reused.

Application system <NUM> can be configured to maintain a data storage subsystem (e.g. a database) storing product models. In response to a request from client system <NUM> indicating a selected product, application system <NUM> can be configured to retrieve a model for the selected product. The model of the selected product can include spatial information and surface detail information. The spatial information can specify the dimensions of the product. In some embodiments, such spatial information can include a mathematical representation of one or more surfaces of a product. In various embodiments, the spatial information can comprise a 2D or 3D model of the product. For example, a 3D model of a lamp can specify the location of points on the surface of the lamp in a three-dimensional space. As a further example, a 2D model of a poster can specify the height and width of the poster. Surface detail information can include textures, colors, patterns, or the like. For example, a product can be associated with a texture mapping, which can be applied to a 2D or 3D model of the product, or to a surface in the 3D model of the image. In various embodiments, some products may not be associated with a spatial model. For example, in some embodiments, products lacking predetermined dimensions, such as flooring, molding, paneling, wallpaper, countertops, or the like, may be associated with surface detail information but not spatial information.

Application system <NUM> can, in some embodiments, be configured to retrieve a product model for the selected product and include the product model in the instructions. Application system <NUM> can alternatively or additionally modify the selected image using the product model and then include the modified image in the instructions.

Application system <NUM> can modify the selected image by rendering the product model at a particular location in the selected image. Applicant system <NUM> can be configured to use a suitable rendering method, such as polygon-based rendering, scanline rendering, ray tracing, rasterization, or the like. Such rendering can include determining pixels in the selected image obscured by the rendered product model and replacing these pixels with corresponding pixels of the rendered product model. For example, application system <NUM> can determine that a wall clock may cover an area of the wall and replace the pixels in the selected image with corresponding pixels in a rendering of the wall clock. In various embodiments, the particular location can be determined based on information included in the request. For example, the user can specify a location for a product (e.g., a lamp) in the image (e.g., on a table depicted in the image). In some embodiments, the particular location can be determined by application system <NUM>. For example, application system <NUM> can determine suitable locations in the image based on semantic segmentation, as described herein, and a type of the product. In this example, application system <NUM> may determine that the product is a lamp and that the image includes a surface suitable for placement of a lamp (e.g., a tabletop, countertop, shelf, or the like). Application system <NUM> may then render the lamp on the suitable surface.

Application system <NUM> can modify the selected image by painting an image of the product on suitable portions of the selected image, using surface detail information for the product and a 3D model of the image. As described above, application system <NUM> can perform semantic segmentation on the image to associate classes with pixels in the image. Within the present context, class may be used to designate the type of object present in the image. In some instances, when a type of the product matches one of the classes, an image of the product generated using the surface detail information can be painted over pixels associated with that class. For example, when the product is a type of flooring, application system <NUM> can replace portions of the selected image associated with the class "floor" with corresponding texture-mapped portions of the flooring.

In step <NUM>, application system <NUM> provides the instructions generated in step <NUM> to client system <NUM>. The instructions can be provided using network <NUM>. The disclosed embodiments are not limited to any particular method or protocol for providing the instructions. In some embodiments, the instructions may include a modified image displaying the selected product. In various embodiments, the instructions may include the selected image (e.g., when the request in step <NUM> indicated an image option but did not include an uploaded image) and a rendered version of a product model (e.g., a rendered version of the product model, appropriately scaled and oriented such that it can be combined with the image, together with information enabling the client system to combine the images). In some embodiments, the instructions can include a product model for the selected product, together with instructions for rendering the product model and displaying the rendered product model in the selected image. For example, the instructions can include a 3D model of the selected image and results of semantic segmentation of the selected image (e.g. associations between pixels in the selected image).

In step <NUM>, client system <NUM> can be configured to display the selected image, modified to include the selected product, using the received instructions. When the instructions include a modified image displaying the selected product, client system <NUM> can be configured to display the modified image. When the instructions include a rendered product model and the selected image, client system <NUM> can be configured to combine the rendered product model and the selected image to generate the modified image. Client system <NUM> can then display the modified image. When the instructions include the product model and instructions for rendering the product model and displaying the rendered product model in the selected image, client system <NUM> can perform such rendering and display the resulting modified image. For example, the instructions include a 3D spatial model of the product, surface detail information for the product, and a 3D model of the image. The 3D model of the image can include perspective information and depth information for surfaces in the image. Client system <NUM> can be configured to use the perspective and 3D model of the image to render the product in the image at locations indicated by the user.

Optionally, in step <NUM>, the client system <NUM> can be configured to update the modified image based on user interactions with the visualization application. Preferably, the user can interact with the modified image to adjust the location and orientation of the displayed product in the image, without such interaction requiring loading of the webpage. For example, the user can rotate the product and move the product to place the product at a different location. Preferably, client system <NUM> can limit the range within which the product can be placed depending on the type of the product and the classification of pixels in the selected image. The method may comprise identifying pixels in the selected image having a classification associated with the type of product, and restricting placement of the product within the image to locations corresponding with the identified pixels having the classification associated with the type of product. For example, the user can move a carpet within the range of pixels identified as representing floors. In some embodiments, client system <NUM> can re-render the product image or / and re-combine the product image with the selected image as described above. In some embodiments, client system <NUM> can transmit the user interaction to application system <NUM> and request display of the product in the room image with updated instructions. In these cases, the system can repeat steps <NUM> to <NUM>.

<FIG> depicts an exemplary client system <NUM> implementing an embodiment of the current invention. <FIG> illustrates the interface of client system <NUM> when visiting host system <NUM>. For the purpose of the illustration, host system <NUM> is a website for a home supplies store. Client system <NUM>, as described above, can be any computing device with a display. For the purpose of illustration, client system <NUM> in this example is a smartphone with a camera. Client system <NUM> requests a webpage from the host system <NUM> (e.g. a website of the home supplies store) and displays the webpage to the user. The webpage can list a group of categories of products (e.g. floors, wallpapers, furniture, etc.). Each of the listed categories is associated with a button labeled with the name of the category, which, if clicked, causes client system <NUM> to request the webpage depicted in <FIG>.

<FIG> depicts an exemplary webpage retrieved from host system <NUM>. This webpage shows a list of products. Similar to <FIG>, <FIG> depicts an exemplary webpage retrieved from host system <NUM>. Unlike <FIG>, <FIG> depicts product information for a single product. During loading of the webpages depicted in <FIG> and <FIG>, client system <NUM> requested a script from application system <NUM>. As described above with regards to <FIG>, the script has executed, causing client system <NUM> to identify the products listed in each of the webpages. Client system <NUM> has communicated with application system <NUM> to determine that visualization information for the products is available. Accordingly, each webpage has been modified to include "See in My Room" buttons beneath each product. For each webpage, the user can click the button to start the visualization application.

<FIG> and <FIG> depict interactions with a visualization application following selection of a "See in My Room" button beneath a product. In <FIG>, the webpage has been modified to include the visualization application. The visualization application is providing an interface for selecting either a predetermined image (e.g. "cozy space") or choosing to upload an image (e.g., "Take Photo of My Room"). When the user selects uploading an image, client system <NUM> can operate the camera of client system <NUM> to enable the user to take a photo of the room. In some embodiments, although not depicted in the <FIG> and <FIG>, the user can select an image previously stored by application system <NUM> or an image previously obtained by client system <NUM>. After the photo is selected or confirmed, client system <NUM> can transmit the photo or an indication of a selected photo to application system <NUM>, which, in return, generates instructions and transmits them to client system <NUM> as described above with regards to <FIG>.

<FIG> depicts display, by the visualization application, of a modified image displaying a product. In this non-limiting example, the floor in the selected image (depicted in <FIG>) has been replaced with an image of a flooring product (the hardwood floor depicted in <FIG>). As described herein, the "floor" pixels in the selected image have been identified through semantic segmentation. The surface detail information (e.g., the pattern, color, and texture of the flooring product) of the product have been texture mapped to the identified pixels.

<FIG> depicts an interface for updating the modified image. In <FIG>, a list of thumbnails of suggested products is displayed in the user interface. In some embodiments, client system <NUM> can be configured to store a list of such products. Alternatively, client system <NUM> can query application system <NUM> to populate the list of products. Although <FIG> depicts the thumbnails at the bottom of the display, the thumbnails can be placed at any portion of the display. As depicted in <FIG>, the user can search for a product by entering text into client system <NUM>. As depicted in <FIG>, the user can apply one or more filters to limit the number of products being displayed. In some embodiments, client system <NUM> can search or filter the product locally. In some embodiments, client system <NUM> can transmit the search request or the filtering request to application system <NUM> and receive from application system <NUM> an updated product list. Following selection of a new product, the modified image can be updated to display the new product. Such updating can include providing, by client system <NUM> to application system <NUM>, a request to display the new product in the selected image. Client system <NUM> can receive instructions for displaying the new product in the selected image in response.

<FIG> depicts interactive modification to the display of a product in an image. In this non-limiting example, the product is a rug. In <FIG>, the rug is displayed in a first position on the floor of the room. The view of the rug is adjusted to match the perspective and viewpoint of the image. Furthermore, the existing ottoman is displayed as being on top of the rug. In <FIG>, the visualization application provides instructions on changing the position and orientation of the rug in the image using a set of pre-defined touch screen gestures (e.g., press and drag to move the object around the room and press and turn with two fingers to rotate the object). In some embodiments, the user can interact with the object using a mouse, a keyboard, or any other available input device. As depicted in <FIG>, in response with the user interaction, the displayed position and orientation of the object can be updated. As described above with regards to <FIG>, such updating can be performed locally by client system <NUM>, using a 3D model of the image and a product model, performed remotely at application system <NUM>, or performed in part locally and in part remotely. For example, client system <NUM> can transmit the user interactions to application system <NUM>, which can modify the images and transmit the modified image back to client system <NUM>. Alternatively, client system <NUM> can transmit the product selection to application system <NUM> and receive a product model and instructions for positioning the product in the selected image. Client system <NUM> can then render the product model, combine the rendered product model with the selected image, and display the resulting modified image to the user without communicating with the Application System <NUM>.

<FIG> depicts a schematic of exemplary computing system <NUM> for performing the envisioned systems and methods, consistent with disclosed embodiments. In some embodiments, computing system <NUM> can include a processor <NUM>, memory <NUM>, display <NUM>, I/O interface(s) <NUM>, and network adapter <NUM>. These units may communicate with each other via bus <NUM>, or wirelessly. The components shown in <FIG> may reside in a single device or multiple devices.

Consistent with disclosed embodiments, processor <NUM> may comprise a central processing unit (CPU), graphical processing unit (GPU), or similar microprocessor having one or more processing cores. Computing system <NUM> may include one or more processors <NUM> and may further operate with one or more other processors that are remote with respect to processors <NUM>. Memory <NUM> may include non-transitory memory containing non-transitory instructions, such as a computer hard disk, random access memory (RAM), removable storage, or remote computer storage. In some aspects, memory <NUM> may be configured to store data and instructions, such as software programs. For example, memory <NUM> may be configured to store data and instructions. In some aspects, processor <NUM> may be configured to execute non-transitory instructions and/or programs stored on memory <NUM> to configure computing system <NUM> to perform operations of the disclosed systems and methods. In various aspects, as would be recognized by one of skill in the art, processor <NUM> may be configured to execute non-transitory instructions and/or programs stored on a remote memory to perform operations of the disclosed systems and methods.

Display <NUM> may be any device which provides a visual output, for example, a computer monitor, an LCD screen, etc. I/O interfaces <NUM> may include hardware and/or a combination of hardware and software for communicating information to computing system <NUM> from a user of computing system <NUM>, such as a keyboard, mouse, trackball, audio input device, touch screen, infrared input interface, or similar device. Network adapter <NUM> may include hardware and/or a combination of hardware and software for enabling computing system <NUM> to exchange information using external networks, such as network <NUM>. For example, network adapter <NUM> may include a wireless wide area network (WWAN) adapter, a Bluetooth module, a near field communication module, or a local area network (LAN) adapter.

<FIG> depicts exemplary method <NUM> for visualization of products in a display interface, in accordance with disclosed embodiments. As depicted in <FIG> and discussed herein, the disclosed embodiments are not limited to displaying products in an image using a visualization application. In some embodiments, application system <NUM> can provide graphic resources, as described herein, to client system <NUM> for use by native tool <NUM> in displaying the product in the image data. A user can then interact with the client system <NUM> to modify the display of the product in the image (e.g., by modifying the product's placement, orientation, or the like). In such embodiments, client system <NUM> need not obtain the visualization application. Furthermore, client system <NUM> need not provide an image to application system <NUM>, a simplification that may improve the user experience. Client system <NUM> may be able to display the product in the image data as the image data is acquired, further improving the user experience. In some instances, for example, a user of client system <NUM> may be empowered to move around an environment and view a virtual image of the product from different angles or perspectives in the environment. Additionally, a more accurate representation can be provided, as the image data may provide more information about the environment than a single image. Furthermore, native tool <NUM> may have access to additional sources of data (e.g., accelerometers in client device <NUM>, or the like) for determining the layout of the environment, or may be more powerful than a practicably downloadable visualization application.

Consistent with disclosed embodiments, the image data can include pictures, video data, a live stream, or the like. The image data can be obtained by client system <NUM> from an imaging device (e.g., a digital camera capable of capturing photographs, recording video, providing the live stream of images, or the like). In some embodiments, the image data can be retrieved from a memory accessible to client system <NUM> or received from another system or device (e.g., a security camera, camcorder, webcam, or the like). In some embodiments, method <NUM> can enable a real-time display of the product in the image data.

Native tool <NUM> may include a graphic rendering programming or engine. Native tool <NUM> can be or include one or more augmented reality tools (e.g., ARKit, ARCore, or the like). Native tool <NUM> can be incorporated into a mobile web browser (e.g., Safari, Chrome, or the like) or a mobile application (e.g., an app) of client system <NUM>. Native tool <NUM> can use the graphic resources received by client system <NUM> from application system <NUM> to generate augmented reality (AR) or virtual reality (VR) media displaying the product in image data acquired by client system <NUM>. In some embodiments, native tool <NUM> can be configured to display the augmented reality (AR) or virtual reality (VR) media in an application embedded in a webpage (e.g. the webpage containing the control selected by the user to start the visualization). In some embodiments, native tool <NUM> can be configured to display the augmented reality (AR) or virtual reality (VR) media in a separate application (e.g., separate from the webpage containing the control selected by the user to start the visualization).

In some such embodiments, client system <NUM> can be or include a computing device such as a terminal or interactive kiosk machine. The terminal or interactive kiosk machine can be configured to support native tool <NUM>.

In some embodiments, method <NUM> can include steps of obtaining a webpage (e.g., from host system <NUM>), obtaining a script (e.g., from application system <NUM>), obtaining identifiers for each of at least some of the products described or referenced in the webpage, and querying for product availability. Such steps can be performed as discussed herein with respect to steps <NUM> to <NUM> of <FIG>. In some embodiments, when visualization information is available for a product and according to the obtained script, client system <NUM> can modify the webpage to display a control for the product. As described above with regards to step <NUM> of method <NUM>, the control can be an interactive element. Displaying the control may include activating the control.

In step <NUM> of method <NUM>, a user can interact with the modified webpage to select a product (e.g., by selecting the control corresponding to the product). The disclosed embodiments are not limited to any type of interaction. In some embodiments, the user can interact with the modified webpage as described above with regards to step <NUM> of method <NUM>.

In step <NUM>, in response to the interaction in step <NUM>, client system <NUM> can obtain graphic resources from application system <NUM>. In some embodiments, client system <NUM> can provide a request to application system <NUM>. In some embodiments, the request can be provided using network <NUM>. The disclosed embodiments are not limited to any particular method or protocol for providing the request. The request can indicate the product for which the graphic resources are required (e.g., it can include a value or other indicator of the product; be directed to a URL or the like specific to the product; or the like). In some embodiments, the request can provide information indicating a type, model, version, or the like of native tool <NUM>. In various embodiments, the request can specify a particular type of graphic resource.

In response to the request, application system <NUM> can provide the graphic resources to client system <NUM>. In some embodiments, application system <NUM> can retrieve the graphic resources from a memory accessible to application system <NUM>. In various embodiments, application system <NUM> can receive the graphic resources from another system (e.g., a system associated with a manufacturer, supplier, or distributor of the product, or the like). In some embodiments, application system <NUM> can create the graphic resources in response to the request. In some embodiments, application system <NUM> can provide graphic resources selected based on at least one of the selected product or the information indicating the native tool of client system <NUM>. In some embodiments, application system <NUM> can generate the graphic resources based on the received request. In some instances, for example, based on the information regarding the native tool of client system <NUM>, application system <NUM> can customize graphic resources to be compatible with the native tools of client system <NUM> (e.g., a USDZ file format when native tool <NUM> is ARKit or the like; or an OBJ/MTL file format when native tool <NUM> is ARToolkit or the like).

In step <NUM>, a browser of client system <NUM> can call or invoke native tool <NUM>. In some embodiments, the script received from application system <NUM> can cause the browser to provide commands to native tool <NUM> (e.g., using an API exposed by native tool <NUM>). The commands can, in some embodiments, include commands to acquire image data. The commands can, in various embodiments, include commands to display the product at a location in an environment depicted in the image data. The commands can include commands to start an application for displaying the product in the environment. In some embodiments the application can be embedded in the browser. In various embodiments, the application can be separate from the browser.

In step <NUM>, client system <NUM> can display the product in obtained image data. The display can be or include an AR/VR representation of the selected product. In some embodiments, the native tool of the client system <NUM> can generate the display by imposing, combining, or otherwise causing the received graphic resources to appear on or at a location in the environment. For example, a 3D representation of a spatial model of the product can appear in a camera feed in real-time, viewable on the display of the client system <NUM>. The client system <NUM> can be configured to update the AR/VR representation of the product based on user interactions with client system <NUM>. For example, the user can interact with the AR/VR representation of a displayed product to adjust the location and orientation of the displayed product in the environment. For example, the user can rotate the product and move the product to place the product at a different location in the environment. In some embodiments, the displayed portion of the environment can change when the camera of client system <NUM> shifts, moves, rotates, or otherwise alters its field of view. In such instances, the native tool of client system <NUM> can fix the displayed product at a location in the environment regardless of the changes in the field of view.

In some embodiments, client system <NUM> can display an AR/VR interface in the environment to enable the user to update the display in the captured environment. In some embodiments, the AR/VR interface can enable the user to select or update a selection of a product for display in the captured environment. In various embodiments, the AR/VR interface can enable the user to modify characteristics of the product displayed in the captured environment (e.g., modifying at least one of a color, shape, size, composition or material, or the like). The AR/VR interface can be generated to be displayed in conjunction with the environment captured in step <NUM>. For example, the native tool of client system <NUM> can generate a control interface in the live camera view displayed, enabling the user to interact with the control interface while viewing the camera feed in real-time. In some embodiments, the AR/VR interface can indicate a list of products available for display in the environment. In some instances, the available products can be represented by graphics or symbols, such as icons, alphanumeric symbols, or a small representative images (such as a thumbnail image). The list of available products can be presented in a menu-like interface (such as a drop-down menu, a scroll menu, or other similar interfaces).

The disclosed embodiments are not limited to using an identifier of a product generated from information extracted from the webpage. In some embodiments, the product to be displayed can be randomly selected by application system <NUM> based on product availability information. For example, when a user selects (e.g., clicks on) a visualization control of the visualization application, the visualization application can start with a default or a random product before receiving user selection of product (e.g., step <NUM>). As a non-limiting example, a banner on a website of host system <NUM> can display the label "Visualizer". When a user selects the banner, an instance of the visualizer can start with a default or random product. The default product can be a predetermined product chosen from among the available products. The random product can be a product chosen from among the available products by a random or pseudo-random algorithm. In some embodiments, the banner can be a selectable advertisement for the product.

The disclosed embodiments are not limited to requesting scripts associated with host system <NUM>. In some embodiments, the script may not be specifically associated with host system <NUM>, but can instead be a generic script. Application system <NUM> can adapt the generic script to host system <NUM>. For example, when a user interacts with the visualization application, the generic script can be customized to host system <NUM> based on a context associated with host system <NUM>, such as the domain of the webpage (e.g., obtained from the current URL), or the contents of the webpage. As a non-limiting example, a generic script can be provided to client system <NUM>. In some embodiments, the generic script can be configured by application system <NUM> based on a context of the webpage accessed by client device <NUM>. When executed, the configured script can modify the web page of client system <NUM> to include an interactive graphical indicator. The graphical indicator can be selectable to start an instance of the visualization application. The instance of the visualization application can provide content appropriate for the webpage, based on the configuration of the script performed by application system <NUM>. In various embodiments, the generic script can be sent to the client system <NUM>. When the visualization application is started, it can provide the contextual information to the application system <NUM>, which can the provide appropriate customization information. For example, if host system <NUM> is associated with a website for a furniture retailer, application system <NUM> can determine that the context of host system <NUM> is a furniture retailer (or is that particular furniture retailer), and a generic script configured for furniture retailers (or for that particular furniture retailer) can be provided to client system <NUM>. If host system <NUM> is associated with a website for a carpet retailer, application system <NUM> can determine that the context of host system <NUM> is a carpet retailer (or is that particular carpet retailer), and the generic script, configured for carpet retailers (or that particular carpet retailer) can be provided to client system <NUM>. In some embodiments, the generic script is a template for the script provided to client system <NUM>, and application system <NUM> generates the script based on a generic script based on the context of host system <NUM>.

The disclosed embodiments are not limited to tracking users with first party cookies. In some embodiments, local storage can be used in place of first-party cookies to track users. As described herein, tracking (e.g., using local storage or first-party cookies) can enable the visualization application to provide the user's uploaded images or otherwise provide an experience personalized to the user (or to a session), even when the user opens the visualization application from different webpages or browsers.

In some embodiments, application system <NUM> can obtain session information for a user from a local storage associated with a browser of client system <NUM>. In some embodiments, the session information can enable application system <NUM> to associate an instance of the visualization application with at least one of the user or a session. In some instances, such an association can be established without using cookies to track the user. The session information can enable application system <NUM> to make resources (e.g., uploaded images, products or product selections, or other resources) associated with the at least one of the user or prior session, available to the visualization application. Accordingly, the user can access the resources, even when engaging with the visualization application on different webpages. For example, a user might start an instance of the visualization application through a first webpage of a first service provider and upload images of rooms to application system <NUM>. The user might then exit the visualization application and access a second webpage of a second service provider. The user might start another instance of the visualization application through the second webpage. The second instance may have access to the session information on the local storage. The second instance may communicate the session information to application system <NUM>, enabling the user to visualize products in the previously uploaded images of the rooms, without having to re-upload the rooms.

In some embodiments, a notification can be forwarded by the visualization application to another system in response to detection of an event. The notification can be a network request or a JavaScript call. The other system can be an analytics system, which can be connected to the one or more of client system <NUM>, host system <NUM>, or application system <NUM>. Such an analytics system can be configured to gather, collect, process, or analyze information or data relating to operations of a computer or network system. In some instances, the analytics system can be configured to track indicia of user engagement, such as page views, clicks, interactions with the visualization application (e.g., uploading images, positioning products in an image, manipulating the position or orientation of such products), durations of interactions, or the like. The analytics system can be configured to track such indicia for a single user, aggregate such indicia across multiple users, or perform statistical or machine learning analysis of such aggregated indicia.

In some embodiments, an image uploaded by a user can be adjusted for display. Such adjustments can include changing a resolution, size, shape, brightness, or other image characteristics. The adjustments can depend on a device that will be displaying the image. The adjustments can depend on the present values of the resolution, size, shape, brightness, or other image characteristics. The adjusted image can overwrite the uploaded image, or the original image can be retained. In some instances, when client system <NUM> includes multiple devices, the uploaded image can be adjusted to appear differently on different devices of client system <NUM>. For example, when client system <NUM> includes a desk-top PC and a mobile device, the uploaded image can be adjusted depending on which device it will be displayed on. Such adjustments can include reducing image size/resolution when being displayed on the mobile device in order to reduce upload/download time. In another example, adjustments can include enhancing image details using a neural network for viewing on a larger monitor of a desk-top PC. In some embodiments, application system <NUM> can detect a device parameter of client system <NUM> (e.g., display screen size, resolution, network speed, processing speed, memory availability, etc.). Based on the detected parameter, application system <NUM> can generate a modified version of the uploaded image for display on client system <NUM>.

In some embodiments, the visualization application can be configured to display a product catalog associated with host system <NUM>. The contents of the catalog can be selected by a vendor associated with host system <NUM> from options provided by a service provider associated with application system <NUM>. For example, a service provider system (which may be application system <NUM> or another system) can provide a set of potential products (or product categories) to a vendor system (which may be host system <NUM> or another system). The potential products can be provided as a webpage, catalog, menu, or like. For example, the service provider system can host a webpage accessible to the vendor using the vendor system. In some embodiments, information for visualizing these potential products (e.g., a three dimensional model, or other such information as described herein) may have been received by the service provider from a manufacturer (or distributor, or the like) of the product. In other embodiments, the service provider may have generated such visualization information (e.g., by scanning instances of the products, or the like).

Consistent with disclosed embodiments, the service provider system can receive a selection of a subset of the potential products from the vendor system (e.g., as form data submitted by the vendor system, or the like). The disclosed embodiments are not limited to any particular method of providing selections to the service provider system.

Consistent with disclosed embodiments, when a user selects a control to start the visualization application on a webpage hosted by host system <NUM>, application system <NUM> can be configured to cause the visualization application to display one or more of the selected potential products. In some embodiments, the webpage can be modified to display the potential products in the visualization application as a catalog. In this manner, a catalog including the visualization functions described herein can be provided for a vendor without requiring extensive involvement or technical knowledge on the part of the vendor. For example, the owner of a local furniture store could interact with a distributor to select products for stocking in the furniture store. The distributor could provide a script enabling a webpage of the furniture store to display a control that starts a visualization application. The visualization application could display a catalog showing the selected products and enabling users of the webpage to display these products in images as described herein. Such functionality might otherwise be well beyond the capabilities of such a store.

In some embodiments, host system <NUM> can include an interactive kiosk machine. The interactive kiosk can allow for a QR code to be scanned to upload an image. In some instances, such an interactive kiosk machine may be located in a store, a mall, office building, or other public space. In some embodiments, the location or the machine can be associated with a retailer or a store.

In some embodiments, surfaces depicted in the uploaded image or the modified image can include landscaping, house exterior, roof, door, garage door, window, side paneling, or the like.

<FIG> depicts a method for interacting with a visualization application, in accordance with disclosed embodiments. Such interactions can occur after modification of the webpage to display the visualization application, such as step <NUM> as described previously with reference to <FIG>. In some embodiments, the visualization experience described in the present disclosure can be generalized to support multiple product types. For example, the script can determine the product types (such as flooring, furniture, or the like) based on the website content of host system <NUM>, and load the visualization experience corresponding to the determined product type. Furthermore, the visualization application can determine the desired product type based on contents of an uploaded image. For example, if only a wall is visible in the uploaded image, then only product or product type that can be applied to the wall, but not to the floor may be determined to be desirable.

In step <NUM>, a user can interact with client system <NUM> to select an image. The image can be of a location (indoor or outdoor) where the product can be placed. As a non-limiting example, the image can be of a room. In some embodiments, the graphical user interface can provide options for uploading an image as described previously.

In step <NUM>, client system <NUM> can provide a request to display the selected product in the selected image to application system <NUM>. The request can be provided using network <NUM>. The disclosed embodiments are not limited to any particular method or protocol for providing the request. In some embodiments, when the image is uploaded by client system <NUM>, the request can include the image. The request can be sent to application system <NUM> as previously described.

In step <NUM>, application system <NUM> can identify one or more locations on the uploaded imaged received in the request of step <NUM>. In some embodiments, application system <NUM> can be configured to identify one or more locations by performing semantic segmentation of the image. As described previously, semantic segmentation can be used to identify image pixels associated with a shared object represented in the image, thereby enabling objects represented in the image to be identified. As a non-limiting example, semantic segmentation can be used to identify floors, walls, countertops, items of furniture (e.g., tables, bookcases, shelving), or the like in the image. Each of these identified portions of the image can be a location. As described previously, application system <NUM> can use one or more machine learning models to perform the semantic segmentation of the image, such as using a convolutional neural network built on an AlexNet, VGG-<NUM>, GoogLeNet, ResNet, or other suitable architecture. As described previously, application system <NUM> can be configured to use multiple machine learning models to perform semantic segmentation of the image. The multiple machine learning models can be trained to identify differing portions of the image.

In step <NUM>, client system <NUM> can receive the locations identified in step <NUM>. Client system <NUM> can be configured to display the uploaded image with graphical indicators identifying the locations in the image. By way of example, <FIG> illustrates an exemplary image with identified locations for portions of the image indicated by graphical indicators, in accordance with disclosed embodiments. Image <NUM> can be an image uploaded by client system <NUM>, depicting a room including various furniture and decorations. Image <NUM> can be displayed to the user on client system <NUM>, including graphical indicators indicating locations in image <NUM>. As depicted, graphical indicator <NUM> indicates a location that is "Countertops," graphical indicator <NUM> indicates a location that is "Walls," graphical indicator <NUM> indicates a location that is "Cabinets," and graphical indicator <NUM> indicates a location that is "Floors. " Consistent with disclosed embodiments, the locations may have been identified in step <NUM>.

In step <NUM>, client system <NUM> can receive user selection of locations. In some embodiments, the user selections can be received via user interactions with the displayed image. In some instances, the user can interact with the displayed graphical indicators. In such instances, each graphical indicator can be (or trigger display of) an interactive interface capable of receiving a user selection. For example, if client system <NUM> is a mobile device having a touch screen, a user can touch the one or more of the graphical indicators to select a location on the touch screen. The disclosed embodiments are not limited to any particular type of interactive interface. Exemplary interfaces could include radio buttons, check boxes, a drop-down menu associated with the overall image (the graphical indicators being menu entries within the drop-down menu), or the like.

In step <NUM>, application system <NUM> can receive location selections from client system <NUM>. For example, when the user selects graphical indicator <NUM> on client system <NUM>, this selection can be communicated by client system <NUM> to application system <NUM>.

In step <NUM>, application system <NUM> generates instructions. In some embodiments, applications system <NUM> generates instructions to display the product in the uploaded image (e.g., image <NUM>) at the selected location. The image of product may be rendered in manner described previously with reference to <FIG>.

In some embodiments, application system <NUM> can identify a product type corresponding to the selected location. For example, when user selects graphical indicator <NUM>, application system <NUM> generates instructions for displaying products that are relevant to "Floors," such as tiles, carpets, etc. to the location corresponding to graphical indicator <NUM>. In some embodiments, the particular product to be displayed can be a product deemed relevant to the user of client system <NUM>. For example, if the user has browsed website of host system <NUM>, the last browsed item having a product type corresponding to the selected location (e.g., carpet when floor is selected) can be the product displayed. In another example, if the user has saved, tagged, or otherwise indicated an interest in a particular product, that particular product can be selected to be displayed on the selected location if the product type matches the selected location.

Optionally, in some embodiments, application system <NUM> can, after identifying locations in step <NUM>, generate instructions to display products at the identified locations. The products displayed can be products that match the product type of the identified locations. For example, a product having the type "floor" can be displayed over the floor segment of the image, while a product having the type "counter" can be displayed over the counter segment of the image.

Optionally, in some embodiments, after receiving location selections in step <NUM>, application system <NUM> can provide instructions to client system <NUM> to indicate products available for display at the one or more selected locations. For example, the user can select "floor" and receive instructions to display a menu of flooring options. The disclosed embodiments are not limited to any particular way of displaying such available products. They may be displayed within the interactive application, in other window, or the like. The user can interact with client system <NUM> to specify a particular product(s) for display at the one of more selected locations. Application system <NUM> can receive an indication of such user interactions and generate instructions to display the indicated product(s) at the one or more selected locations.

In step <NUM>, client system <NUM> receives instructions to display products at the selection locations. In step <NUM>, the visualization application displays the products on the selected locations according to the received instructions on client system <NUM>.

The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to precise forms or embodiments disclosed. Modifications and adaptations of the embodiments will be apparent from consideration of the specification and practice of the disclosed embodiments. For example, the described implementations include hardware, but systems and methods consistent with the present disclosure can be implemented with hardware and software. In addition, while certain components have been described as being coupled to one another, such components may be integrated with one another or distributed in any suitable fashion.

Moreover, while illustrative embodiments have been described herein, the scope includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations or alterations based on the present disclosure. The elements in the claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application, which examples are to be construed as nonexclusive. Further, the steps of the disclosed methods can be modified in any manner, including reordering steps or inserting or deleting steps.

The features and advantages of the disclosure are apparent from the detailed specification, and thus, it is intended that the appended claims cover all systems and methods falling within the scope of the disclosure. As used herein, the indefinite articles "a" and "an" mean "one or more. " Similarly, the use of a plural term does not necessarily denote a plurality unless it is unambiguous in the given context. Words such as "and" or "or" mean "and/or" unless specifically directed otherwise. Further, since numerous modifications and variations will readily occur from studying the present disclosure, it is not desired to limit the disclosure to the exact construction and operation illustrated and described, and accordingly, all suitable modifications may be resorted to, falling within the scope of the disclosure.

Claim 1:
A system for visualizing products in a single-page application, comprising:
at least one processor (<NUM>); and
at least one memory (<NUM>) containing first instructions that, when executed by the at least one processor (<NUM>), cause the system to perform first operations comprising:
receiving, from a client system (<NUM>) and during rendering of a webpage received by the client system (<NUM>) in response to a first request, a second request for a script, the script associated with a host system (<NUM>) and executable by the client system (<NUM>) to perform second operations comprising:
modifying the webpage to include a visualization control;
displaying a visualization application in response to a selection of the visualization control in the webpage by a user of the client system (<NUM>);
transmitting, to an application system (<NUM>), a third request to display a product in a first image;
receiving, from the application system (<NUM>), second instructions to display the product at an identified location in the first image; and
displaying, by the visualization application, the product in accordance with the second instructions; and
providing the script to the client system (<NUM>).