Patent Description:
The following includes information that may be useful in understanding the invention. It is not an admission that any of the information specifically or implicitly referenced herein is prior art, or essential, to the described or claimed invention.

Augmented reality (AR) applications have been developed in which virtual objects are digitally overlaid on images of real-world objects. In some cases, virtual objects such as gaming characters are added to images of a scene displayed by a smartphone or tablet as a user navigates the scene. In other cases, cartoon features such as cartoon animal facial features are added to an image of a face of a user of a smartphone or tablet, the image having been captured by the camera of the smartphone or tablet. However, in these conventional AR systems, the cartoonish, or unrealistic, nature of the virtual objects allows significant leeway in the accuracy of the size, shape, orientation, and placement of the virtual objects relative to the real-world objects.

The use of AR systems in the fashion industry provides potential opportunities for enhanced marketing campaigns, customer engagement and sales. However, existing AR systems, such as Virtual try-on (VTO) systems, provide inaccurate and unrealistic sizing and fit of accessories (such as spectacles) and/or clothing relative to customers' true sizes and proportions. As a result, such systems lead to increased consumer frustration, decreased use or reliance of such systems and, ultimately, revenue loss.

Thus, there is a need to create an improved VTO system that addresses at least the above-stated issues, and will improve the customers' experience thereto.

<CIT> discloses a a try-on system to display the appearance of the user wearing a virtual wearable item.

The invention described and claimed herein has many attributes and aspects including, but not limited to, those set forth or described or referenced in this Summary. It is not intended to be all-inclusive and the invention described and claimed herein are not limited to or by the features or embodiments identified in this Summary, which is included for purposes of illustration only and not restriction.

According to some aspects of the present disclosure, a computer-implemented method is provided. The computer-implemented method includes capturing, in a client device, a first image of a user, the first image including a reference token for a user-wearable item and displaying, in the client device, images of multiple user-wearable items for the user. The computer-implemented method also includes receiving an input from the user, the input indicative of a selected user-wearable item from the user-wearable items on display and segmenting the first image to separate the reference frame from a background comprising a portion of a physiognomy of the user. The computer-implemented method also includes replacing a segment of the reference frame in the first image with an image of the selected user-wearable item in a second image of the user and displaying, in the client device, the second image of the user.

According to some aspects of the present disclosure, a computer-implemented method is provided. The computer-implemented method includes receiving, from a client device, a first image of a user, the first image including a reference token for a user-wearable item and providing for display, to the client device, images of multiple user-wearable items for the user, stored in a database. The computer-implemented method also includes receiving an input from the user, the input indicative of a selected user-wearable item from the user-wearable items on display, providing for display, in the client device, a second image of the user, the second image having a segment of the reference token replaced with an image of the selected user-wearable item, and receiving, from the client device, a request for the selected user-wearable item.

According to some aspects of the present disclosure, a system is provided that includes a memory circuit storing instructions and a processor circuit configured to execute the instructions. When executing the instructions, the processor causes the system to capture, in a client device, a first image of a user, the first image including a reference token for a user-wearable item, and to display, in the client device, images of multiple user-wearable items for the user, to receive an input from the user, the input indicative of a selected user-wearable item from the user-wearable items on display. The processor also causes the system to segment the first image to separate the reference token from a background comprising a portion of a physiognomy of the user, to replace a segment of the reference token in the first image with an image for the selected user-wearable item in a second image of the user, and to display, in the client device, the second image of the user.

The drawings are for illustration purposes only, show example non-limiting embodiments, and not necessarily drawn to scale. In the drawings:.

In the figures, like elements and steps are labeled with like numeral references unless otherwise stated in the text.

In the following detailed description, numerous specific details are set forth to provide a full understanding of the present disclosure. It will be apparent, however, to one ordinarily skilled in the art, that the embodiments of the present disclosure may be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail so as not to obscure the disclosure.

VTO systems and methods are disclosed herein. Virtual try-on systems and methods can be particularly useful for viewing how spectacles or spectacles frames would appear on the face of the user in the real world, without the user having access to the real-world spectacles or frames. In some embodiments, virtual reality (VR) displays include systems that display real images superimposed with artificially generated elements. However, it should also be appreciated that the systems and methods described herein can be applied for virtual try-on of other objects on other portions of the user's body, such as for virtual try-on of spectacles accessories, headwear, earrings, other body-piercing accessories, jewelry, clothing, or the like. Virtual try-on systems and methods as disclosed herein allow the user to visualize the way a particular real-world item would appear on the particular body of the user.

Many existing VTO solutions estimate the size of a user's face by assuming that all faces are the same size and scaling the virtual spectacle frames in proportion with the face. Thus, users with smaller- or larger-than-average heads may have the impression that a frame fits them when in reality it is too large or too small. This leads to consumer frustration when the user receives a pair of spectacles and finds out the spectacles do not look as desired, are uncomfortable, or simply do not fit the consumer. In some embodiments, a VTO system may request the user to pose with a reference object of a known size, with standardized dimensions (e.g., a credit card). For some users, this approach may seem uncomfortable or cumbersome. In some embodiments, a VTO system may use three-dimensional (3D) cameras to estimate the size of the user's face and scale the virtual glasses accordingly. However, 3D cameras may not be available for at least some users. In some scenarios, a VTO system may thus guess at how large to make a pair of virtual glasses relative to the facial features of a user in an image. This is because it is not straightforward to estimate the physical size of a user's facial features from an image, particularly when there is no scale reference in or for the image. Some VTO systems may assume that the user's head is an average size in order to determine the appropriate size of the virtual spectacles. However, an assumption of this type would generate unrealistic results for users who have heads that are smaller or larger than average.

Virtual Try-On (VTO) systems as disclosed herein use augmented reality to show a customer how a pair of spectacle frames will look on their face. Customers rely on VTO as disclosed herein to have an accurate sense of the frames' size relative to their face. In some embodiments, a VTO system as disclosed herein makes use of a reference spectacle frame provided to the customer beforehand. The customer puts on the reference frame and the VTO system superimposes a virtual frame, using the reference frame as a size reference.

Furthermore, without accurate three-dimensional information about a subject's head and facial features, a VTO system may improperly estimate the position and size of the spectacles on the subject's face. Accordingly, the use of typical features may show unrealistic results for potential customers, resulting in frustration and revenue loss.

To avoid an inaccurate rendition of a potential customer with spectacles, VTO systems as disclosed herein may request that the user inputs, to the VTO system, a first image of the user wearing a reference frame for a pair of spectacles. The reference frame for a pair of spectacles may include known dimensions. Accordingly, the first image may be used to accurately assess the dimensions and the relative distribution of a user's facial features.

In accordance with aspects of the subject disclosure, VTO systems and methods are provided that provide a realistic representation of the actual appearance of real-world spectacles on a user's face, using a captured image of a user's face wearing a reference frame for a pair of spectacles. Because the size of the user's face in the captured image is directly measured, the VTO systems and methods described herein can render a virtual representation of the spectacles in the correct relative size to the user's face. Accurate placement (e.g., positioning and/or orientation) of the virtual representation of the spectacles relative to the image of the user's face, as they would rest on the actual user's face, is described in further detail hereinafter. Moreover, some embodiments may further provide the ability for the user to try on multiple types, classes, and models of real-world spectacles, using the VTO application. For example, the user may try on tinted lenses, or gradient lenses, or frames having different colors, textures, and designs.

Embodiments as disclosed herein provide a technical solution to the technical problem of providing a remote rendition of an image of a user wearing a virtual pair of spectacles, prior to purchasing. Accordingly, embodiments as disclosed herein provide an enhanced use for a mobile computer device using the display as a virtual mirror to render an augmented reality image of the user wearing a desired pair of spectacles. Further, embodiments as disclosed herein make use of the technical advantage of broadband network computer technology to quickly render and adjust images in a time-sequence, to capture a user's natural movements and gestures while wearing the pair of spectacles. In some embodiments, a 3D model rendering tool may reside at least partially or fully within the application installed in the client device. By avoiding physically going to a store to try on a pair of spectacles, a VTO system as disclosed herein provides a significant commercial advantage for users and enhances sales capability and marketing reach out for vendors.

Through the use of a reference pair of spectacles, embodiments as disclosed herein achieve superior sizing accuracy relative to virtual try-on tools based on 2D imaging. Unlike yet other virtual try-on tools, it does not require a 3D camera, which makes the disclosed system available to a wider customer base. More generally, embodiments consistent with the present disclosure include the use of a reference frame or token for any other type of user-wearable item such as a wearable device, clothing, and the like. Accordingly, in some embodiments, a reference frame or token is provided to a potential customer. The customer takes an image of itself wearing the reference frame or token and an application in the mobile device displays for the user a virtual mirror image of the user with the wearable item. The user may strike a pose, change profiles, take pictures and videos from a different angle, and the application will return a realistic image or video of the user with the wearable item.

Although many examples provided herein describe a user's device or physical features as being identified and/or stored, each user may grant explicit permission for such user information to be collected, shared, and/or stored. The explicit permission may be granted using privacy controls integrated into the disclosed system. Each user may be provided notice that such user information will be stored with explicit consent, and each user may at any time end having the information stored, and may delete any stored user information. The stored user information may be encrypted to protect user security.

The user can at any time delete the user information from the memory circuit and/or opt out of having the user information stored in the memory circuit. Additionally, the user can, at any time, adjust appropriate privacy settings to selectively limit the types of user information stored in the memory circuit, or select the memory circuit in which the user information is stored (e.g., locally on the user's device as opposed to remotely on a server). The user information does not include and/or share the specific identification of the user (e.g., the user's name) unless otherwise specifically provided or directed by the user.

<FIG> illustrates an example architecture <NUM> for virtual try-on of spectacles, suitable for practicing some implementations of the disclosure. The architecture <NUM> includes servers <NUM> and client devices <NUM> connected over a network <NUM>. As shown, third-party servers <NUM> may also be communicatively coupled to servers <NUM> and/or client devices <NUM> via network <NUM>. Servers <NUM> may include database <NUM> of information associated with objects that can be virtually tried-on with one or more of client devices <NUM>. For example, database <NUM> of servers <NUM> may include three-dimensional models of various pairs of spectacles frames, with or without lenses, that are available for purchase (e.g., from the server itself, from a physical brick-and-mortar store, or from another server such as a third-party server). In some implementations, servers <NUM> may also store code that, when executed by a processor circuit of the server, causes the processor circuit to provide a try-on interface (e.g., a try-on web interface) that performs the try-on methods and operations described herein. In some implementations, the code for the try-on interface may also, or alternatively, be stored at one or more of client devices <NUM>. In these implementations, a client device running a try-on interface (e.g., generated by a try-on application running on the client device) may obtain the three-dimensional models of the spectacles from servers <NUM> and perform try-on operations as described herein at the client device.

Third-party servers may include servers <NUM> that handle sales of the spectacles for which the three-dimensional representations are stored at database <NUM>. Servers <NUM> may store three-dimensional representations of other objects that can be tried on using the try-on interface provided by other servers <NUM> and/or client devices <NUM> (e.g., by storing three-dimensional models of the other objects in database <NUM>) and providing the three dimensional models to servers <NUM> and/or client devices <NUM>.

Servers <NUM> can each be implemented with any device having an appropriate processor circuit, memory circuit, and communications capability for hosting a try-on interface and/or for storing three-dimensional models of spectacles or other objects. Client devices <NUM> can be, for example, desktop computers, mobile computers, tablet computers (e.g., including e-book readers), mobile electronic devices (e.g., a smartphone, tablet, or PDA), or any other devices having appropriate imaging (e.g., visible-light imaging and/or infrared imaging), illumination (e.g., visible light illumination and/or infrared light illumination), processor circuit, memory circuit, and communications capabilities for providing a virtual try-on interface. The network <NUM> can include, for example, any one or more of a local area network (LAN), a wide area network (WAN), the Internet, and the like. Further, the network <NUM> can include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, tree or hierarchical network, and the like.

<FIG> is a block diagram of a system <NUM> illustrating an example server <NUM> and client device <NUM> in architecture <NUM>, according to some embodiments. Client device <NUM> and server <NUM> are communicatively coupled over network <NUM> via respective communications modules <NUM>-<NUM> and <NUM>-<NUM> (hereinafter, collectively referred to as "communications modules <NUM>"). Communications modules <NUM> are configured to interface with network <NUM> to send and receive information, such as data, requests, responses, and commands to other devices on the network. Communications modules <NUM> may include, for example, modems or Ethernet cards. Client device <NUM> may be coupled with an input device <NUM> and with an output device <NUM>. Input device <NUM> may include a keyboard, a mouse, a pointer, or even a touch-screen display that a user (e.g., a consumer) may utilize to interact with client device <NUM>. Likewise, output device <NUM> may include a display and a speaker with which the user may retrieve results from client device <NUM>. Client device <NUM> may also include a processor circuit <NUM>-<NUM>, configured to execute instructions stored in a memory circuit <NUM>-<NUM>, and to cause client device <NUM> to perform at least some of the steps in methods consistent with the present disclosure. Memory circuit <NUM>-<NUM> may further include an application <NUM> including specific instructions which, when executed by processor circuit <NUM>-<NUM>, cause a payload <NUM> hosted by server <NUM> to be displayed for the consumer. Accordingly, application <NUM> may be installed by server <NUM> and perform scripts and other routines provided by server <NUM>. In some embodiments, application <NUM> may be configured to display advertisement payload <NUM> provided by virtual embedding engine <NUM>. Payload <NUM> may include multiple digital promotions, reference frames, stock items, and the like, presented to the consumer by server <NUM>. In some embodiments, the consumer may select at least some of the reference frames or purchase stock items activating buttons and other fields in payload <NUM> in memory circuit <NUM>.

Server <NUM> includes a memory circuit <NUM>-<NUM>, a processor circuit <NUM>-<NUM>, and communications module <NUM>-<NUM>. Processor circuit <NUM>-<NUM> is configured to execute instructions, such as instructions physically coded into processor circuit <NUM>-<NUM>, instructions received from software in memory circuit <NUM>-<NUM>, or a combination of both. Memory circuit <NUM>-<NUM> includes a virtual embedding engine <NUM> for integrating images, videos, and other multimedia files stored in a stock item database <NUM> into an augmented reality image or video in payload <NUM>. Virtual embedding engine <NUM> may form a virtual image or video combining an image or video provided by application <NUM> with a suitably modified image from stock item database <NUM>. Server <NUM> then provides payload <NUM> to a user of client device <NUM> that is a consumer of a retail store or chain of stores, or who desires to buy a user-wearable item online.

Virtual embedding engine <NUM> integrates payload <NUM> using a segmentation tool <NUM> and a geometric rendition tool <NUM>. Segmentation tool <NUM> may include instructions to select or create a pixel mask indicative of a reference frame from pixels in an image provided by application <NUM>.

Geometric rendition tool <NUM> may include a model to determine an orientation and a position of an object of known dimensions (e.g., a reference frame stored in reference frame database <NUM>) in a three-dimensional space. In some embodiments, the model in geometric rendition tool <NUM> may include instructions to simulate multiple positions and orientations of the reference frame and selecting an orientation and position of the reference frame that best matches (or closely matches, within a threshold) a pixel mask provided by segmentation tool <NUM>.

<FIG> is a schematic diagram illustrating multiple spectacle frames <NUM>-<NUM> ("A"), <NUM>-<NUM> ("B"), and <NUM>-<NUM> (hereinafter, collectively referred to as "spectacle frames <NUM>") in an artificial reality try-on application <NUM>, according to some embodiments. In some embodiments, the user of a client device <NUM> accesses a portal for a provider of spectacles hosted by a server (e.g., server <NUM>) through application <NUM>. The display of client device <NUM> then provides images of spectacle frames <NUM> for the user to choose and try on, according to some embodiments. The display may also illustrate different features <NUM>-<NUM> and <NUM>-<NUM> (hereinafter, collectively referred to as "features <NUM>") for each of spectacle frames <NUM>. Features <NUM> may include the material, color, texture, and shape of the frame of a pair of spectacles. In some embodiments, features <NUM> may also include the shade or type of glasses in the pair of spectacles (e.g., polarized, plated, gradient color, tinted, sunglasses, and the like).

Application <NUM> may include several buttons, <NUM>-<NUM> ("Shop"), <NUM>-<NUM> ("Try-On"), <NUM>-<NUM> ("Locations"), <NUM>-<NUM> ("Account"), and <NUM>-<NUM> ("Cart," hereinafter, collectively referred to as "buttons <NUM>"). Shop button <NUM>-<NUM> may direct the user to a display as illustrated in the figure, wherein different stock items from a database (e.g., stock item database <NUM>) are displayed for the user selection. Locations button <NUM>-<NUM> may direct the user to a map displaying different locations of stores where the user may find selected items for purchase. Account button <NUM>-<NUM> may direct the user to a personal account in the server hosting application <NUM> (e.g., server <NUM> sponsored by a store, a spectacle provider, a brand, a manufacturer, and the like). Cart button <NUM>-<NUM> may collect and store items selected by the user when Shop button <NUM>-<NUM> is activated.

Try-on button <NUM>-<NUM> directs the user to a VTO portal as disclosed herein. In the VTO portal, the user may try on a reference object and use client device <NUM> as a virtual mirror to see a realistic portrait of him/herself wearing a selected pair of spectacles, as disclosed herein.

<FIG> is a schematic diagram illustrating a VTO portal <NUM> in application <NUM>, according to some embodiments. VTO portal shows a virtual mirror image <NUM> of the user wearing a selected pair of spectacles <NUM> (e.g., selected from any one of spectacle frames <NUM> after pressing shop button <NUM>-<NUM>). A field <NUM> may include buttons to "swipe for more eyeglasses," "try on," or "buy for (price).

In some embodiments, the user receives (by physical delivery such as mail, and the like) a reference frame before accessing application <NUM>. The reference frame may be a physical object having similar size and shape as spectacle frame <NUM>. In some embodiments, the reference frame may be distinctively colored. The user then takes an image of him/herself using client device <NUM> while wearing the reference frame. Accordingly, VTO portal <NUM> can easily segment the pixels corresponding to the reference frame from the pixels corresponding to the rest of the scene. For those pixels, VTO portal <NUM> then displays a virtual mirror image <NUM> that includes the color, shape, and other features of selected spectacle frame <NUM> instead of those of the reference frame. Accordingly, the user can virtually try on several different colorways.

In some embodiments, the reference frame may not be distinctively colored. In yet other embodiments, the reference frame may be a conventional spectacle frame of a specific color or combination of colors. VTO portal <NUM> then provides a virtual mirror image <NUM> where the color of the pixels corresponding to the reference frame are replaced by the color of selected spectacle frame <NUM>.

In some embodiments, the frame of selected spectacle frame <NUM> may be the same as the reference frame, but the spectacle lenses may be different. Accordingly, VTO portal <NUM> (or the server hosting the portal) segments the spectacle lenses. Thus, virtual mirror image <NUM> includes spectacle lenses representing the user's desired type or style. To achieve this, a VTO system as disclosed herein alters the appearance of the pixels corresponding to the spectacle lenses to represent an alternative lens (e.g., tinted, sunglasses, polarized, plated, gradient color, and the like).

In some embodiments, the reference frame may not match the size or shape of selected spectacle frame <NUM>. Instead, the reference frame, which may be a conventional spectacle frame or a purpose-designed reference frame, may serve as a size reference. The VTO system may thus assess the size and shape of a user's head/face (e.g., by creating a 3D model of the user's head), so as to accurately mount selected spectacle frame <NUM> on the user's face.

<FIG> illustrates multiple virtual mirror images 535A, 535B, and 535C (hereinafter, collectively referred to as "virtual images <NUM>") from a VTO application <NUM>, according to some embodiments. VTO portal <NUM> includes a field <NUM> that the user activates to switch from a selected spectacle frame 515A to a selected spectacle frame 515B. Seamlessly, as in a real mirror, VTO portal <NUM> provides virtual images <NUM> showing (e.g., in real time) a view of the user with the recently selected spectacle frame, in different poses and profiles (e.g., virtual images 535B and 535C).

<FIG> is a schematic diagram <NUM> illustrating a 3D model <NUM> from a two-dimensional (2D) image <NUM> of a user, including facial landmark locations <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> (hereinafter, collectively referred to as "landmark locations <NUM>"), according to some embodiments.

2D image <NUM> may be provided by the user wearing a reference frame 615r via a client device (e.g., client devices <NUM>, <NUM>, and <NUM>), to a server. The server may process 2D image <NUM> with a virtual embedding engine having a segmentation tool and a geometric rendition tool <NUM>, to generate 3D model <NUM>. Accordingly, geometric rendition tool <NUM> may use landmark locations <NUM> where the user's head contacts reference frame 615r and a 3D model for reference frame 615r to ascertain size and shape of essential points in the user's head. Then, by selecting a 3D mesh that best fits landmark locations <NUM>, geometric rendition tool <NUM> may generate 3D model <NUM>. 3D model <NUM> may be used by the virtual embedding engine to provide a VTO system with a virtual image of the user, wearing a virtual pair of spectacles, shifting the head in different profiles and poses.

<FIG> is a flow chart illustrating steps in a method <NUM> for providing an augmented reality try-on interface in a client device to a user, according to some embodiments. In some embodiments, at least one or more of the steps in method <NUM> may be performed by one or more devices such as a client device or a server in an architecture as disclosed herein (e.g., client devices <NUM>, servers <NUM>, and architecture <NUM>). Accordingly, in some embodiments, at least one or more of the steps in method <NUM> may be performed by an application hosted by the server and running in the client device, wherein the client device and the server communicate with each other via communications modules, through a network (e.g., application <NUM>, communications modules <NUM>, and network <NUM>). Moreover, the application may include commands stored in a first memory circuit, the server may host the application via a virtual embedding engine including instructions stored in a second memory circuit, and the client device and server may store data in, and retrieve data from, a reference frame database (e.g., memory circuits <NUM>, virtual embedding engine <NUM>, and reference frame database <NUM>). The instructions in the memory circuits may be executed by processor circuits to cause the client device and the server to perform at least partially one or more of the steps in method <NUM> (e.g., processor circuits <NUM>). In some embodiments, the virtual embedding engine includes a segmentation tool and a geometric tool, as disclosed herein (e.g., segmentation tool <NUM> and geometric rendition tool <NUM>). Methods consistent with the present disclosure may include at least one step from method <NUM>, and one or more steps in method <NUM> performed in a different order, overlapping in time, simultaneously, quasi-simultaneously, or at least partially overlapping in time.

Step <NUM> includes capturing, in a client device, a first image of a user, the first image including a reference frame for a user-wearable item. In some embodiments, the user-wearable items may include a pair of spectacles, a piece of clothing (e.g., a hat), jewelry, makeup, and the like.

Step <NUM> includes displaying, in the client device, images of multiple user-wearable items for the user.

Step <NUM> includes receiving an input from the user, the input indicative of a selected user-wearable item from the user-wearable items on display.

Step <NUM> includes segmenting the first image to separate the reference token from a background comprising a portion of a physiognomy of the user. For example, in some embodiments the portion of the physiognomy of the user may include a face of the user, a head of the user, an arm, a leg, a wrist, a hand, a finger, and the like. In some embodiments, step <NUM> includes identifying reference token pixels in the first image, segmenting the reference token pixels, and replacing the color of the reference token pixels with a background color or with the color of the selected user-wearable item, depending on their location.

Step <NUM> includes replacing a segment of the reference token in the first image with an image of the selected user-wearable item in a second image of the user. In some embodiments, step <NUM> includes replacing reference token pixels with virtual object pixels, wherein the virtual token pixels are selected from a reference frame database and modified with respect to orientation and position by the geometric rendition tool. In some embodiments, step <NUM> includes determining an orientation and a position of reference token with a geometric rendition tool and an internal model. In some embodiments, step <NUM> includes forming an internal model of the reference token and simulating multiple positions and orientations of the reference token. In some embodiments, step <NUM> includes selecting one or more positions and orientations that best match (or better match within a threshold) to a mask provided by the segmentation tool in step <NUM>. In some embodiments, once the position and orientation of the reference token is determined, step <NUM> includes positioning and orienting an image of the selected user-wearable item (e.g., virtual object) to match the reference token. In some embodiments, step <NUM> further comprises replacing the pixels in the pixel mask with the corresponding pixels from the selected user-wearable item.

Step <NUM> includes displaying, in the client device, the second image of the user.

<FIG> is a flow chart illustrating steps in a method <NUM> for hosting an augmented reality try-on interface from a server, according to some embodiments. In some embodiments, at least one or more of the steps in method <NUM> may be performed by one or more devices such as a client device or a server in an architecture as disclosed herein (e.g., client devices <NUM>, servers <NUM>, and architecture <NUM>). Accordingly, in some embodiments, at least one or more of the steps in method <NUM> may be performed by an application hosted by the server and running in the client device, wherein the client device and the server communicate with each other via communications modules, through a network (e.g., application <NUM>, communications modules <NUM>, and network <NUM>). Moreover, the application may include commands stored in a first memory circuit, the server may host the application via a virtual embedding engine including instructions stored in a second memory circuit, and the client device and server may store data in, and retrieve data from, a reference frame database (e.g., memory circuits <NUM>, virtual embedding engine <NUM>, and reference frame database <NUM>). The instructions in the memory circuits may be executed by processor circuits to cause the client device and the server to perform at least partially one or more of the steps in method <NUM> (e.g., processor circuits <NUM>). In some embodiments, the virtual embedding engine includes a segmentation tool and a geometric tool, as disclosed herein (e.g., segmentation tool <NUM> and geometric rendition tool <NUM>). Methods consistent with the present disclosure may include at least one step from method <NUM>, and one or more steps in method <NUM> performed in a different order, overlapping in time, simultaneously, quasi-simultaneously, or at least partially overlapping in time.

Step <NUM> includes receiving, from a client device, a first image of a user, the first image including a reference token for a user-wearable item. In some embodiments, the user-wearable items may include a pair of spectacles, a piece of clothing (e.g., a hat), jewelry, makeup, and the like.

Step <NUM> includes providing for display, to the client device, images of multiple user-wearable items for the user stored in a database.

Step <NUM> includes receiving an input from the user, the input indicative of a selected user-wearable item from the pairs of spectacles on display.

Step <NUM> includes providing for display, in the client device, a second image of the user, the second image having a segment of the reference token replaced with an image of the selected user-wearable item. In some embodiments, step <NUM> further includes selecting a pixel in the first image based on an estimated position and orientation of a head of the user provided by a geometric rendition engine to identify the segment for the reference token. In some embodiments, step <NUM> further includes selecting multiple pixels from the first image and forming a mask to identify the segment of the reference token. In some embodiments, step <NUM> further includes determining an orientation and a position of the reference frame with a geometric rendition tool, and replacing the segment of the reference token in the first image with an image of a selected user-wearable item based on the orientation and the position of the reference token. In some embodiments, step <NUM> further includes positioning and orienting an image of the selected user-wearable item to match the reference token and to replace a segment of the reference token in the first image. In some embodiments, the reference token for a user-wearable item is a known object that fits on a portion of a physiognomy of the user. Accordingly, step <NUM> may include obtaining a model of a head of the user based on a two-dimensional projection of the known object in the first image, and superimposing a three dimensional model of the selected user-wearable item on the model of the head of the user.

Step <NUM> includes receiving, from the client device, a request for the selected user-wearable item.

<FIG> is a block diagram illustrating an exemplary computer system <NUM> with which the user device <NUM>, and servers <NUM>, of <FIG> can be implemented. In certain aspects, the computer system <NUM> may be implemented using hardware or a combination of software and hardware, either in a dedicated server, or integrated into another entity, or distributed across multiple entities.

Computer system <NUM> includes a bus <NUM> or other communication mechanism for communicating information, and a processor circuit <NUM> coupled with bus <NUM> for processing information. By way of example, the computer system <NUM> may be implemented with one or more processor circuits <NUM>. Processor circuit <NUM> may be a general-purpose microprocessor circuit, a microcontroller, a Digital Signal Processor circuit (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable entity that can perform calculations or other manipulations of information.

Computer system <NUM> can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor circuit firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them stored in an included memory circuit, such as a Random Access Memory circuit (RAM), a flash memory circuit, a Read-Only Memory circuit (ROM), a Programmable Read-Only Memory circuit (PROM), an Erasable PROM (EPROM), registers, a hard disk, a removable disk, a CD-ROM, a DVD, or any other suitable storage device, coupled to bus <NUM> for storing information and instructions to be executed by processor circuit <NUM>. The processor circuit <NUM> and the memory circuit <NUM> can be supplemented by, or incorporated in, special purpose logic circuitry.

The instructions may be stored in the memory circuit <NUM> and implemented in one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, the computer system <NUM>, and according to any method well known to those of skill in the art, including, but not limited to, computer languages such as data-oriented languages (e.g., SQL, dBase), system languages (e.g., C, Objective-C, C++, Assembly), architectural languages (e.g., Java,. NET), and application languages (e.g., PHP, Ruby, Perl, Python). Instructions may also be implemented in computer languages such as array languages, aspect-oriented languages, assembly languages, authoring languages, command line interface languages, compiled languages, concurrent languages, curly-bracket languages, dataflow languages, data-structured languages, declarative languages, esoteric languages, extension languages, fourth-generation languages, functional languages, interactive mode languages, interpreted languages, iterative languages, list-based languages, little languages, logic-based languages, machine languages, macro languages, metaprogramming languages, multiparadigm languages, numerical analysis, non-English-based languages, object-oriented class-based languages, object-oriented prototype-based languages, off-side rule languages, procedural languages, reflective languages, rule-based languages, scripting languages, stack-based languages, synchronous languages, syntax handling languages, visual languages, wirth languages, and xml-based languages. Memory circuit <NUM> may also be used for storing temporary variable or other intermediate information during execution of instructions to be executed by processor circuit <NUM>.

A computer program as discussed herein does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subprograms, or portions of code). The processes and logic flows described in this specification can be performed by one or more programmable processor circuits executing one or more computer programs to perform functions by operating on input data and generating output.

Computer system <NUM> further includes a data storage device <NUM> such as a magnetic disk or optical disk, coupled to bus <NUM> for storing information and instructions. Computer system <NUM> may be coupled via input/output module <NUM> to various devices. The input/output module <NUM> can be any input/output module. Exemplary input/output modules <NUM> include data ports such as USB ports. The input/output module <NUM> is configured to connect to a communications module <NUM>. Exemplary communications modules <NUM> include networking interface cards, such as Ethernet cards and modems. In certain aspects, the input/output module <NUM> is configured to connect to a plurality of devices, such as an input device <NUM> and/or an output device <NUM>. Exemplary input devices <NUM> include a keyboard and a pointing device (e.g., a mouse or a trackball), by which a user can provide input to the computer system <NUM>. Other kinds of input devices <NUM> can be used to provide for interaction with a user as well, such as a tactile input device, visual input device, audio input device, or brain-computer interface device. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, tactile, or brain wave input. Exemplary output devices <NUM> include display devices, such as an LCD (liquid crystal display) or light-emitting diode (LED) display, for displaying information to the user.

According to one aspect of the present disclosure, user device <NUM> and servers <NUM> can be implemented using a computer system <NUM> in response to processor circuit <NUM> executing one or more sequences of one or more instructions contained in memory circuit <NUM>. Such instructions may be read into memory circuit <NUM> from another machine-readable medium, such as data storage device <NUM>. Execution of the sequences of instructions contained in main memory circuit <NUM> causes processor circuit <NUM> to perform the process steps described herein. One or more processor circuits in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in memory circuit <NUM>. In alternative aspects, hard-wired circuitry may be used in place of or in combination with software instructions to implement various aspects of the present disclosure. Thus, aspects of the present disclosure are not limited to any specific combination of hardware circuitry and software.

Various aspects of the subject matter described in this specification can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The communication network (e.g., network <NUM>) can include, for example, any one or more of a LAN, a WAN, the Internet, and the like. Further, the communication network can include, but is not limited to, for example, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, tree or hierarchical network, or the like. The communications modules can be, for example, modems or Ethernet cards.

Computer system <NUM> can include clients and servers. Computer system <NUM> can be, for example, and without limitation, a desktop computer, laptop computer, or tablet computer. Computer system <NUM> can also be embedded in another device, for example, and without limitation, a mobile telephone, a PDA, a mobile audio player, a Global Positioning System (GPS) receiver, a video game console, and/or a television set top box.

The term "machine-readable storage medium" or "computer-readable medium" as used herein refers to any medium or media that participates in providing instructions to processor circuit <NUM> for execution. Such a medium may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as data storage device <NUM>. Volatile media include dynamic memory circuit, such as memory circuit <NUM>. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise bus <NUM>. Common forms of machine-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM, a FLASH EPROM, any other memory circuit chip or cartridge, or any other medium from which a computer can read. The machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory circuit device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them.

As used herein, the phrase "at least one of" preceding a series of items, with the terms "and" or "or" to separate any of the items, modifies the list as a whole, rather than each member of the list (i.e., each item).

To the extent that the term "include," "have," or the like is used in the description or the claims, such term is intended to be inclusive in a manner similar to the term "comprise" as "comprise" is interpreted when employed as a transitional word in a claim.

A reference to an element in the singular is not intended to mean "one and only one" unless specifically stated, but rather "one or more. " All structural and functional equivalents to the elements of the various configurations described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and intended to be encompassed by the subject technology. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the above description.

While this specification contains many specifics, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of particular implementations of the subject matter.

Claim 1:
A computer-implemented method, comprising:
capturing, by a client device, a first image of a user, the first image including a reference token for a user-wearable item, the reference token being worn by the user in the first image and having a known three-dimensional model;
automatically generating, by the client device, a three-dimensional model of a face of the user from the first image of the user, the three-dimensional model of the face of the user including dimensions based on the known-three-dimensional model of the reference token;
displaying, by the client device, images of multiple user-wearable items for the user;
receiving an input from the user, the input indicative of a selected user-wearable item from the user-wearable items on display, the selected user-wearable item having an associated three-dimensional model;
segmenting, by the client device, the first image to separate the reference token from a background comprising a portion of a physiognomy of the user;
replacing, by the client device, a segment for the reference token in the first image with an image for the selected user-wearable item in a second image of the user based on the three-dimensional model of the face of the user and the three-dimensional model of the selected user-wearable item; and
displaying, by the client device, the second image of the user.