Patent Description:
The use of computer systems and computer-related technologies continues to increase at a rapid pace. This increased use of computer systems has influenced the advances made to computer-related technologies. Indeed, computer systems have increasingly become an integral part of the business world and the activities of individual consumers. Computers have opened up an entire industry of internet shopping. In many ways, online shopping has changed the way consumers purchase products. For example, a consumer may want to know what they will look like in and/or with a product. On the webpage of a certain product, a photograph of a model with the particular product may be shown. However, users may want to see more accurate depictions of themselves in relation to various products.

The document <CIT> discloses a method of spectacle frame <NUM>-D simulation fitting based upon a database of product information and digitized user images acquired via devices connected to a computer. Consumers can choose from a wide variety of spectacle frames. They may use digital cameras, network cameras or scanned photos to submit facial images. By way of calibration steps and additional side-view images of the consumers when test-fitting spectacle frames, a simulated <NUM>-D presentation of a consumer wearing the intended spectacle frame can be viewed for the purpose of fitting and purchasing.

The document <CIT> discloses a method of measuring the interpupillary distance of a person using a computer or a portable telephone. The distance is expressed in the form of a number of pixels. The number of pixels is transmitted to a remote server via the Internet. A real value of the interpupillary distance is calculated from the number of pixels. A distance between a digital camera of the terminal, a focal distance and the person is measured by a direct measurement unit e.g. laser or infra-red pointer.

The document <CIT> discloses a table, which shows pre-stored distances between a human face and a camera device as a function of the number of pixels representing the interpupillary distance. The interpupillary distance is assumed to be similar (approximately <NUM>) from one person to another.

The document <CIT> discloses scaling a three-dimensional model. The user is asked to hold an object of known size in front of the camera so as to determine the accurate scaling factor.

According to at least one embodiment, a computer-implemented method according to claim <NUM> is described.

A computing device according to claim <NUM> and a computer-program product according to claim <NUM> are also described. Preferred embodiments are described in the following detailed description in conjunction with the accompanying drawings and claims.

The accompanying drawings illustrate a number of exemplary embodiments and are a part of the specification.

While the embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the instant disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.

In various situations, it may be desirable to scale an object. For example, it may be desirable to scale a three-dimensional (3D) model of a user so that two or more 3D models may be mated and scaled according to a common scale. For instance, the systems and methods described herein may allow for proper scaling of 3D models when virtually tying-on products (e.g., virtually trying-on a pair of glasses). Accordingly, a scaled 3D model of the user may be mated with a scaled 3D model of a pair of glasses. Although examples used herein may describe the scaling of a user and/or a pair of glasses, it is understood that the systems and methods described herein may be used to scale a model of any object.

<FIG> is a block diagram illustrating one embodiment of an environment <NUM> in which the present systems and methods may be implemented. In some embodiments, the systems and methods described herein may be performed on a single device (e.g., device <NUM>). For example, the systems and method described herein may be performed by a scaling module <NUM> that is located on the device <NUM>. Examples of device <NUM> include mobile devices, smart phones, personal computing devices, computers, servers, etc..

In some configurations, a device <NUM> may include the scaling module <NUM>, a camera <NUM>, and a display <NUM>. In one example, the device <NUM> may be communicatively coupled to a database <NUM>. In one embodiment, the database <NUM> may be internal to device <NUM>. In one embodiment, the database <NUM> may be external to device <NUM>. In some embodiments, portions of database <NUM> may be both internal and external to device <NUM>. In some configurations, the database <NUM> may include model data <NUM> and pixel density data <NUM>.

In one embodiment, the scaling module <NUM> may scale a model of an object. Scaling module <NUM> is configured to scale a 3D model of an object. In one example, scaling a 3D model of a user enables the user to view an image on the display <NUM> of the scaled, 3D model of the user in relation to another 3D object. For instance, the image may depict a user virtually trying-on a pair of glasses with both the user and the glasses being scaled according to a common scaling standard determined by scaling module <NUM>. Thus, scaling module <NUM> may scale the 3D model of the user and the 3D model of the pair of glasses, such that the glasses appear in proper scale in relation to the user as they would if the user were to wear an actual pair of the glasses. The scaled models may then be mated to render a 2D image of the user wearing the glasses.

Scaling module <NUM> may store scaling information in database <NUM>. Thus, model data <NUM> may include scaling information determined by scaling module <NUM>, image data captured by camera <NUM>, information and data regarding a model of a user, information and data regarding a model of an object, and algorithms used by scaling module <NUM> to determine one or more distances in a particular unit of distance associated with an image of a user captured by camera <NUM>. Pixel density data <NUM> may include information and data regarding a camera sensor, including the sensor size, a pixel density or resolution of the sensor (e.g., <NUM>,<NUM> horizontal pixel count by <NUM> vertical pixel count for a <NUM> megapixel (MP) sensor, etc.), a pixel density of an image (e.g., horizontal and vertical pixels in the image), a pixel density per unit length from the camera (e.g., the number of pixels per inch for an object that is a certain number of inches from the camera when an image of the object is captured, such as <NUM> pixels per inch when the depth of the object from the camera is <NUM> inches at the time the image is captured, etc.), and so forth.

Accordingly, in one embodiment, the 3D model of an object and/or user may be obtained based on the model data <NUM>. In one example, the model data <NUM> may be based on an average model that may be adjusted according to measurement information determined about the object (e.g., a morphable model approach). In one example, the 3D model of the object and/or user may be a linear combination of the average model. In some embodiments, the model data <NUM> may include one or more definitions of color (e.g., pixel information) for the 3D model. In one example, the 3D model may have an arbitrary size. In some embodiments, the scaled 3D model (as scaled by the systems and methods described herein, for example) may be stored in the model data <NUM>. In some cases, a rendered, 2D image based on the scaled 3D model may be displayed via the display <NUM>. For example, an image of a virtual try-on based on the scaled 3D representation of a user and a 3D model of glasses scaled according to determined scaling may be displayed on display <NUM>.

<FIG> is a block diagram illustrating another embodiment of an environment <NUM> in which the present systems and methods may be implemented. In some embodiments, a device <NUM>-a may communicate with a server <NUM> via a network <NUM>. Examples of networks <NUM> include local area networks (LAN), wide area networks (WAN), virtual private networks (VPN), cellular networks (using <NUM> and/or LTE, for example), etc. In some configurations, the network <NUM> may be the internet. In some configurations, the device <NUM>-a may be one example of the device <NUM> illustrated in <FIG>. For example, the device <NUM>-a may include the camera <NUM>, the display <NUM>, and an application <NUM>.

In some embodiments, the server <NUM> may include the scaling module <NUM>. In one embodiment, the server <NUM> may be coupled to the database <NUM>. For example, the scaling module <NUM> (from device <NUM>-a and/or server <NUM>) may access the model data <NUM> in the database <NUM> via the server <NUM>. The database <NUM> may be internal or external to the server <NUM>, or both (e.g., a copy of model data <NUM> and/or pixel density data <NUM> stored on a storage device located in server and synchronized with the content on an external database <NUM>). In some embodiments, the device <NUM>-a may not include a scaling module <NUM>. For example, the device <NUM>-a may include an application <NUM> that allows device <NUM>-a to interface with the scaling module <NUM> located on server <NUM>. In some embodiments, both the device <NUM>-a and the server <NUM> may include a scaling module <NUM> where at least a portion of the functions of scaling module <NUM> are performed separately on device <NUM>-a or server <NUM>, and/or at least a portion of the functions of scaling module <NUM> are performed concurrently on device <NUM>-a and server <NUM>.

In some configurations, the application <NUM> may capture one or more images via camera <NUM>. In one embodiment, upon capturing the image, the application <NUM> may transmit the captured image to the server <NUM>. In some cases, the scaling module <NUM> may obtain the image and may generate a scaled 3D model of the user. In one example, the scaling module <NUM> may transmit scaling information and/or information based on the scaled 3D model of the user to the device <NUM>-a. In some configurations, the application <NUM> may obtain the scaling information and/or information based on the scaled 3D model of the object and may output a 2D image based on the scaled 3D model of the object to be displayed via the display <NUM>.

<FIG> is a block diagram illustrating one example of a scaling module <NUM>-a. The scaling module <NUM>-a may be one example of the scaling module <NUM> illustrated in <FIG> or <FIG>. The scaling module <NUM>-a may include a range finding module <NUM>, an image capturing module <NUM>, a querying module <NUM>, a feature detection module <NUM>, a pixel counting module <NUM>, a conversion module <NUM>, a scaling module <NUM>, and a pixel density module <NUM>.

In one embodiment, rangefinder module <NUM> may measure a distance of an object from the mobile device. For example, rangefinder module <NUM> may employ optical, electroacoustic, and/or electronic means to measure a distance to an object. In some embodiments, rangefinder module <NUM> may include a coincidence range finder. Rangefinder module <NUM> may produce two or more images of an object (e.g., using mirrors and/or prisms). The rangefinder module <NUM> may sight the object through a viewfinder and adjust a mechanism to bring the two or more images into alignment. The rangefinder module <NUM> may scale the amount of adjustment to the mechanism to determine the distance to the object. In some cases, rangefinder module <NUM> may use coincidence and/or stereoscopic rangefinder methods. Thus, rangefinder module <NUM> may use a pair of eyepieces through which a single image of an object may be seen. A pattern of lines may appear to float in a space in the view of the eyepieces. A control mechanism may be adjusted until the pattern appears to be at the same distance as the object, which in turn adjusts a value on a scale. The rangefinder module <NUM> may read the distance to the object by reading a value on the scale that results from adjusting the control mechanism. In some cases, rangefinder module <NUM> may employ a laser rangefinder. A laser rangefinder may use an invisible, eye-safe Class <NUM> Laser beam which bounces off an object. The rangefinder module <NUM> may use a high-speed digital clock to measure the time it takes for the laser beam to reach the target object and return to the camera. Based on the measured time, the rangefinder module <NUM> may use digital electronics to calculate the distance to the target object. In some cases, the rangefinder module <NUM> may employ a light emitting diode (LED) rangefinder that operates in the same manner as a laser rangefinder. In some embodiments, rangefinder module <NUM> may employ ultrasound to measure the distance to an object similar to the way the laser rangefinder measures a laser. Thus, instead of measuring the time it takes for a laser to bounce off an object, rangefinder module <NUM> may emit a high-frequency sound wave towards the target object and measure the time it takes for the high-frequency sound wave to bounce off the object and return to the camera.

In one embodiment, image capturing module <NUM> may capture an image of the object. In some cases, image capturing module <NUM> may capture one or more images of the object upon determining the distance to the object via the rangefinder module <NUM>. Upon determining the distance to the object via the rangefinder module <NUM>, querying module <NUM> may query a database of pixel densities (e.g., database <NUM>) for an image pixel density at the measured distance of the object from the mobile device. The database may contain a predetermined number of image pixel densities for a given number of distances. For example, for a camera of a given megapixel count (e.g., <NUM> MP), the pixel density of a captured image at <NUM> inches depth from the camera may measure to be <NUM> pixels per inch, at <NUM> inches from the camera the pixel density of the captured image may be <NUM> pixels per inch, and so forth.

In some embodiments, feature detection module <NUM> may detect a feature of the object from the captured image of the object. In some cases, detecting a feature of the object may include detecting a pupil of a user. Pixel counting module <NUM> may count a number of pixels associated with the detected feature of the object. Conversion module <NUM> may determine a distance associated with the detected feature of the object based on the number of pixels associated with the detected feature of the object. For example, the conversion module <NUM> may determine the distance by determining a value of a quotient resulting from dividing the number of pixels associated with the detected feature of the object by the queried pixel density at the measured distance of the object.

Determining a distance associated with the detected feature may include determining a pupil distance of the user. Thus, pixel counting module <NUM> may determine that the number of pixels associated with the distance between the user's pupils is <NUM> pixels. Querying module <NUM> may query a database to determine that the image pixel density at the determined distance of the user from the camera to be <NUM> pixels per inch. Accordingly, conversion module <NUM> may divide the number of pixels, <NUM> pixels, by the pixel density, <NUM> pixels per inch, to determine that there are <NUM> inches, or about <NUM>, between the user's pupils. Accordingly, scaling module <NUM> may scale a depiction of the object based on the determined distance associated with the detected feature of the object. For example, scaling module <NUM> may scale a three-dimensional model of a user based on the determined distance associated with the detected feature of the user (e.g., pupil distance). In some cases, which are not covered by the wording of the claims, scaling module <NUM> may scale a two-dimensional image of an object (e.g., a user).

In one embodiment, pixel density module <NUM> may determine a sensor pixel density of a camera sensor. For example, pixel density module <NUM> may determine the pixel density of a particular sensor is <NUM> MP. Pixel density module <NUM> may determine a pixel density of an image captured by the camera of the mobile device for a predetermined distance from the mobile device. In some embodiments, the pixel density module <NUM> may determine the pixel density of an image based at least on the sensor pixel density of the sensor and/or the sensor size. The scaling module <NUM>-a may store the determined pixel density for each predetermined distance from the mobile device in a database (e.g., database <NUM>).

<FIG> is a diagram <NUM> illustrating an example of a device <NUM>-b for capturing an image of an object. The depicted device <NUM>-b may be one example of the devices <NUM> illustrated in <FIG> and/or <NUM>. As depicted, the device <NUM>-b may include a camera <NUM>-a, a rangefinder <NUM>, and a display <NUM>-a. The camera <NUM>-a and display <NUM>-a may be examples of the respective camera <NUM> and display <NUM> illustrated in <FIG> and/or <NUM>.

As depicted, device <NUM>-b may capture an image of a user <NUM>. At the time the image is captured (e.g., just before the image is captured, while the image is being captured, just after the image is captured, etc.), a rangefinder <NUM> may determine a distance between the camera <NUM>-a and the user <NUM>. As described above, pixel density data <NUM> may include information and data regarding a pixel density per unit length from the camera. For example, pixel density data <NUM> may include data regarding the pixel density of an image at a first distance <NUM>, the pixel density of an image at a second distance <NUM>, and/or the pixel density of an image at a third distance <NUM>. For instance, it may be determined that an image of an object at the first distance <NUM> would have <NUM> pixels per inch, that an image of an object at the second distance <NUM> would have <NUM> pixels per inch, and an image of an object at the third distance <NUM> would have <NUM> pixels per inch, and so forth.

As depicted, the rangefinder <NUM> may emit a signal <NUM> towards the user <NUM>. The emitted signal may bounce off the user <NUM> and a reflected signal <NUM> may return to the rangefinder <NUM>. The scaling module <NUM> in conjunction with the rangefinder <NUM> may determine from the reflected signal <NUM> (e.g., time between emission of the emitted signal <NUM> and receipt of the reflected signal <NUM>) that the user <NUM> is situated at a distance from the camera equivalent to the third distance <NUM>. Accordingly, scaling module <NUM> may use information associated with the distance <NUM> between the camera <NUM>-a and the user <NUM> to determine a size of a feature of the user (e.g., distance between the pupils, etc.). Scaling module <NUM> may use this determined size information to scale a model of the user in relation to one or more other objects.

<FIG> is a diagram illustrating an example arrangement <NUM> of a captured image of a user <NUM> for use in the systems and methods described herein. The arrangement <NUM> depicts a front view of an image of a user <NUM>. In one embodiment, the image of the user <NUM> may represent a resultant image of user <NUM> captured by camera <NUM>-a in relation to the arrangement of <FIG>. In some embodiments, scaling module <NUM> may determine the pixel density (e.g., pixels per inch, pixels per millimeter, etc.) associated with a detected feature of an object, where the pixel density is determined in relation to a determined distance of the object from the camera when the image was captured. In some cases, scaling module <NUM> may determine that distance <NUM> represents the numbers of pixels per millimeter. For example, as depicted, scaling module <NUM> may determine there are four pixels per millimeter in relation to a determined distance between the camera and a detectable feature of the user. Thus, a distance <NUM> between two points on the image of the user (e.g., pupil distance) may be determined based on the determined pixel density of the image <NUM> at the determined distance between the user and the camera. For example, scaling module <NUM> may determine that there are <NUM> pixels between the two points that make up the distance <NUM>. Knowing the distance between the user and the camera (e.g., distance <NUM> of <FIG>), scaling module <NUM> may determine that there are <NUM> pixels per mm in image <NUM> at that determined distance between the user and the camera. Accordingly, scaling module <NUM> may determine the quotient that results by dividing the number of pixels between distance <NUM> (e.g., <NUM> pixels) by the determined pixel density (e.g., <NUM> pixels per mm) to determine that the distance <NUM> is equivalent to a value around <NUM>. Based on this determined distance, scaling module <NUM> may scale a model of the user, as described above.

<FIG> is a flow diagram illustrating one example of a method <NUM> for determining a distance between a camera and an object whose image is being captured by a camera. In some configurations, the method <NUM> may be implemented by the scaling module <NUM> illustrated in <FIG>, <FIG>, or <FIG>.

At block <NUM>, a distance of an object from a mobile computing device may be measured via a processor of the mobile computing device in conjunction with a rangefinder. At block <NUM>, an image of the object may be captured via the processor. At block <NUM>, a database of pixel densities for a pixel density at the measured distance of the object from the mobile device may be queried via the processor.

<FIG> is a flow diagram illustrating one example of a method <NUM> for scaling a model of an object based on a determined distance of the object from a camera when an image of the object is captured. In some configurations, the method <NUM> may be implemented by the scaling module <NUM> illustrated in <FIG>, <FIG>, or <FIG>.

At block <NUM>, a feature of an object may be detected from an image of the object. In some cases, detecting a feature of the object may include detecting a pupil of a user. At block <NUM>, a number of pixels associated with the detected feature of the object may be counted. At block <NUM>, a distance associated with the detected feature of the object may be determined based on a quotient resulting from dividing the number of pixels associated with the detected feature of the object by the queried pixel density at the measured distance of the object. In some cases, determining a distance associated with the detected feature may include determining a pupil distance of the user. At block <NUM>, a depiction of the object may be scaled based on the determined distance associated with the detected feature of the object.

<FIG> is a flow diagram illustrating another example of a method <NUM> for calibrating a mobile device to determine a unit of length in relation to an image of an object based on a determined distance of the object from the camera. In some configurations, the method <NUM> may be implemented by the scaling module <NUM> illustrated in <FIG>, <FIG>, or <FIG>.

At block <NUM>, a sensor pixel density of a camera sensor may be determined. At block <NUM>, a pixel density of an image captured by the camera of the mobile device may be determined for a predetermined distance from the mobile device based at least on the sensor pixel density of the sensor. At block <NUM>, the determined pixel density may be stored for each predetermined distance from the mobile device in a database.

<FIG> depicts a block diagram of a computer system <NUM> suitable for implementing the present systems and methods. For example, the computer system <NUM> may be suitable for implementing the device <NUM> illustrated in <FIG>, <FIG>, or <FIG> and/or the server <NUM> illustrated in <FIG>. Computer system <NUM> includes a bus <NUM> which interconnects major subsystems of computer system <NUM>, such as a central processor <NUM>, a system memory <NUM> (typically RAM, but which may also include ROM, flash RAM, or the like), an input/output controller <NUM>, an external audio device, such as a speaker system <NUM> via an audio output interface <NUM>, an external device, such as a display screen <NUM> via display adapter <NUM>, a keyboard <NUM> (interfaced with a keyboard controller <NUM>) (or other input device), multiple universal serial bus (USB) devices <NUM> (interfaced with a USB controller <NUM>), and a storage interface <NUM>. Also included are a mouse <NUM> (or other point-and-click device) interfaced through a serial port <NUM> and a network interface <NUM> (coupled directly to bus <NUM>).

Bus <NUM> allows data communication between central processor <NUM> and system memory <NUM>, which may include read-only memory (ROM) or flash memory (neither shown), and random access memory (RAM) (not shown), as previously noted. The RAM is generally the main memory into which the operating system and application programs are loaded. The ROM or flash memory can contain, among other code, the Basic Input-Output system (BIOS) which controls basic hardware operation such as the interaction with peripheral components or devices. For example, the scaling module <NUM>-b to implement the present systems and methods may be stored within the system memory <NUM>. Applications (e.g., application <NUM>) resident with computer system <NUM> are generally stored on and accessed via a non-transitory computer readable medium, such as a hard disk drive (e.g., fixed disk <NUM>) or other storage medium. Additionally, applications can be in the form of electronic signals modulated in accordance with the application and data communication technology when accessed via interface <NUM>.

Storage interface <NUM>, as with the other storage interfaces of computer system <NUM>, can connect to a standard computer readable medium for storage and/or retrieval of information, such as a fixed disk drive <NUM>. Fixed disk drive <NUM> may be a part of computer system <NUM> or may be separate and accessed through other interface systems. Network interface <NUM> may provide a direct connection to a remote server via a direct network link to the Internet via a POP (point of presence). Network interface <NUM> may provide such connection using wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection, or the like.

Many other devices or subsystems (not shown) may be connected in a similar manner (e.g., document scanners, digital cameras, and so on). Conversely, all of the devices shown in <FIG> need not be present to practice the present systems and methods. The devices and subsystems can be interconnected in different ways from that shown in <FIG>. The operation of a computer system such as that shown in <FIG> is readily known in the art and is not discussed in detail in this application. Code to implement the present disclosure can be stored in a non-transitory computer-readable medium such as one or more of system memory <NUM> or fixed disk <NUM>. The operating system provided on computer system <NUM> may be iOS®, MS-DOS®, MS-WINDOWS®, OS/<NUM>®, UNIX®, Linux®, or another known operating system.

While the foregoing disclosure sets forth various embodiments using specific block diagrams, flowcharts, and examples, each block diagram component, flowchart step, operation, and/or component described and/or illustrated herein may be implemented, individually and/or collectively, using a wide range of hardware, software, or firmware (or any combination thereof) configurations. In addition, any disclosure of components contained within other components should be considered exemplary in nature since many other architectures can be implemented to achieve the same functionality.

The various exemplary methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

Furthermore, while various embodiments have been described and/or illustrated herein in the context of fully functional computing systems, one or more of these exemplary embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the exemplary embodiments disclosed herein.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings, within the scope of the claims. The embodiments were chosen and described in order to best explain the principles of the present systems and methods and their practical applications, to thereby enable others skilled in the art to best utilize the present systems and methods and various embodiments with various modifications as may be suited to the particular use contemplated.

Claim 1:
A computer-implemented method for scaling at least one of a first three-dimensional, 3D, model or a second 3D model, wherein each of the first 3D model and the second 3D model is based on corresponding model data (<NUM>), by processing at least part of the model data (<NUM>) corresponding to at least one of the first 3D model or the second 3D model, wherein the first 3D model represents a first object, and wherein the second 3D model represents a second object, the method comprising:
measuring, via a processor of a computing device (<NUM>) in conjunction with a rangefinder, a distance of the first object from the computing device (<NUM>);
capturing, via the processor, an image of the first object;
querying, via the processor, a database (<NUM>) of pixel density data (<NUM>) for a pixel density at the measured distance of the first object from the computing device;
detecting, via the processor, a first feature of the first object;
determining, via the processor, a number of pixels associated with the detected first feature of the first object;
determining, via the processor, a size associated with the detected first feature of the first object, wherein determining the size comprises determining a quotient resulting from dividing the number of pixels associated with the detected first feature of the first object by the queried pixel density at the measured distance of the first object;
scaling, via the processor, at least one of the first 3D model or the second 3D model, based on the determined size, by processing at least part of the corresponding model data (<NUM>); and
mating, via the processor, the second 3D model with the first 3D model, based on a result of the scaling.