Patent Description:
In particular, <CIT> already discloses a method of creating time lapse video of a live subject, the method comprising:.

Within the cosmetics and personal care industries, a known technique for measuring differences in personal appearance over time is to capture consistently posed and lighted images over time at designated intervals. In a simple example, consistently posed and lighted images are captured to show differences between a consumer's appearance before and after an event, such as the consumer's use of a particular personal care device or product. The consistent poses and lighting allow images to be analyzed to detect differences in appearance related to, e.g., use of a particular personal care device or product.

However, it is very difficult for consumers working at home to capture consistently posed and lighted images. The technical skill and lighting equipment required for capturing such images are not attainable for the average consumer. On the other hand, if images or video of the consumer are captured in uncontrolled environments (e.g., without regard to lighting, maintaining a consistent pose, etc.), those images cannot currently be used to reliably track progress in personal care routines due to technical difficulties of processing images from uncontrolled environments. What is desired are techniques that can overcome these technical limitations in order to accurately track progress of skin treatments, use of cosmetics, or personal care routines without the use of professional photographers or professional photography equipment.

These problems are solved by a method of creating time lapse video of a live subject according to independent claim <NUM>. The dependent claims relate to advantageous embodiments.

The foregoing aspects and many of the attendant advantages of the present disclosure will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:.

In some embodiments of the present disclosure, a computer system produces time series images to accurately illustrate gradual changes in personal appearance, such as changes in skin conditions over time. The time series images are assembled from images captured without the need for professional photographers or professional photography equipment. In an embodiment, a user captures video or a series of still images of themselves at periodic intervals (e.g., once per day) which, after pre-processing, can be used to generate output such as a time-morph video to demonstrate changes over time. In a cosmetics or personal care context, such videos are used to demonstrate progress made in personal care (e.g., improvements in skin condition, skin color, skin texture, hair condition, etc.) Such videos can be shared with others (e.g., via social media platforms or directly from one computing device to another).

In an embodiment, a user captures a large number of source images (e.g., <NUM>, <NUM>, hundreds, etc.) of a region of interest of a live subject (e.g., the user's face or a portion of the user's face), which may be taken at different time instances in different environments, from a variety of angles in a range of possible lighting conditions. The images are normalized to an objective baseline, to allow accurate comparisons and illustration of changes over time. The normalization can be used to reduce effects of variations in lighting/exposure, orientation, camera jitter, lens condition, and the like. These images are post-processed to obtain a representative image, which may include selecting an image from among the source images or using portions of two or more source images to synthesize a new image (e.g., by averaging, compositing, or otherwise combining image data from the source images). In an embodiment, a representative image is associated with a time period in which the corresponding source images are captured (e.g., weekly, daily, twice daily, etc.).

In an embodiment, the synthesized images are combined to form a time series morph video that depicts physiological or other changes in appearance over time, such as making wrinkles or blemishes less apparent, rejuvenation of skin, improvements in skin tone, and the like. By employing a consistent approach to how the representative image is processed for each time period, the variations of conditions in the synthesized images are much reduced relative to the variations that may be present in the source images. This allows for greater ease and accuracy in comparing the images to track changes in the consumer's appearance over time.

<FIG> is a schematic illustration of a non-limiting example embodiment of a system that obtains images and generates time lapse video according to various aspects of the present disclosure. As shown, a user <NUM> captures one or more digital source images of her face with a mobile computing device <NUM>. The user <NUM> may be in any type of location, and therefore the lighting conditions and other environmental factors in which the image is captured may be unknown.

In some embodiments, the mobile computing device <NUM> captures and transmits a time series of images to a time-lapse video processing computer system <NUM>. The images may be obtained from video or from a series of still images. Lighting conditions and viewing angles may differ among the captured source images. In an illustrative scenario, depicted in <FIG>, the user <NUM> configures the mobile computing device <NUM> to capture video or still images using a front-facing camera (also known as a "selfie" camera), and moves the mobile computing device <NUM> in a semi-circular motion to capture images of a region of interest (e.g., the user's face or a portion thereof, such as the user's forehead, eyes, etc.) from different angles. In another embodiment, a camera (e.g., a front-facing camera of a laptop computer or other computing device) is maintained in a fixed position while the user turns her head in different directions to capture images of a region of interest from different angles.

By capturing images at different angles, the images provide a more complete representation of the region of interest. This may be useful for tracking progress on improving skin tone, reducing blemishes or acne lesions, or minimizing the appearance of wrinkles or "crow's feet," which may present themselves in different ways on different portions of the user's face or be more apparent when viewed from different angles. Further adjustments may be made during the image capture process, including electronic or optical image stabilization. In addition, post-processing techniques may be used to improve the utility of source images. For example, individual source images may be stitched into a panoramic image from multiple images in order to increase the effective field of view.

<FIG> is a block diagram that illustrates non-limiting example embodiments of a mobile computing device and a time-lapse video processing computer system according to various aspects of the present disclosure. As discussed above, the mobile computing device <NUM> (a smartphone, a tablet computing device, a laptop computer, or another computing device) is used to capture still images or video of a user <NUM>. The mobile computing device <NUM> transmits the video or still images to the time-lapse video processing computer system <NUM> (e.g., a desktop computing device, a server computing device, a cloud computing device, or another computing device or combination of computing devices), which generates time-lapse video of the user <NUM> or modifies an existing time-lapse video with new images from a current image capture session. The mobile computing device <NUM> and the computer system <NUM> may communicate using any suitable communication technology, such as wireless communication technologies including but not limited to Wi-Fi, Wi-MAX, Bluetooth, <NUM>, <NUM>, <NUM>, <NUM>, and LTE; or wired communication technologies including but not limited to Ethernet, FireWire, and USB. In some embodiments, the communication between the mobile computing device <NUM> and the computer system <NUM> may occur at least in part over the Internet.

In the illustrated embodiment, the mobile computing device <NUM> includes a camera <NUM>, a data collection engine <NUM>, and a user interface engine <NUM>. The camera <NUM> is configured to capture digital source images for subsequent use in generating or modifying time-lapse video. In some embodiments, the mobile computing device <NUM> may include multiple cameras, including but not limited to a front-facing camera and a rear-facing camera. In some embodiments, the front-facing camera (that is, a camera located on a same face of the mobile computing device <NUM> as a display of the mobile computing device <NUM>) is used as the camera <NUM> for capturing images. In other embodiments, a computing device is used in combination with an external digital camera to capture source images.

The data collection engine <NUM> is configured to use the camera <NUM> to capture images of the user <NUM>. The user interface engine <NUM> is configured to present a user interface to the user <NUM>. In some embodiments, the user interface includes a graphical user interface to assist a user in obtaining high-quality images, viewing tutorials that assist the user <NUM> in capturing high-quality images, viewing time-lapse video generated from the images, or the like. In an embodiment, the user interface provides guidance (e.g., in the form of visual guides such as arrows or targets, progress indicators, audio/haptic feedback, synthesized speech, etc.) to encourage a user to continue taking images or to take images from particular angles in order to ensure that sufficient data is collected to synthesize a representative image for a particular time instance. By doing so, the user interface engine <NUM> helps to ensure that representative images are of sufficient quality and consistency to be accurately compared with other synthesized images captured at different time instances.

In the illustrated embodiment, the computer system <NUM> includes a time-lapse video engine <NUM> and an image normalization engine <NUM>. As shown, the computer system <NUM> also includes an image data store <NUM>, in which captured digital source images can be stored until they are needed for further processing. The time-lapse video engine <NUM> is configured to generate time-lapse video based on captured source images.

In an embodiment, a set of digital source images is captured by the mobile computing device <NUM> in a first time period and transmitted by the mobile computing device <NUM> to the computer system <NUM>. In an embodiment, the user interface engine <NUM> provides special instructions or image capture tools for capturing a particular region of interest (e.g., eyes, forehead, neck, etc.).

In an illustrative scenario, the mobile computing device <NUM> (or some other computing device) controls a digital video camera to capture source images as video frames. The captured video is provided as input to a feature detection engine, which detects and tracks facial features over time in the video. The video need not be of a long duration to be effective. For example, the video may capture sufficient feature data over a few seconds to detect and accurately represent facial features. In an embodiment, the video camera remains in a fixed location as the user turns her head in different directions to face the camera at different angles. Taken together, the frames of the video allow facial features to be captured in sufficient detail for making useful comparisons.

In some embodiments, the same features or similar features are detected and tracked in a similar way in subsequent videos, in order to maintain consistency and track changes in facial features over time. In an embodiment, the facial features are parameterized (e.g., using color values, light values, of pixels corresponding to the detected features) to assist in comparisons of features captured at different time instances. For example, comparisons to measure changes in appearance of wrinkles or blemishes over time may involve detecting facial features associated with known wrinklelblemish locations at different time instances and determining light or color values in these areas to allow comparisons over time. These changes can be represented in time-lapse videos in accordance with embodiments described herein.

The time-lapse video engine <NUM> obtains a representative image from the set of digital source images, associating that representative image with the first time period. In an embodiment, feature points are identified in the source images to assist in obtaining the representative images. In an embodiment, the time-lapse video engine identifies feature points in source images as part of the process of obtaining a representative image, such as by selecting an image in which key feature points are of highest quality, averaging image regions containing such feature points to form a composite image. Alternatively, a representative image can be obtained in some other way.

The time-lapse video engine <NUM> combines that representative image with additional representative images from earlier time periods to form a time-lapse video segment. The time-lapse video segment can be annotated with additional features that allow progress to be more easily seen by a user, such as highlighting or outlining areas of interest (e.g., wrinkles, eyelashes, etc.) such that changes in those areas of interest over time can be more easily seen.

In an illustrative scenario, a user applies an eyelash serum onto her eyelashes twice daily for thirty days. Each day, the user captures a series of images of her eyes with a mobile computing device <NUM> and uploads those images to the time-lapse video processing computer system <NUM>. In an embodiment, such images are captured as video or in a still-image burst mode, in which a series of still images are captured (e.g., with a single button push) at a predefined rate, e.g., <NUM> images per second. In an embodiment, the user interface engine <NUM> provides special instructions for capturing eyelash images, such as by instructing the user to hold the camera close to the eye area and to take one set of images with open eyes and one set of images with closed eyes. The time-lapse video engine <NUM> obtains a representative image from the set of digital source images. In an embodiment useful to understand the claimed invention, the obtaining may be by selecting one or more high-quality (e.g., good lighting, limited blurring) open-eye and closed-eye images. According to the claimed invention, the obtaining is by combining selected portions in a composite image, e.g. by combining portions of these images to form a composite image as the representative image. In an embodiment, the representative image undergoes further processing by identifying feature points in the image (e.g., eyes, eyebrows, eyelids, etc.). In an embodiment, such feature points are identified using computer vision techniques such as edge detection, corner detection, and the like, and are tracked across frames of a time-lapse video. These feature points can be used in further processing to assist the user in tracking progress. In this example, feature points can be used to identify the eyelash area and draw a bounding box around the perimeter of the eyelashes. When these bounding boxes are displayed in frames of a time-lapse video, comparisons of the eyelash area in frames of the time-lapse video show the user's progress in growing thicker, longer eyelashes.

The image normalization engine <NUM> is configured to pre-process images before they are provided to the time-lapse video engine <NUM> in order to improve the quality of the time-lapse video. Image normalization of source images may include, for example, performing noise reduction or filtering; adjusting orientation; cropping; adjusting brightness/exposure; adjusting contrast; adjusting contrast. An illustrative image normalization process is described below with reference to <FIG>.

Many alternatives to the arrangements and usage scenarios depicted in <FIG> and <FIG> are possible. For example, although <FIG> illustrates various components as being provided by the mobile computing device <NUM> or the computer system <NUM>, in some embodiments, the arrangement or functionality of the components may be different. For example, in some embodiments, the time-lapse video engine <NUM>, the image normalization engine <NUM>, and the image data store <NUM> may be present on the mobile computing device <NUM> so that the mobile computing device <NUM> can generate time-lapse video without transmitting the images to the computer system <NUM>. In this regard, all of the components illustrated in <FIG> may be provided by a single computing device. As yet another example, in some embodiments multiple computing devices work together to provide the components that are depicted in <FIG> as being provided by the computer system <NUM>.

<FIG> is a flowchart that illustrates a non-limiting example embodiment of a method <NUM> of generating a time lapse video of a region of interest of a live subject according to various aspects of the present disclosure. The method <NUM> is performed by a computer system including one or more computing devices. In an embodiment, the method <NUM> is performed by time-lapse video processing computer system <NUM>. Alternatively, the method <NUM> is performed by some other computer system.

At block <NUM>, a computer system obtains a set of digital source images (e.g., video or still images) of a region of interest of a live subject (e.g., the face of the live subject or portion of the face). In an embodiment, the computer system obtains the set of digital source images by receiving them from a client computing device that captured the set of digital source images, such as the mobile computing device <NUM>, or by retrieving them from some other location. Alternatively, the computer system may obtain the set of digital source images by capturing them directly. The set of digital source images is captured in a first time period (e.g., by a client computing device such as the mobile computing device <NUM>, or by some other computing device). In an embodiment, the set of digital source images is captured at different angles, in different lighting conditions, or with other variations.

At block <NUM>, the computer system normalizes the set of digital source images. In an embodiment, the normalizing function includes one or more of performing noise reduction in one or more of the digital source images; adjusting the orientation of one or more of the digital source images; cropping one or more of the digital source images; adjusting brightness of one or more of the digital source images; or adjusting contrast of one or more of the digital source images, a combination of two or more of these normalization techniques, or other normalization techniques.

At block <NUM>, the computer system obtains a first representative image that includes the region of interest by synthesizing the first representative image from the normalized set of digital source images. The first representative image is associated with the first time period. Obtaining the first representative image includes synthesizing the first representative image by selecting portions of two more of the normalized digital source images and combining the selected portions in a composite image. In an embodiment, the digital source images are taken from different angles, and the computer system synthesizes the first representative image by selecting portions of the images taken from different angles and forming a composite image that includes the selected portions or is derived from the selected portions.

At block <NUM>, the computer system combines the first representative image with one or more additional representative images that include the region of interest of the live subject to form a time lapse video of the region of interest of the live subject. The one or more additional representative images are associated with one or more additional time periods. In an embodiment, the computer system causes the time-lapse video to be displayed or facilitates display of the time-lapse video, such as by transmitting the time-lapse video to a client computing device in response to a request received from the client computing device.

Many alternatives to the process depicted in <FIG> are possible. For example, in some embodiments useful to understand the claimed invention normalization may be performed at some other time or performed by some other computing system or device, such as during an image capture phase at a client computing device. In this regard, normalization is omitted from the method <NUM> in some embodiments not falling within the scope of the claimed invention.

<FIG> is a schematic drawing that illustrates a non-limiting example embodiment of normalization of an image that includes a face of a live subject, according to various aspects of the present disclosure. As shown, the mobile computing device <NUM> has captured an image <NUM> of a subject and is presenting the image <NUM> on a display device of the mobile computing device <NUM>. The image <NUM> includes an off-center face that takes up only a small portion of the overall image <NUM>. As such, it may be difficult to compare the image <NUM> to other images to identify a representative image for this time period. To allow for more accurate comparisons, it may be desirable to reduce the amount of the image that does not include the region of interest, such as by reducing non-face area in the image, and to place the region of interest in a consistent location.

In the example shown in <FIG>, the region of interest is the subject's face. In this example, the image normalization engine <NUM> uses a feature detection algorithm (e.g., a specialized face detection algorithm) to detect the portion of the image <NUM> that depicts the face in a first normalization action <NUM>. In an embodiment, the image normalization engine <NUM> uses the face detection algorithm to find a bounding box <NUM> that includes the face. In a second normalization action <NUM>, the image normalization engine <NUM> alters the image <NUM> to center the bounding box <NUM> within the image <NUM>. In a third normalization action <NUM>, the image normalization engine <NUM> zooms in on the image <NUM> to cause the bounding box <NUM> to be as large as possible within the image <NUM>. In other embodiments, the image normalization engine employs different normalization actions, such as cropping the image to the bounding box <NUM>, reducing or increasing bit depth, undersampling or oversampling pixels of the image, or the like. Such normalization actions are used, in some embodiments, to match other images collected at different time instances, which may be taken with different cameras, at different viewing angles or orientations, or in different lighting conditions.

In general, the word "engine," as used herein, refers to logic embodied in hardware or software instructions, which can be written in a programming language, such as C, C++, COBOL, JAVA™, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Microsoft. NET™, and/or the like. An engine may be compiled into executable programs or written in interpreted programming languages. Software engines may be callable from other engines or from themselves. Generally, the engines described herein refer to logical modules that can be merged with other engines, or can be divided into sub-engines. The engines can be stored in any type of computer-readable medium or computer storage device and be stored on and executed by one or more general purpose computers, thus creating a special purpose computer configured to provide the engine or the functionality thereof.

As understood by one of ordinary skill in the art, a "data store" as described herein may be any suitable device configured to store data for access by a computing device. One example of a data store is a highly reliable, high-speed relational database management system (DBMS) executing on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, any other suitable storage technique and/or device capable of quickly and reliably providing the stored data in response to queries may be used, and the computing device may be accessible locally or may be provided as a cloud-based service. A data store may also include data stored in an organized manner on a computer-readable storage medium, as described further below. One of ordinary skill in the art will recognize that separate data stores described herein may be combined into a single data store, and/or a single data store described herein may be separated into multiple data stores, without departing from the scope of the present disclosure.

<FIG> is a block diagram that illustrates aspects of an exemplary computing device <NUM> appropriate for use as a computing device of the present disclosure. While multiple different types of computing devices were discussed above, the exemplary computing device <NUM> describes various elements that are common to many different types of computing devices. While <FIG> is described with reference to a computing device that is implemented as a device on a network, the description below is applicable to servers, personal computers, mobile phones, smart phones, tablet computers, embedded computing devices, and other devices that may be used to implement portions of embodiments of the present disclosure. Moreover, those of ordinary skill in the art and others will recognize that the computing device <NUM> may be any one of any number of currently available or yet to be developed devices.

In its most basic configuration, the computing device <NUM> includes at least one processor <NUM> and a system memory <NUM> connected by a communication bus <NUM>. Depending on the exact configuration and type of device, the system memory <NUM> may be volatile or nonvolatile memory, such as read only memory ("ROM"), random access memory ("RAM"), EEPROM, flash memory, or similar memory technology. Those of ordinary skill in the art and others will recognize that system memory <NUM> typically stores data and/or program modules that are immediately accessible to and/or currently being operated on by the processor <NUM>. In this regard, the processor <NUM> may serve as a computational center of the computing device <NUM> by supporting the execution of instructions.

As further illustrated in <FIG>, the computing device <NUM> may include a network interface <NUM> comprising one or more components for communicating with other devices over a network. Embodiments of the present disclosure may access basic services that utilize the network interface <NUM> to perform communications using common network protocols. The network interface <NUM> may also include a wireless network interface configured to communicate via one or more wireless communication protocols, such as WiFi, <NUM>, <NUM>, LTE, WiMAX, Bluetooth, Bluetooth low energy, and/or the like. As will be appreciated by one of ordinary skill in the art, the network interface <NUM> illustrated in <FIG> may represent one or more wireless interfaces or physical communication interfaces described and illustrated above with respect to particular components of the system <NUM>.

In the exemplary embodiment depicted in <FIG>, the computing device <NUM> also includes a storage medium <NUM>. However, services may be accessed using a computing device that does not include means for persisting data to a local storage medium. Therefore, the storage medium <NUM> depicted in <FIG> is represented with a dashed line to indicate that the storage medium <NUM> is optional. In any event, the storage medium <NUM> may be volatile or nonvolatile, removable or nonremovable, implemented using any technology capable of storing information such as, but not limited to, a hard drive, solid state drive, CD ROM, DVD, or other disk storage, magnetic cassettes, magnetic tape, magnetic disk storage, and/or the like.

As used herein, the term "computer-readable medium" includes volatile and non-volatile and removable and non-removable media implemented in any method or technology capable of storing information, such as computer readable instructions, data structures, program modules, or other data. In this regard, the system memory <NUM> and storage medium <NUM> depicted in <FIG> are merely examples of computer-readable media.

Suitable implementations of computing devices that include a processor <NUM>, system memory <NUM>, communication bus <NUM>, storage medium <NUM>, and network interface <NUM> are known and commercially available. For ease of illustration and because it is not important for an understanding of the claimed subject matter, <FIG> does not show some of the typical components of many computing devices. In this regard, the computing device <NUM> may include input devices, such as a keyboard, keypad, mouse, microphone, touch input device, touch screen, tablet, and/or the like. Such input devices may be coupled to the computing device <NUM> by wired or wireless connections including RF, infrared, serial, parallel, Bluetooth, Bluetooth low energy, USB, or other suitable connections protocols using wireless or physical connections. Similarly, the computing device <NUM> may also include output devices such as a display, speakers, printer, etc. Since these devices are well known in the art, they are not illustrated or described further herein.

Claim 1:
A method (<NUM>) of creating time lapse video of a live subject (<NUM>), the method (<NUM>) comprising:
obtaining, by a computer system (<NUM>), a set of digital source images of a region of interest of a live subject, wherein the set of digital source images is captured in a first time period; and
normalizing, by the computer system (<NUM>), the set of digital source images;
wherein the method further comprises:
obtaining, by the computer system (<NUM>), a first representative image that is representative of the digital source images of the region of interest captured in the first time period by synthesizing the first representative image from the set of normalized digital source images, including synthesizing the first representative image by selecting portions of two or more of the digital source images captured in the first time period and combining, by the computer system (<NUM>), the selected portions in a composite image; and
combining, by the computer system (<NUM>), the first representative image with one or more additional representative images that are representative of digital source images of the region of interest of the live subject captured in one or more additional time periods to form a time lapse video of the region of interest of the live subject, wherein the one or more additional representative images are selected or synthesized from additional sets of normalized digital source images captured in the one or more respective additional time periods.