Abstract:
Systems and methods for applying cosmetics are provided using an incoherent light projector shining light on the face, capturing the reflected light using a camera and the projector, communicating with the camera and the projector and a structured light depth processor to generate a depth image output. A control device communicates with the structure light depth sensor to receive the output, to receive the face profiles and generate motion trajectory commands, and a robot communicates with the control device to receive the commands to apply the cosmetics to the face in accordance with the face profiles. Methods for applying the cosmetics include receiving a face profile, receiving a depth sensor input representing a face, extracting face features, matching the face profile to the face features, and generating a guide or outputting robot trajectory to apply the cosmetics.

Description:
BACKGROUND 
       [0001]    The invention relates to systems and methods for three-dimensional (3D) scanning and robotic application of cosmetics to humans. 
         [0002]    Today, many women and men are using decorative and care cosmetics. Cosmetics can be liquid, cream, emulsions, pressed and loose powders, and dispersions. The FDA that regulates use of cosmetics in the United States says that cosmetics are substances applied to the human body “for cleansing, beautifying, promoting attractiveness, or altering the appearance without affecting the body&#39;s structure or functions.” Some examples of cosmetics include skin creams, lotions, powders, lipsticks, perfumes, fingernail and toe polish, eye and facial makeup, hair color or spray. For additional details, see Wikipedia  Cosmetics  (2015), which is incorporated by reference herein. 
         [0003]    Decorative cosmetics used to improve the user&#39;s appearance are often referred to as makeup or make-up. Typically, the wearer of the makeup will apply it to their facial features (e.g., eyebrows, cheeks, or nose) in an elaborate painstaking fashion. Because of the need to safely apply cosmetics on the face, eye area are usually applied by hand either with a makeup tool such as a brush, sponge, or the fingertips. 
         [0004]    Many cosmetics are applied by a finger, a hand or a tool such as sponge held by a hand. Primers are applied using a finger used to reduce pore size, extend wear and permit smooth application of makeup. Lipstick is usually applied with a brush to add color and texture to the lips. Lip gloss adds shine and color. Lip balm moisturizes and protects the lip. Concealer is often applied by finger and is thicker and more solid than foundation and can be used to cover imperfections and contour the nose, cheeks, and jaw. Foundations again are applied with a finger in moist powder, liquid, and spray form smooth imperfections and help make-up last longer. Rouge or blush are often brushed on as a moist powder, cream, and liquid forms can color and define cheeks. Highlights are applied using a finger and can be used to accent facial features. Bronzer can add tan color, glow, and shimmer. Mascara can darken, lengthen, thicken, and draw attention to eyelashes. Eyeliner and eyebrow pencils can enlarge the appearance of eyes and eyebrows. 
         [0005]    Although people refer to beauty advisers at retailers and magazines for the latest style, they apply cosmetics by hand daily or at intervals will splurge and hire a makeup professional or cosmetician to advise and provide facial and body treatments for clients. Daily application by a professional is usually expensive. 
       SUMMARY OF THE INVENTION 
       [0006]    In a feature of the invention, a system is used to apply cosmetics to a face in accordance with face profiles, including an incoherent light projector for shining spatial varying light to the face, at least one camera spaced from the face and the incoherent light projector, to capture reflected light from the face, a structured light depth processor to communicate with the camera and the projector and to generate an depth image output, a control device that communicates with the structure light depth processor to receive the depth image output, to receive the face profiles, and to generate a plurality of motion trajectory commands, and a robot with an applicator, wherein the robot communicates with the control processor to receive the motion trajectory commands to apply the cosmetics with the applicator to the face in accordance with the face profiles. 
         [0007]    In another feature of the invention, a method is performed in a control device of controlling a robot that applies cosmetics to a face, comprising: (a) receiving a face profile; (b) receiving a depth processor input representing a face; (c) extracting a plurality of features of the face; (d) extracting a robot position with respect to the face from the depth sensor input; (e) matching the face profile of step (a) to the features of step (c); (f) generating robot trajectory based on steps (d) and (e); (g) outputting the robot trajectory to the robot to apply the cosmetics; and (h) repeating steps (b) to (g). 
         [0008]    In still another feature of the invention, a method performed in a control device of displaying guides that show how to apply cosmetics to a face by hand, comprising the steps of: (a) receiving a face profile; (b) receiving a depth sensor input representing a face; (c) extracting a plurality of features of the face; (d) matching the face profile of step (a) to the features of step (c); (e) generating guides to apply cosmetics; and (f) outputting the guides on the face to a display. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]      FIG. 1  illustrates a user and an embodiment of the system where the user has no head rest. 
           [0010]      FIG. 2  illustrates hardware architecture of an embodiment of the system. 
           [0011]      FIG. 3  illustrates a method of control for a system using a feedback loop. 
           [0012]      FIG. 4  illustrates a user and an embodiment of the system without showing the frame. 
           [0013]      FIG. 5  illustrates a user and an embodiment of the system where the user has a head rest. 
           [0014]      FIG. 6  illustrates a method of control for a system displaying guides to apply cosmetics. 
           [0015]      FIG. 7  illustrates a user interface that displays and allows selection of face profiles for application of cosmetics. 
           [0016]      FIG. 8  illustrates the operation of the depth sensing using cameras and incoherent light projectors. 
           [0017]      FIG. 9  illustrates a user interface that displays makeup guides and allows selection of cosmetic applicators. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0018]    The following description includes the best mode of carrying out the invention. The detailed description illustrates the principles of the invention and should not be taken in a limiting sense. The scope of the invention is determined by reference to the claims. Each part (or step) is assigned its own part (or step) number throughout the specification and drawings. The method drawings illustrate a specific sequence of steps, but the steps can be performed in parallel and/or in different sequence to achieve the same result. 
         [0019]      FIG. 1  illustrates an embodiment of the system that employs 3D scanning to register the location of facial features. A user&#39;s head  10  is oriented so the face is in front of an incoherent light projector  14  that bathes the user&#39;s face in spatially varying light. Some of that light reflects from the user&#39;s face to a camera  12  and a camera  16  that provide information that tells in 3D the facial features. In operation, ambient light is reduced (e.g., blocked or lights turned off) to a level that prevents saturation of the camera  16 . 
         [0020]    It is not essential to the invention what type of physical structure is used to support the cameras and incoherent light projectors. In the illustrated embodiment, a frame  18  with three panels is used. As shown, the frame  18  has a middle panel between a left and right side panels. Each side panel forms an obtuse angle with the middle panel. The middle panel of the frame  18  supports cameras  16 ,  26  and a user interface  28 . The left panel supports the cameras  12  and  32 , the projectors  14  and  30 . The right panel supports another set of cameras  76  and  78  (See  FIG. 8 ). Alternatively, the frame could be a convex-shaped structure (not shown) that supports and orients the cameras and the projectors toward the user&#39;s face. 
         [0021]    A robotic arm  20  communicating with a control device  42  ( FIG. 2 ) will controllably apply the decorative or care cosmetics (for brevity I refer to both types of cosmetics as makeup in the application) to the user&#39;s face based on face profiles stored in the control device and the information from the 3D scanning that will be described in detail below. 
         [0022]      FIG. 2  illustrates a hardware architecture that implements an embodiment of the system. A control device  42  includes a computer that can communicate with other hardware, computers and data storage. Hennessy and Patterson,  Computer Architecture: A Quantitative Approach  (2012), and Patterson and Hennessy,  Computer Organization and Design: The Hardware/Software Interface  (2013), which are incorporated by reference herein, describe computer hardware and software, storage systems, caching, and networks. 
         [0023]    The processors used in the control device  42  are not essential to the invention and could be any general-purpose processor (e.g. Intel Xeon), a graphic processor (GPU). Each processor can read and write data to memory and/or through a link, e.g., Fibre Channel (FC), Serial ATA (SATA), Serial attached SCSI (SAS), Ethernet, Wi-Fi, to a local storage device  48  or to a network accessible storage device  50 . The storage devices can be a hard disk drive, a disk array, and/or solid state disk. In an embodiment, the network storage device  50  permits the face profiles to be accessed and/or edited by face profile editor  51  over the Internet. The control device  42  controls and communicates with a robot  20 , a user interface  28 , and a structured light depth sensor  41  through USB, Wi-Fi, or Ethernet. The control device  42  executes an operating system such as Linux or Windows. 
         [0024]    In an embodiment, a suitable part for the control device  42  and the storage device  48  is Centaurus Rogue2 Gaming Computer manufactured by Centaurus Computer at 6450 Matlea Court, Charlotte, N.C. 28215. For additional details see www.centauruscomputers.com. 
         [0025]    The structured light depth sensor  41  includes a number of light projectors  36 , a number of cameras  38  and a structured light depth processor  40 .  FIG. 8  further describes the structured light depth sensor  41 . The structured light depth processor communicates with at least one incoherent light projector  36  through USB or VGA and at least one camera  38  through the Mobile Industry Processor Interface (MIPI) specification described in  MIPI Alliance Standard for Camera Serial Interface CSI -2, which is incorporated by reference herein, or other camera interfaces such as Camera Link® or USB. The structured light depth processor  40  may executes a real time operating system (RTOS) such as Wind River, Nucleus, μCOS or another suitable operating system. 
         [0026]    In an embodiment, a suitable part for the structured light depth processor  40  is the Zyng7000 FPGA manufactured by Xilinx, Inc. at 2100 Logic Drive, San Jose, Calif. 95124. 
         [0027]    In an embodiment, a suitable part for the incoherent light projector  36  is the HW8G3 Pico Engine, manufactured by Imagine Optix at 10030 Green Level Church Road, Suite 802-1260, Cary, N.C. 27519. For additional details see www.imagineoptix.com and http://www.picoprojector-info.com. 
         [0028]    In an embodiment, a suitable part for the camera  38  is the LI-VM01CM manufactured by Leopard Imaging Inc. at 1130 Cadillac Court, Milpitas, Calif. 95035. 
         [0029]    In an embodiment, a suitable part for the robot  20  is the Lynxmotion model AL5D 4DOF robotic arm manufactured by Lynxmotion, a RobotShop Inc. company, at 555 VT Route 78 Suite 367, Swanton, Vt. 05488. RobotShop Inc. is in Mirabel, Quebec, Canada. For additional details see www.robotshop.com. 
         [0030]    In an embodiment, a suitable part for the user interface  28  is a touch display such as ME176CX manufactured by ASUS at 800 Corporate Way, Fremont, Calif. 94539. Additional details about implementation of touch screens are described in Wikipedia  Touchscreen  (2015), incorporated by reference herein. 
         [0031]      FIG. 3  illustrates a method that the control device  42  of  FIG. 2  is configured to perform using a feedback loop. At step  52  the control device  42  loads at least one face profile from local storage device  48  or the network storage device  50 . At step  54 , the control device  42  will get depth sensor input from the structured light depth sensor  41 . At step  58 , the control device  42  will extract the position of the robot  20 . At step  56 , in parallel or serially with respect to step  58 , the control device  42  extracts the face features obtained by the structured light depth sensor  41 . At step  60 , the control device  42  will match the face features obtained at step  56  with the face profile obtained at step  52 . The OpenCV Reference Manual—Release 2.4.10.0 at pages 426-427 (2014), describes a suitable extraction and tracking (i.e., matching) algorithm known as the Fast Retina Keypoint (FREAK) algorithm (hereinafter “the OpenCV manual”), which is incorporated by reference. At step  62 , the control device  42  will generate the trajectory of the robot  20 . The algorithm for step  62  can be based on a painting algorithm described in World Academy of Science, Engineering and Technology Vol: 5 2011-11-27, “Development of Roller-Based Interior Wall Painting Robot” or constructed from the Open Motion Planning Library as discussed in Open Motion Planning Library: A Primer Dec. 22, 2014, which are incorporated as reference herein. At step  64 , the control device  42  will output the trajectory of step  62  to the robot  20 , and return (i.e., the feedback loop) to step  54  to repeat the method of  FIG. 3 . 
         [0032]      FIG. 4  illustrates the front view of the face while the robotic arm  20  is applying the makeup to the user&#39;s face. Using the input from the structured light depth sensor  41 , the control device  42  will extract a set of feature points (e.g., feature points  66 ,  68 ,  70 ,  72 , and  74 ) for positioning. The feature points are small unique areas in term of shape and color on the face. For example, the end of the eyebrows (e.g., feature point  68 ), the corner of the nose (e.g., feature point  70 ), or the end of the lip ((e.g., feature point  72  and  74 ) are feature points that can be uniquely identified and located in a face profile. In an embodiment, the control device  42  ( FIG. 2 ) will use feature points to align the loaded face profile to the user&#39;s face for accurate application of makeup either at an initial position or preferably frame by frame. In an embodiment, additional feature points beyond those illustrated in  FIG. 4  is extracted to form a 3D representation of the face. 
         [0033]      FIG. 5  illustrates a system that implements a second embodiment of the invention, which has a head rest. This is the same setup up as in  FIG. 1  with the exception of head rest. When using the head rest, the control device can use the algorithm in  FIG. 3  without the step  58 . The face profile is captured at the beginning of the operation. Once the application starts, the robot completes the operation without further command from the control device. 
         [0034]      FIG. 6  illustrates a method performed in a control device  42  of  FIG. 2  for displaying guides to apply makeup by hand. At step  52 , the control device  42  loads a face profile from local storage device  48  or network storage device  50 . At step  54 , the control device  42  then captures the face profile from the depth sensor  41 . Using the input obtained at the step  54 , the control device  42  extracts the facial feature points. The algorithm for feature extraction and matching is the same as described in the specification of  FIG. 3  and in the OpenCV manual, which is incorporated by reference herein. At step  60 , the control device  42  matches the feature points obtained in step  52  with the feature points obtained in step  54 . At step  61 , the control device  42  generates and displays a makeup guide  122  ( FIG. 9 ) on the face to show where to apply the makeup. When the user brings an applicator within field of view of at least one camera, an applicator location  120  (e.g., a circle or fingertip) is displayed on the face as shown in  FIG. 9 . The control device  42  then continues at step  54 . 
         [0035]      FIG. 7  illustrates the user interface  28  in  FIG. 2 , which is a computer implemented touch screen or GUI that displays face profiles with selectable buttons to permit the user to select which face profile to use for application of makeup. Face profile names  90  are shown. Selection buttons  92  are adjacent to the profile name. Once a user selected a profile  94 , that profile name  98  and the face profile  100  are shown. Then the user starts the application by clicking on the makeup button  96 . When the application is complete, the done message  102  is displayed. 
         [0036]      FIG. 8  illustrates the operation of the depth sensing using cameras and incoherent light projectors. The high resolution depth profile of the face can be acquired by using the incoherent light projectors  14  and  76  with projected area as represented by dotted lines  106  and  107 , and using the cameras  12 ,  16  and  78  with the field of view represented by the solid lines  104  and  105 . Multiple lights from the projectors  14  and  76  can shine on the face. Multiple cameras  12 ,  16 , and  78  acquire images of the face simultaneously. The structured light pattern from the incoherent light projector varies in space as well as in time. For example, a black and white stripe pattern may shift (e.g., left to right) across the field of view in time. The reflection of the face of the head  10  will show varying light intensity. In an embodiment, the light intensity on the face is captured by at least two cameras. Correlations between the pixels of the camera(s) are obtained. In another embodiment, the light intensity on the face is captured by at least one camera. Correlations between the pixels of the camera and the projector are obtained. In either embodiment, from the correlation the distance of the face from the camera can be calculated through triangulation. For additional details, see Wikipedia  Structured Light  3 D Scanner  (2015), which is incorporated by reference herein. 
         [0037]      FIG. 9  illustrates in an embodiment the user interface  28  ( FIG. 2 ) displays face profiles with selectable buttons  110  to permit the user to select a makeup applicator  114 . Once the user selects applicator button  112  as shown, the interface  28  displays an applicator name  116  (e.g., # 2  Blue Pen), a face profile  118 , and a makeup guide  122 . The user starts applying makeup following the makeup guide  122 . When the applicator is within the field of view of at least one of the cameras, the applicator location  120  is shown on the face in the display. The user moves the applicator to follow the makeup guide  122  to apply the makeup. User can then select the next makeup applicator (e.g., # 4  Brush) to continue.