Abstract:
A system for facial recognition, consisting of a global database of facial characteristics of known users, a server which accepts from registration pictures taken on a user&#39;s camera device, accepts and processes training pictures and videos of other users to enable algorithm improvement, and a static camera which processes user pictures to determine whether the user is a known user or not based on the global database and a local database associated with each static camera.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention generally relates to the recognition of persons based on their facial characteristics. 
         [0002]    Systems for processing persons based on their identity have several common characteristics. They include a centralized database of information about the person or persons who can be processed, and one or more cameras. For entry systems, these cameras will be in fixed locations, with different lighting between cameras and depending on the time of day, weather conditions, etc. 
         [0003]    There are also ways to fool these systems. One example of that is to hold a picture of the person up to the camera. Another is to have a picture of the person on an article of clothing, such as a t-shirt. 
         [0004]    What is needed is a system for determining the identify of persons which can rapidly identify persons in varying lighting conditions, while overcoming some possible ways of getting around the system. 
       SUMMARY 
       [0005]    A system for facial recognition, consisting of a global database of facial characteristics of known users, a server which accepts from registration pictures taken on a user&#39;s camera device, accepts and processes training pictures and videos of other users to enable algorithm improvement, and a static camera which processes user pictures to determine whether the user is a known user or not based on the global database and a local database associated with each static camera. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0006]      FIG. 1  shows an embodiment of the physical system including that necessary to enroll, train, and detect a known person. 
           [0007]      FIG. 2  shows the details of one embodiment of the components of the system necessary to enroll, train and detect a known person. 
           [0008]      FIG. 3  shows one or more embodiments of the details of the process needed to detect a known person. 
           [0009]      FIG. 4  shows one or more embodiments of the details needed to train the system. 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0010]      FIG. 1  shows one embodiment of the system. A computer  104  is configured to accept pictures from one or more enrollment camera devices  102 , such as cell phones, tablets or computers. The computer will determine facial characteristics of a known person in the picture, and determine the quality of the information. In one or more embodiments, the computer  104  will interact with the user via the enrollment camera device  102 , prompting the user for identification information and verifying that the pictures contain adequate information by making sure that only one person is visible in the picture, and that there are adequate views of the person. In one or more embodiments, an action module  114  is called by the computer  104  based on the identification of the known person and location of the remote camera  110 . For instance, if the remote camera  110  is over a door and the person identified is known by the system to have access to that door, one embodiment of the action module  114  will be to trigger a request to unlock that door. In one or more embodiments, the action module  114  is associated with a database which defines the attributes of the action. For instance, a person could only have access to certain doors, and then only during certain time periods. 
         [0011]    A known person is a person of interest to the system, such as a person who needs access to a facility, or is a member of a group of interest, where the system will notify one or more action modules to react. An other person is a person not of interest but known to the system, whose facial characteristics can be compared to different persons (other or known) to improve the identification rules. In one such embodiment, this could be implements using a defined set of rules. For example, a rule which says if a person appears at a door during a specific time period, then the door is unlocked. Otherwise, the appearance is logged and ignored. 
         [0012]      FIG. 2  shows one embodiment of the system with its software modules. The enrollment manager  202  accepts picture data and known person information from the enrollment camera  204 . In one or more embodiments, the enrollment manager  202  determines the quality of the picture data on several levels. First, is there a single identifiable face in the pictures. Is it difficult to tell whether there is more than one person in the picture? Second, is the picture quality such that facial characteristics are difficult to recognize? Third, are there multiple useful views of the face (direct, left profile, right profile) such that adequately detailed facial characteristic data can be extracted from the information. In one or more embodiments, the features detected in different views are weighted by importance so that the values associated with features face forward get more weight than values associated with profile views, but the profile views still help to improve the calculations. 
         [0013]    In one or more embodiments the software used to define the rules for finding known persons from the facial features uses an algorithm based on the convolution neural network to discover the relationships between facial features and known persons. Convolutional Neural Network is a type of neural network inspired by how the animal visual cortex works. They have a wide range of applications in image and video (see Wikipedia, https://en.wikipedia.org/wiki/Convolutional_neural_network). 
         [0014]    If adequate information can be extracted by the enrollment manager  202 , then the facial characteristics and known person information is saved in the universal database  208 . The universal database  208  contains location independent information about a known person. In one or more embodiments, the universal rules module  216  compares multiple known persons information to create a set of rules to differentiate them. For instance, if there were only two known persons, and one had a ratio of distance between the eyes to distance between the ears of 0.7 and the other was 0.6, one rule might be if the ratio is less than 0.65 it is one known person and greater than 0.6 it is the other. In one or more embodiments, the ratios and values will vary based on the angle of view have a mean and variance, so that one has a probability of whether it is one known person or another. 
         [0015]    At various fixed locations, the system interacts with one or more deployment cameras  214 . Each deployment camera  214  is pointed at a background that will vary during the day in terms of light and objects. For instance, one location may be the entrance of a building that gets natural light during the day and one or more floodlights at night. Another may be in a hallway where different carts, pictures or other objects are stored against the opposite wall. In one or more embodiments, the placement of the deployment camera  214  is such that the expected size and distance of a person can be regulated. For instance, by placing the deployment camera above the height of an average person, then directing the person to stand some specified distance from the camera, the size of the face will not vary from a tall to a short person very much. 
         [0016]    Associated with each deployment camera  214  is a deployment camera manager module  210 . The deployment camera manager module  210  accepts picture data from the deployment camera  214  and removes the background information. This is done by recording the background information on a periodic basis so that it can be removed from the object that has inserted itself into the foreground. In one or more embodiments, the deployment camera manager module  210  adjusts the identification algorithm for changes in lighting. For instance, features may appear thicker in darker lighting and thinner in lighter conditions. The deployment camera manager module  210  identifies the face from the facial characteristics, then aligns the face to get a standard size face to work with. 
         [0017]    Using the universal rules module  216 , the identify of a known person is attempted. In one or more embodiments, a fixed probability threshold is used to determine whether or not this is a known person. On not exceeding that threshold, more pictures are accepted until either some lower threshold is reached or some time limit is exceeded. In one or more embodiments, this lower threshold will be a function of the number of acceptable quality pictures. For instance, there could be a fixed threshold of 95%, a threshold after 5 pictures or less of 90%, after 10 pictures of 90% and a timeout of 5 seconds. 
         [0018]    In one or more embodiments, a per camera adapted database  212  is maintained for each deployment camera  214 . In one or more embodiments, the per camera adapted database  212  is maintained over some period of time or some number of known users. Based on the picture quality and the facial data, this can be used to update rules stored in both the universal database  208  and the per camera adapted database  212 . 
         [0019]    In one or more embodiments, there are two ways to update the universal database from the per camera adapted database. First, calculate an image quality value for each picture based on contrast, and angle of view (profile vs face forward). Add high quality images to the universal database and associate them with the known person, effectively re-enrolling him. 
         [0020]    Second, one can examine a group of persons from a specific deployment camera and, using convolutional neural networks or similar techniques, learn a mapping from the facial features detected by a specific deployment camera to the known person. 
         [0021]    To prevent a user from being accepted by displaying a picture in front of a camera, there are several processing steps that can be used. In one or more embodiments, the deployment camera manager module  210  can look for multiple views of the user, either by examining multiple input frames or requesting that the user turn his head. In other embodiments, the head size can be assumed to be within a certain range such that if it is smaller or larger, it is assumed to be invalid (i.e. trying to hold a picture up to the camera). 
         [0022]    In one or more embodiment, the deployment camera manager module interacts with a display that shows a short phrase for the user to read and examine the movement of the lips using a lip reading module associated with the deployment camera module. In one or more embodiments, the lip reading module tracks the location of landmarks on the lower and upper lip using standard motion based tracking. If the lips motion appears to be in line with utterances of that phrase within a determined probability score, then it is assumed to be a real person. 
         [0023]      FIG. 3  shows one or more embodiments of the process of accepting new picture data. When the picture data is received, a background filter  302  is used to remove the background information from the person image. The person image is scanned to attempt feature recognition  304 . In one or more embodiments, the features are detected and a process of alignment of the features are done to using the universal database  106  and the per camera adapted database  108  to detect what known person(s) it might be. In one or more embodiments, the alignment is done using convolutional neural network techniques to align and center a face. The rules associated with known persons can then be applied to calculate a match score. Based on a calculated match score  306 , the result is to reject it or execute an action  308 . In one or more embodiments, the match score is calculated by computing the distance of aligned features of an image from the new picture data compared to that of the universal database or per camera adapted database. The distance can be computed using either the dot product or euclidean distance between features. Finally, the database is updated through a process called re-enroll  310 , where the new information is compared against a random subset of other known persons to improve the matching algorithms. 
         [0024]    In one or more embodiments, the rules are further improved using training data from various sources, such as a set of photos of a known person, TED™ talks or YouTube™ videos. The training manager module  206  accepts a video or picture input. A user interacts with the training manager to identify the person of interest and provide some identifying information for him. In one or more embodiments, the training manager module will track that person through the video or set of pictures, extracting in some cases hundreds of usable pictures of that person and identifying facial features. That data so extracted can be used against the universal database to further refine rules for finding known persons.  FIG. 4  shows one embodiment of the process of using training data. A video is selected by a user and presented to the training manager  402 . The Training manager module  206  interacts with the user to select a face of interest  404 , after which the training manager module  206  can track the person as he she moves through the video and acquire as many usable images of the face as possible. These images are then processed to recognize features  406 , and the resulting data is compared to random sets of users to improve the rules  408 .