FACIAL ENROLLMENT AND RECOGNITION SYSTEM

A facial recognition system includes a memory for storing a facial image database, wherein the facial image database includes a plurality of entries each corresponding to a different person, and wherein each entry includes a person identifier along with one or more facial images of the person. The facial recognition system further includes a facial recognition module that is operatively coupled to the memory. The facial recognition module is configured to receive a new facial image, and to select one or more facial recognition engines based on one or more facial image parameters of the new facial image, and to use the selected facial recognition engines to compare the new facial image with facial models that are based upon facial images stored in the facial image database in order to identify the person in the new facial image.

BACKGROUND

The present disclosure relates generally to facial recognition systems, and more particularly, to facial recognition systems that employ a facial recognition engine to compare a facial image with a representation that is based upon facial images stored in a facial image database created and maintained by the facial recognition system.

SUMMARY

The disclosure relates to facial recognition systems that employ one or more facial recognition engines to compare a facial image with a representation that is based upon facial images stored in a facial image database created and maintained by the facial recognition system. In some instances, the facial recognition system may be configured to monitor still and/or video sources capturing facial images of individuals within a space, and then utilize one or more facial recognition engines to identify the individuals within the space. In some cases, the facial recognition system may report back to a building automation system with the identity of the individuals seen within the space so that the building automation system may take appropriate action. For example, if the building automation system includes an HVAC system, the HVAC system may change a temperature set point in response to being informed that a particular person has arrived home. If the building automation system includes a security system, the security system may unlock a door of a building in response to being informed that a particular authorized person is present at the door. These are just examples.

A particular example of the disclosure is a facial recognition system that includes an input, an output and a memory for storing a facial image database that includes a plurality of entries each corresponding to a different person, and wherein each entry includes a person identifier along with one or more facial images of the person. The system includes a facial recognition module that is operably coupled to the memory, the input and the output. The facial recognition module is configured to receive a new facial image via the input and to ascertain one or more facial image parameters from the new facial image, and then select a subset of facial recognition engines from a larger set of available facial recognition engines based at least in part on one or more of the ascertained facial image parameters. The ascertained facial image parameters may include, for example, the size of the facial image in pixels, a relative brightness of the facial image, a relative contrast of the facial image, a relative back lighting of the facial image, a relative blurriness of the facial image and/or any other suitable image parameter(s). In some cases, the ascertained facial image parameters may include whether the captured image shows the individual looking directly at the camera, or up or down and/or to the left or to the right. In some cases, the ascertained facial image parameters may include whether and/or how much of the face is obstructed by a hat, hair, glasses or other object. Some facial recognition engines may perform better on facial images under certain facial image parameters than other facial recognition engines. In any event, each of the selected facial recognition engines may compare the new facial image to facial representations that are based upon the facial images in the facial image database to try to identify the person identifier that likely corresponds to the new facial image. The facial recognition module may then send a person ID to a control module via the output, wherein the control module may control one or more building control devices based at least in part on the person ID (e.g. change a setpoint, unlock a door, etc.).

Another example of the disclosure includes a method of recognizing individuals within a building space. Access is gained to a facial image database that includes a plurality of enrolled persons, where the facial image database includes a facial image for each of the plurality of enrolled persons under each of a plurality of different facial conditions. One or more video feeds are monitored that provide images of spaces within the building space, at least some of the images including images of persons within the building space. The one or more video feeds are processed to detect one or more facial images of a person within the building space and one or more facial recognition engines may be selected to compare the detected facial image with facial models that are based on the facial images in the facial image database. Selecting the one or more facial recognition engines is based at least in part on one or more image criteria of the detected facial image. An identified one of the plurality of enrolled persons included in the facial image database that is identified in the detected facial image may be received from the selected one or more facial recognition engines and the identified one of the plurality of enrolled persons may be reported to a building automation system. One or more building control devices of the building automation system may be controlled based at least in part on the identified one of the plurality of enrolled persons.

Another example of the disclosure includes a method of identifying an individual. The method includes monitoring a video feed that provides a series of images of activity in or around a building and extracting one or more images from the series of images of the video feed. The extracted one or more images are analyzed to find facial images, and the facial images are quantified to find a query-able facial image. One or more facial recognition engines are selected based at least in part upon one or more image properties of the query-able facial image. The query-able facial image is sent to the selected one or more facial recognition engines. The selected one or more facial recognition engines are configured to compare the query-able facial image with facial models that are based upon facial images within the facial image database. Facial recognition engine results are provided that include an identity of a person shown within the query-able facial image (if the person is present in the facial image database) and in some cases an associated confidence value. One or more building control devices may then be controlled based at least in part on the identity of the person shown within the query-able facial image.

The above summary of some illustrative embodiments is not intended to describe each disclosed embodiment or every implementation of the present disclosure. The Figures, and Description, which follow, more particularly exemplify some of these embodiments.

DESCRIPTION

The following description should be read with reference to the drawings in which similar structures in different drawings are numbered the same. The drawings, which are not necessarily to scale, depict illustrative embodiments and are not intended to limit the scope of the disclosure.

FIG. 1is a schematic block diagram of an illustrative facial recognition system10that may, for example, be configured to create and update a database of facial images and to use that database of facial images to identify an individual. In cases, the facial recognition system10includes an enrollment module12and a capture module14. While illustrated as separate components, the enrollment module12and the capture module14may individually or in combination be manifested in a controller that may be part of a building automation system (e.g. an HVAC panel such as a thermostat, a security panel, etc.). In some cases, the enrollment module12and/or the capture module14, or at least some functionally of one or both modules, may be manifested in a server or a cloud-based application. In some cases, the enrollment module12and/or the capture module14, or at least some functionally of one or both modules, may be manifested in mobile device such as a tablet computer, laptop computer or smartphone. In some cases, the enrollment module12and/or the capture module14, or at least some functionally of one or both modules, may be manifested in a desktop computer.

The enrollment module12, may, for example, be responsible for creating and maintaining a facial images database16. As will be discussed, the enrollment module12may obtain facial images from a variety of different sources and may in some cases analyze the facial images for quality before storing the facial images in the facial images database16. In some cases, the enrollment module12may be responsible for periodically updating the facial images stored in the facial images database16to account for changing styles (hair styles, facial hair, glasses, etc.), aging and the like of the individuals in the facial images database16.

The capture module14may be responsible for obtaining facial images of individuals to be identified. In some cases, the capture module14may receive live video of a space, and may analyze the live video to find facial images of persons to be identified. Once the capture module14finds one or more facial images of person(s) that are to be identified, the capture module14may provide the one or more facial images to a facial recognition module18. One of the tasks of the facial recognition module18is to determine which of a variety of different facial recognition engines20are to be used to query the facial images database16in order to identify the persons in the one or more facial images. In some cases, which facial recognition engine20to use may be determined at least in part by one or more image parameters of the one or more facial images. In some cases, the facial recognition module18includes a facial recognition engine evaluation module22that determines which of the facial recognition engines20are to be used. In some cases, a single facial recognition engine20may be used. In other situations, two, three or more distinct facial recognition engines20may be used. InFIG. 1, the facial recognition engines20include an ENGINE #1 labeled as24, an ENGINE #2 labeled as26through an ENGINE #N labeled as28.

The facial recognition engine evaluation module22may analyze a facial image to determine one or more image parameters of the facial image, and may utilize the one or more image parameters to determine which of the facial recognition engines20should be used in order to identify the person shown in the facial image. In some instances, the facial recognition engine evaluation module22(or the facial recognition module18itself) may evaluate multiple images of a particular individual (e.g. multiple images of a video sequence) to determine which of the multiple images are most likely to provide good results. In some cases, the image parameters may include, for example, the size of the facial image in pixels, a relative brightness of the facial image, a relative contrast of the facial image, a relative back lighting of the facial image, a relative blurriness of the facial image, and/or any other suitable image parameter(s). In some cases, the image parameters may include whether the captured image shows the individual looking directly at the camera, or up or down and/or to the left or to the right. In some cases, the image parameters may include whether and/or how much of the face is obstructed by a hat, hair, glasses or other object. The engine evaluation model may store a table that maps certain image parameters to certain facial recognition engines20.

In some cases, a specific facial recognition engine20may be selected if the image parameters fall within a range suitable for that facial recognition engine20. The suitable range thresholds may be selected manually based on engine specifications, or may be based upon prior testing performed on a sample of facial images. In some cases, the suitable range thresholds may also be selected automatically, and may be adjusted over time, based on comparison of different facial recognition engine results on facial images processed by the system. Examples are shown in Table One, below:

It will be appreciated that these are just examples, and a variety of other image parameters may be considered. For each facial recognition engine, the input parameters may be described as a range, as specific values, or a specific set of conditions. In some cases, a weighed sum combining various parameters may also be considered. For example, if the face size for a particular facial image is between 40 and 90 pixels, facial sharpness needs to be greater than or equal to 0.8; and if the face size is greater than 90 pixels, facial sharpness needs to be greater than or equal to 0.7.

A variety of commercially available facial recognition engines may be utilized. Examples of suitable cloud-based APIs (Application Programming Interface) for facial recognition include but are not limited to Microsoft Face API and Amazon Rekognition. Examples of suitable facial recognition engines that may be employed as on-site software or be integrated into other products include but are not limited to NEC Face Recognition, MorphoArgusand Cognitec.

Once the facial recognition engine evaluation module22determines which of the facial recognition engines20to select (often a subset of a larger set of available facial engines20), the facial recognition module18may provide the selected facial recognition engine(s)20with the facial image. In some cases, one or more of the facial recognition engines20may be hosted on a remote server, but this is not required. Upon querying, the selected facial recognition engine(s)20may return an identity of the person shown in the facial image(s) that was sent to the selected facial recognition engine(s)20. The selected facial recognition engine(s)20may also return a confidence value that provides an indication of how confident (i.e. likely) that the identity of the person is correct. For example, if the best available facial image sent to the selected facial recognition engine(s) is poorly lit and blurry, the confidence value may be relatively low. Conversely, if the best available facial image sent to the selected facial recognition engine(s)20is well-light and clear, and is an image of the individual looking directly or nearly directly at the camera, the confidence value may be relatively high.

FIG. 2is a schematic block diagram of an illustrative enrollment module12. As noted, a function of the enrollment module12is to obtain facial images that may be placed in the facial images database16(FIG. 1) and subsequently used to build or update a facial model based upon facial images in the facial images database16in order to identify persons in captured facial images. In some instances, the enrollment module12includes an image input module30. The image input module30may obtain facial images from a variety of different sources. Examples of suitable image sources include a selfies module32, a photos module34, a captured images module36and a social media module38. In some cases, the selfies module32may instruct individuals who are expected to be in the building space and are enrolling in the facial recognition system to take a series of selfies. For example, the individuals may be instructed to take and upload selfies showing themselves, or at least their faces, looking directly at the camera, looking left, looking right, looking up and looking down. In some cases, the selfies module32may instruct the individual to take multiple selfies with their hair up and their hair down, with and without facial jewelry like earrings, nose piercings, lip piercings, and/or various glasses, for example. In some cases, the selfies module32may be implemented, at least in part, on a mobile device such as a smartphone or tablet computer, but this is not required.

In some cases, the photos module34may be configured to go through online and/or otherwise electronic photo libraries looking for suitable facial images. These can include photo libraries stored on a personal computer, on the cloud, on a mobile device, and/or any other device. The photos module34may assemble multiple facial images for a particular individual, may display the multiple facial images, and ask the individual to confirm that each of the images is in fact of that individual. In some instances, the captured images module36may include or otherwise be operably coupled with a still camera, a video camera and the like, and may capture facial images as individuals move about the space. The captured images module36may compile these images, and in some cases may ask the individuals to confirm their identity.

The social media module38may scan social media accounts, such as but not limited to Facebook, Snapchat and the like, looking for suitable facial images of an individual. In some cases, the social media module38may display the found images and ask for identity confirmation.

In some cases, the enrollment module12may include an image quality assessment module40that receives facial images from the image input module30and analyzes the received facial images to confirm that the images are of sufficient quality to be of use. The facial images that are believed to be of sufficient quality, and represent a suitable variety of poses and images (looking left, looking right, etc.) may be passed on to an image organization module42. In some cases, a particular facial image pose may be determined to be of less-than sufficient quality, and the image quality assessment module40may ask the image input module30to obtain a higher quality facial image of that particular facial image pose if possible.

Facial images that are deemed to be of sufficient quality, and of appropriate facial poses, are forwarded to the image organization module42. In some cases, the image organization module42may at least partially contribute to the organization of facial images within the facial images database16(FIG. 1). Facial images may be organized in any suitable manner. In some cases, for example, the facial images for a particular individual may include images of the individual looking directly at the camera, images of the individual looking above the camera, images of the individual looking below the camera, images of the individual looking to the left of the camera, and images of the individual looking to the right of the camera. In some cases, facial images for a particular individual may be organized by whether they are wearing their hair up or down, have facial hair or are clean-shaven, whether or not they are wear jewelry, glasses and the like. In some cases, facial images for a particular individual may also be organized by the size of the facial image in pixels, the relative brightness of the facial image, the relative contrast of the facial image, the relative back lighting of the facial image, the relative blurriness of the facial image and/or by any other suitable image parameter. These are just examples.

FIG. 3is a schematic block diagram of the capture module14. While the capture module14is configured to capture facial images of individuals within a space so that they can be identified, in some cases, the capture module14may also assist the enrollment module12in initially capturing facial images for populating the facial images database16(FIG. 1). In some cases, the capture module14includes a video capture module50. In some instances, the video capture module50may be operably coupled to one or more still cameras and/or video cameras that are distributed within a space. In some cases, still images may be captured by the still cameras and/or from image frames captured by the video cameras. In some cases, still images may be captured 30 times (frames) per second using a video camera, although this is just an example.

The still images may be forwarded to a face detection module52, which analyzes the still images looking for facial images. Once a possible facial image is detected, in some cases, subsequent still images are analyzed by a face tracking module54looking for confirmation the individual is still there and/or looking for better quality facial images of that individual for subsequent identification. A face image evaluation module56may review the facial images to ascertain whether and which of the captured image(s) are of sufficient quality and/or pose to be of use in identifying the individual shown in the captured image(s).

FIG. 4is a schematic block diagram of an illustrative facial recognition system60. The facial recognition system60includes an input62and an output64. A memory67may be configured to store the facial images database16. In some cases, the facial images database16includes a plurality of entries each corresponding to a different person, and each entry includes a person identifier along with one or more facial images of the person. In some instances, the facial images database16may include multiple facial images for each person identifier, with some of the multiple facial images representing the person at one or more of different facial angles, different facial lighting, different facial size in terms of pixels, and different facial obstructions. Examples of different facial obstructions include but are not limited to differing hair style, wearing glasses, not wearing glasses, wearing a hat, not wearing a hat, and differing ages.

The facial recognition module18is operably coupled to the input62, the output64and to the memory67and is configured to receive a new facial image via the input62and to ascertain one or more facial image parameters from the new facial image. The facial recognition module18is configured to select a subset of facial recognition engines20(FIG. 1) from a larger set of available facial recognition engines20(FIG. 1) based at least in part on one or more of the ascertained facial image parameters. The ascertained facial image parameters may include, for example, the size of the facial image in pixels, a relative brightness of the facial image, a relative contrast of the facial image, a relative back lighting of the facial image, a relative blurriness of the facial image and/or any other suitable image parameter(s). In some cases, the ascertained facial image parameters may include whether the captured image shows the individual looking directly at the camera, or up or down and/or to the left or to the right. In some cases, the ascertained facial image parameters may include whether and/or how much of the face is obstructed by a hat, hair, glasses or other object.

At least some of the facial recognition engines20may include cloud-based facial recognition engines, but this is not required. In some cases, the selected subset of facial recognition engines20may include two or more distinct facial recognition engines20. In some cases, the selected subset of facial recognition engines20may include only a single facial recognition engine20.

Each of the facial recognition engines20is configured to compare the new facial image to facial models or other facial representations that are based upon the facial images in the facial image database16and to identify a person identifier that likely corresponds to the new facial image. The facial recognition module18may be configured to evaluate the person identifiers and confidence levels returned by the selected facial recognition engines20. If the returned person identifiers are the same, the facial recognition module18will as sign a high confidence to the output person ID. However, if the returned person identifiers differ, the facial recognition module18may output the person ID with the highest combined confidence, or may select additional facial recognition engines to evaluate the new facial image. In case of disagreement, the output person ID will be assigned lower confidence. The facial recognition module18is configured to send a person ID to a control module66via the output64, wherein the control module66is configured to control one or more building control devices68,70based at least in part on the person ID. In some cases, the facial recognition module18may be further configured to process the person identifiers identified by each of the subset of facial recognition engines20to determine the person ID that is sent to the control module66.

In some cases, at least one of the facial recognition engines20may provide a confidence level of the person identifier that likely corresponds to the new facial image. In some cases, particularly if the new facial image was sent to multiple facial recognition engines20for identification, the facial recognition module18may be configured to determine a confidence level in the person ID that is based at least in part on the confidence level of the person identifier provided by each of one or more of the subset of facial recognition engines20. In some cases, if the confidence level in the person ID is below a threshold confidence level, the facial recognition module18may select a different facial recognition engine20, or a different subset of facial recognition engines20, and may try again. This may be repeated until an acceptable confidence level is achieved. If an acceptable confidence level cannot be achieved, the facial recognition module18may report to the output module64that an unknown person was seen in the new facial image.

FIG. 5is a schematic illustration of an illustrative facial images database116. In some cases, the facial images database116may be considered as being an illustrative but non-limiting example of the facial images database16. It will be appreciated that the information within the facial images database16may be organized in any suitable fashion. The facial images database116includes a plurality of facial images that are organized by individual. To illustrate, the facial images database116may include an INDIVIDUAL #1 labeled as118, an INDIVIDUAL #2 labeled as120through an INDIVIDUAL #P labeled as122. A number of facial images are organized underneath each individual118,120,122. As illustrated, the individual118includes an IMAGE #1 labeled as124, an IMAGE #2 labeled as126through an IMAGE #M labeled as128. Similarly, the individual120includes an IMAGE #1 labeled as134, an IMAGE #2 labeled as136through an IMAGE #M labeled as138and the individual122includes an IMAGE #1 labeled as144, an IMAGE #2 labeled as146through an IMAGE #M labeled as148.

In some cases, the images for each individual118,120,122may be organized in a similar fashion. For example, the image124may represent a straight on view of the individual118, the image134may represent a straight on view of the individual120and the image144may represent a straight on view of the individual122. The images126,136,146may represent left profiles of the individuals118,120,122, respectively. These are just examples. It will be appreciated that the various views of each individual118,120,122, and perhaps views with and without facial obstructions and/or other characteristics, may be organized in a similar manner. In some cases, facial images for a particular individual may be organized by whether they are wearing their hair up or down, have facial hair or are clean-shaven, whether or not they are wear jewelry, glasses and the like. In some cases, facial images for a particular individual may be organized by the size of the facial image in pixels, the relative brightness of the facial image, the relative contrast of the facial image, the relative back lighting of the facial image, the relative blurriness of the facial image and/or by any other suitable image parameter. These are just examples.

FIG. 6is a flow diagram showing an illustrative method150of recognizing individuals within or around a building space. As generally shown at block152, access is gained to a facial image database (such as the facial images databases16,116) that includes a plurality of enrolled persons, where the facial image database includes a facial image for each of the plurality of enrolled persons under each of a plurality of different facial conditions. In some cases, the plurality of different facial conditions include two or more of the person looking up, the person looking down, the person looking to the left, the person looking to the right and the person looking straight ahead.

In some cases, the facial image database organizes the facial images for each of the plurality of enrolled persons at each of a plurality of different facial conditions into predetermined separate categories. Examples of the separate categories include one or more of the person with glasses, the person without glasses, the person with their hair worn up, the person with their hair worn down, the person clean shaven, the person not clean shaven, the person wearing jewelry, the person not wearing jewelry, and the person wearing a hat, the person not wearing a hat and the person wearing a scarf. In some instances, the separate categories include one or more of the person looking to the left, the person looking to the right, the person looking up, the person looking down, and the person looking straight ahead. In some cases, the separate categories may include the size of the facial image in pixels, the relative brightness of the facial image, the relative contrast of the facial image, the relative back lighting of the facial image, the relative blurriness of the facial image and/or by any other suitable image parameter. These are just examples.

One or more video feeds providing images of spaces within or around the building space may be monitored, at least some of the images include images of persons within the building space, as seen at block154. As seen at block156, the one or more video feeds may be processed to detect one or more facial images of a person within or around the building space. One or more facial recognition engines may be selected, as generally indicated at block158. In some cases, selecting the one or more facial recognition engines is based at least in part on one or more image criteria of the one or more detected facial images. The one or more image criteria may include, for example, the size of the facial image in pixels, a relative brightness of the facial image, a relative contrast of the facial image, a relative back lighting of the facial image, a relative blurriness of the facial image and/or any other suitable image parameter(s). In some cases, the one or more image criteria may include whether the captured image shows the individual looking directly at the camera, or up or down and/or to the left or to the right. In some cases, the one or more image criteria may include whether and/or how much of the face is obstructed by a hat, hair, glasses or other object. Some facial recognition engines may perform better on facial images under certain image criteria than other facial recognition engines.

As seen at block160, an identified one of the plurality of enrolled persons included in the facial image database that is identified in the one or more detected facial images may be received from the selected one or more facial recognition engines. The identified one of the plurality of enrolled persons may be reported to a building automation system, as generally seen at block162, and one or more building control devices of the building automation system may be controlled based at least in part on the identified one of the plurality of enrolled persons, as indicated at block164.

In some cases, the building automation system includes an HVAC system, and the building control device may include a building control user interface device that allows the identified one of the plurality of enrolled persons to change one or more building control parameters only when the identified one of the plurality of enrolled persons has been granted permission to change one or more building control parameters. In some instances, the building automation system includes an access control system, and a building access device is controlled to allow entry of the identified one of the plurality of enrolled persons only when the identified one of the plurality of enrolled persons has been granted permission to enter.

FIG. 7is a flow diagram illustrating a method170of identifying an individual. In some cases, the method170includes creating a facial images database by soliciting facial images of each of a plurality of enrolled persons under each of a plurality of facial conditions that include one or more of the person looking to the left, the person looking to the right, the person looking up, the person looking down, and the person looking straight ahead. A video feed that provides a series of images of activity in or around a building is monitored, as indicated at block174. One or more images may be extracted from the series of images of the video feed, as seen at block176. The extracted one or more images may be analyzed to find facial images, as indicated at block178, and are quantified to find a query-able facial image, as noted at block180.

As indicated at block182, one or more facial recognition engines may be selected based at least in part upon one or more image properties of the query-able facial image and the query-able facial image may be sent to the selected one or more facial recognition engines as indicated at block184, where the selected one or more facial recognition engines are configured to compare the query-able facial image with facial models that are based upon facial images stored within the facial image database. As noted at block186, facial recognition engine results that include an identity of a person shown within the query-able facial image as well as an associated confidence value may be provided. In some cases, and as indicated at block188, one or more building control devices may be controlled based at least in part on the identity of the person shown within the query-able facial image.