Multimodal biometric authentication system and method with photoplethysmography (PPG) bulk absorption biometric

A multimodal biometric authentication system utilizes a bulk absorption characteristic of human tissue that is measurable using a photoplethysmography (PPG) sensor. One disclosed method of operation includes extracting bulk absorption features from biometric data obtained using a PPG sensor and generating a first biometric template. Additional biometric features are also extracted from biometric data obtained using a second biometric sensor and a second biometric template is also generated. An authentication output signal is provided in response to the first biometric template matching a first stored corresponding enrolled biometric template and the second biometric template matching a second stored corresponding enrolled biometric template.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to biometric authentication systems and more particularly to biometric fusion (also referred to as “multimodal”) authentication systems that use two or more biometrics or that use two or more algorithms applied to the same biometric data.

BACKGROUND

Biometric authentication systems offer a quick and convenient means of personal authentication without the need to remember and enter passwords. Various forms of biometric authentication have been developed and each form presents its own unique set of challenges. Generally speaking, biometric authentication systems are subject to spoofing or falsing.

For example, fingerprint biometric authentication systems can sometimes be subject to falsing by an unauthorized person's finger or may even be “fooled” by a synthetic false finger. This susceptibility to falsing is an issue with many types of biometric authentication or identification systems. Because of this susceptibility, the concept of “biometric fusion” or “multimodal biometric” systems is gaining in popularity. In such multimodal systems, it becomes more difficult to spoof or fake the biometric inputs because there is either more than one biometric required, or multiple algorithms are applied to the same biometric, and therefore the probability of spoofing to gain unauthorized access is significantly reduced.

DETAILED DESCRIPTION

The present disclosure provides multimodal biometric authentication systems utilizing a bulk absorption characteristic of human tissue that is measurable using a photoplethysmography (PPG) sensor. The bulk absorption characteristic can be measured using reflective PPG techniques and sensors incorporated into wearable devices, mobile device or combinations thereof.

An aspect of the present disclosure is a method that includes extracting bulk absorption features from biometric data obtained using a PPG sensor and generating a first biometric template. Additional biometric features are also extracted from biometric data obtained using a second biometric sensor and a second biometric template is also generated. Accordingly, an authentication output signal is provided in response to the first biometric template matching a first stored corresponding enrolled biometric template and the second biometric template matching a second stored corresponding enrolled biometric template. In some embodiments, the second biometric sensor may be a second PPG sensor that operates in a different spectral range from the first PPG sensor. In such embodiments, a multimodal biometric authentication system may be implemented by requiring the bulk absorption characteristics, measured within different spectral ranges, to match in order to provide an authentication output signal. In yet other embodiments, the combined bulk absorption characteristics at two or more spectral ranges may be required, along with another different biometric.

In another aspect of the present disclosure, a method includes extracting bulk absorption features from biometric data obtained using a PPG sensor in a first device and generating a first biometric template in the first device; sending a first authentication output signal, from the first device to a second device over a wireless link, in response to the first biometric template matching a first stored corresponding enrolled biometric template stored in the first device; extracting biometric features from biometric data obtained using a second biometric sensor in a second device and generating a second biometric template in the second device; and providing a second authentication output signal in response to receiving the first authentication output signal from the first device, and the second biometric template matching a second stored corresponding enrolled biometric template stored in the second device.

Another aspect of the present disclosure is a multimodal biometric authentication apparatus that includes a PPG sensor; a second biometric sensor; non-volatile, non-transitory memory, containing at least two enrolled biometric templates, a first enrolled biometric template related to a bulk absorption biometric and a second enrolled biometric template related a biometric measurable by the second biometric sensor; and a processor, operatively coupled to the PPG sensor, to the second biometric sensor, and to the memory.

The processor is operative to extract bulk absorption features from biometric data obtained using the PPG sensor and generate a first biometric template; extract biometric features from biometric data obtained using the second biometric sensor and generate a second biometric template; and provide an authentication output signal in response to the first biometric template matching the first enrolled biometric template and the second biometric template matching the second enrolled biometric template. In some embodiments, the second biometric sensor may be a second PPG sensor having a spectral range different than the first PPG sensor.

Another aspect of the present disclosure is a wearable device that includes a PPG sensor; a second biometric sensor; a transceiver; and a controller, operatively coupled to the PPG sensor, to the second biometric sensor, and to the transceiver, The controller is operative to obtain biometric data collected using the PPG sensor and using the second biometric sensor; and send the biometric data obtained using the PPG sensor and using the second biometric sensor to a second device over a wireless link using the transceiver.

The wearable device may be included in a multimodal biometric authentication system along with a mobile device. The mobile device includes a transceiver; a controller, operatively coupled to the transceiver; non-volatile, non-transitory memory, containing at least two enrolled biometric templates, with a first enrolled biometric template related to a bulk absorption biometric and with a second enrolled biometric template related to a biometric measurable by the second biometric sensor in the wearable device; and data processing logic, operatively coupled to the controller, The data processing logic is operative to extract bulk absorption features from biometric data obtained using the PPG sensor in the wearable device and generate a first biometric template; extract biometric features from biometric data obtained using the second biometric sensor in the wearable device and generate a second biometric template; and provide an authentication output signal in response to the first biometric template matching the first enrolled biometric template and the second biometric template matching the second enrolled biometric template.

Another aspect of the present disclosure is a wearable device that includes a PPG sensor; a second biometric sensor; a transceiver; and data processing logic, operatively coupled to the PPG sensor and to the second biometric sensor. The data processing logic is operative to extract bulk absorption features from biometric data obtained using the PPG sensor and generate a first biometric template; and extract biometric features from biometric data obtained using the second biometric sensor and generate a second biometric template. A controller is operatively coupled to the data processing logic, and to the transceiver. The controller is operative to send the first biometric template and the second biometric template to a second device over a wireless link using the transceiver.

The wearable device may be included in a multimodal biometric authentication along with a mobile device. The mobile device includes a transceiver; non-volatile, non-transitory memory, containing at least two enrolled biometric templates: a first enrolled biometric template related to a bulk absorption biometric and a second enrolled biometric template related to a biometric measurable by the second biometric sensor in the wearable device. A controller is operatively coupled to the transceiver and to the memory. The controller is operative to provide an authentication output signal in response to the first biometric template matching the first enrolled biometric template and the second biometric template matching the second enrolled biometric template. The controller is further operative to send the authentication output signal to either the wearable device or to an external third device over a wireless link using the transceiver in the mobile device.

Another aspect of the present disclosure is a multimodal biometric authentication system that includes a first device and a second device. The first device and the second device each include a biometric sensor; a transceiver; non-volatile, non-transitory memory that contains at least one enrolled biometric template, with at least one of the first or the second devices having a first enrolled biometric template related to a bulk absorption biometric and the other of the first or second devices having a second enrolled biometric template; and data processing logic, operatively coupled to the biometric sensor.

The data processing logic is operative to extract bulk absorption features from biometric data obtained using the biometric sensor and generate a first biometric template. A controller is operatively coupled to the data processing logic, and to the transceiver. The controller is operative to provide an authentication output signal in response to the biometric template matching the at least one enrolled biometric template; and send the authentication output signal to the other device over a wireless link using the transceiver.

In embodiments of the multimodal biometric authentication system, the biometric sensor of the first device is a PPG sensor; and the biometric sensor of the second device may be either a microphone, a fingerprint sensor, a palm reader, or an iris scanner. In other embodiments, the biometric sensor of the first device is a PPG sensor having a first spectral range and the biometric sensor of the second device is a PPG sensor having a second spectral range.

Turning now to the drawings wherein like numerals represent like components,FIG. 1is a block diagram of a multimodal biometric authentication system100that includes a photoplethysmography (PPG) bulk absorption biometric in accordance with an embodiment. The authentication system100includes a PPG sensor101and a second biometric sensor103which may be any of various types of biometric sensors. For example, the second biometric sensor103may be a fingerprint reader, a palm print reader, an iris scanner, one or more microphones operatively coupled to a voice recognition system, or a second PPG sensor in some embodiments. The PPG sensor101and the second biometric sensor103are operatively coupled to a processor105. The various components and devices that are described herein as being “operatively coupled” refers to such operative coupling as that having, in some embodiments, one or more intermediate or intervening components that may exist between, or along the connection path between two such components such that the components are understood to be operatively coupled in that data or commands or control signals can be sent from one to the other and vice versa.

The processor105obtains raw sensor data from each biometric sensor and extracts the relevant biometric features. The processor105is operative to use the extracted biometric features to create corresponding biometric templates for comparison with enrolled biometric templates stored in an enrolled templates database107. The database107contains one or more enrolled biometric templates related to one or more users of the multimodal biometric authentication system100. An enrolled biometric template is a biometric template that a user has voluntarily provided to the authentication system, using a separate biometric template enrollment security procedure, to verify their identity and make an association between the enrolled biometric template and the user's identity. A biometric template represents a signature related to a biological characteristic of an individual, for example, a person's bulk absorption characteristic for their skin tissue. Such biological characteristics may be measured at various points on the human body for example, at a fingertip or on the wrist or ankle. Data processing logic may then generate a biometric template from obtained biometric sensor data and compare it with the enrolled biometric template and determine whether the templates match.

If a match is found between both of the generated templates with corresponding enrolled template stored in the enrolled templates database107, then the processor105will produce an authentication output signal109which will enable access to the protected system111. The multimodal authentication system100is referred to as “multimodal” in that both the PPG sensor101and the second biometric sensor103must produce a match with the enrolled templates database107. The biometric features extracted from the PPG sensor101that are used to generate the biometric template is limited to the bulk absorption of the user which is related to a DC component of the PPG sensor101output. In other words, only one of three possible components of the PPG data is utilized.

A PPG signal may be considered to consist of three components; a component related to arterial blood volumetric changes (which can be related to heart activity); a component related to venous blood volume changes (which is a slow rate signal that modulates the overall PPG signal); and a DC component which may be related to the optical properties of the biological tissue. These optical properties include, for example absorptivity and reflectivity which may be mathematically modeled as transmission and reflection coefficients within the spectral frequency range of interest.

A method of operation of the system shown inFIG. 1is illustrated in the flowchart ofFIG. 2. The method of operation begins, and in operation block201the processor105obtains first biometric data using PPG sensor101. In operation block203, the processor105extracts bulk absorption features from the first biometric data and generates a first biometric template. In operation block205, the processor105obtains second biometric data from second biometric sensor103. In operation block207the processor105extracts second biometric features from the second biometric data and generates a second biometric template. In operation block209, the processor105compares the generated first biometric template and the generated second biometric template to respective enrolled biometric templates contained in the database107. In decision block211, the processor105checks whether both of the first biometric template and the second biometric template form a match with templates contained in the database107. If a match is found, then in operation block215the processor105provides the authentication output signal109and the method of operation ends as shown. If a match with the database107is not found in decision block211, then the processor105denies access to the protected system111as shown in operation block213, and the method of operation terminates.

FIG. 3is an example plot of photoplethysmography sensor voltage versus LED current which corresponds to bulk absorption for various users. The PPG sensor101will include at least one LED and a photodiode. In some embodiments, the PPG sensor101will include multiple LEDs and multiple photodiodes. TheFIG. 3example plot may be considered to correspond to a single LED and photodiode pair in a biometric PPG sensor used to obtain the measurement. Each one of the curves shown in the example plot is related to a specific user's bulk absorption characteristic. As can be seen from the example plot, individual users have bulk absorption characteristics that are different enough from each other such that a determination of which user is which can be ascertained. This PPG bulk absorption characteristic information is therefore useful in combination with other biometric data in a multimodal biometric authentication system as discussed briefly above with respect toFIG. 1andFIG. 2.

Another type of multimodal biometric system that can be implemented using the PPG bulk absorption characteristic illustrated in the example plot ofFIG. 3is a system using PPG sensor measurements within different spectral ranges. For example, PPG measurements may be taken within a red spectral range and another set of PPG measurements may be taken within, for example, a green spectral range. A red spectral range (i.e. infrared) is often used for transmission PPG system measurements because red light achieves better penetration through human tissue. In contrast, a green spectral range is often used for reflective PPG measurements because green does not have the ability to penetrate through human tissue as well and therefore more of the green spectral light applied is reflected from layers of the tissue. Therefore by taking PPG measurements using two different spectral ranges a different PPG characteristic template can be created for each spectral range. Therefore, in one example embodiment related toFIG. 1, the second biometric sensor103may be a second PPG sensor having a spectral range different from that of PPG sensor101. In other embodiments, only one PPG sensor may be used where the PPG sensor has the capability of using multiple spectral ranges.

FIG. 4is a flowchart showing a method of operation using either a single multispectral PPG sensor or using a first PPG sensor having a first spectral range and a second PPG sensor having a second spectral range in accordance with an embodiment. The method of operation will be described with respect to the multimodal biometric authentication system100shown inFIG. 1with the second biometric sensor103assumed to be a second PPG sensor with a spectral range that differs from PPG sensor101. The method of operation begins, and in operation block401, the processor105obtains first biometric data using the PPG sensor101in a first spectral range. In operation block403, the processor105extracts bulk absorption features from the first biometric data and generates a first bulk absorption biometric template. In operation block405, the processor105obtains second biometric data using a second PPG sensor (i.e. biometric sensor103in this example) in a second spectral range. In operation block407, the processor105extracts bulk absorption features from the second biometric data and generates a second bulk absorption biometric template. In operation block409, the processor105compares the first bulk absorption biometric template and the second bulk absorption biometric template to respective enrolled bulk absorption biometric templates in the database107.

In decision block411, the processor105checks whether both of the first bulk absorption biometric template and the second bulk absorption biometric template match enrolled templates within the database107. If a match is detected, then in operation block415the processor105provides the authentication output signal109to allow access to the protected system111and the method of operation terminates as shown. If a match is not detected between one or the other, or both, of the biometric templates and an enrolled template in the database107, then access is denied as shown in operation block413and the method of operation also terminates. In other words, both PPG characteristics taken at the different spectral ranges must each match respective enrolled biometric templates in order for the authentication output signal109to be provided.

Some additional example embodiments that employ the bulk absorption biometric characteristic along with a second biometric will now be described. It is to be understood that in the various embodiments, the second biometric may be a second bulk absorption biometric characteristic measured within a spectral range different from the first bulk absorption biometric characteristic as described above with respect toFIG. 4. It is also to be understood that other biometrics other than bulk absorption biometric characteristics may be used such as, but not limited to, fingerprint detection, voice print detection, an iris scan, body odor detection, body heat signature detection or some other biometric.

A first such example embodiment is illustrated in the block diagram ofFIG. 5. An example multimodal biometric authentication system includes a first device510having a PPG sensor501and a second biometric sensor502, and a second device520that includes authentication data processing logic515. The first device510includes a transceiver507operatively coupled to the controller505. The controller505is operatively coupled to a data collector503which is further operatively coupled to the PPG sensor501and to the second biometric sensor502. The data collector503is operative to obtain raw sensor data from the PPG sensor501and from the second biometric sensor502. The controller505is operative to communicate with the transceiver507to pass the raw sensor data to the second device520over a wireless link530between the transceiver507and the transceiver509within the second device520.

The transceiver509, in the second device520, is operatively coupled to a controller511. The controller511is further operatively coupled to authentication data processing logic515. The controller511is operative to receive the raw sensor data from transceiver509sent to it from the first device510. The controller511hands the raw sensor data to the authentication data processing logic515which proceeds to extract biometric features and perform template generation. The authentication data processing logic515is also operative to compare the generated templates to enrolled templates stored in the database513. If a match is found, then the authentication data processing logic515also produces the authentication output signal517. The authentication output signal517may be returned to the first device510over the wireless link530, may be sent to a third device (not shown), or may enable access to a protected system included on the second device520. For example, in some embodiments, the authentication output signal517unlocks the second device520, the first device510, or both devices.

In some embodiments, the first device510may be categorized as a “wearable device.” As mobile devices decrease in size due to continuing advances in miniaturization technologies, some have become “wearable devices” in the sense that these devices may be worn by a user as a fashion accessory such as jewelry, an article of clothing, a portion of an article of clothing, etc. A wearable device may have any suitable structure and therefore the possible wearable devices may include a ring, a wristwatch (also referred to as a “smartwatch”), a button or brooch which may include a pin for attaching to clothing, or a patch that may be sewn to, or into, clothing such as a shirt or blouse, etc. Other example wearable devices may include a bracelet, an anklet, a belt buckle, etc.

The wireless transceiver507of the wearable device, and the wireless transceiver509of the second device, may utilize any suitable wireless technology such as IEEE 802.11 (also referred to as WiFi®), Bluetooth®, Wireless USB, ZigBee, Bluetooth® Low Energy (also referred to as “Bluetooth® Smart”) or any other suitable wireless technology that may form the wireless link530between the first device510and the second device520to transfer information or command and control signaling there-between. The first device510and the second device may go through a pairing procedure or a connection procedure depending on the wireless technology employed. The second device520, may be a mobile device such as, but not limited to, a mobile phone (also referred to as a “smartphone”), a laptop computer, electronic book reader, personal digital assistant (PDA), electronic game console, or similar device. In some embodiments, one of the devices may be at a fixed location and the other device may be a mobile device or wearable. Some examples of this are when the multimodal authentication system is used for door access, or other fixed system access such as access to a fixed control panel or fixed computer equipment. In these cases, the user may have a portable device (i.e. the wearable or mobile device) which communicates with a fixed device in the two-step authentication process.

Therefore the wireless transceivers of both devices can each receive wireless signals from, and send wireless signals to, the other device wireless transceiver over the wireless link530. The data collector503is operatively coupled to the transceiver507such that it may send data over the wireless link530to the second device520. The data collector503is also operative to receive command and control signals from the second device520by way of the transceiver507and the wireless link530. For example, a controller511within the second device520may send a command signal to the data collector503to initiate data collection from the PPG sensor501or from the second biometric sensor502.

The authentication process may be initiated in the various embodiments using a variety of approaches. In one approach, the PPG sensor501may send out a reflective signal periodically to determine whether a user is present (i.e. whether human tissue is nearby and being detected). Because the first device510, the second device520, or both may be powered by batteries within the devices in some embodiments, this approach is performed periodically in order to conserve battery power for the respective device. For a fixed device where power may be provided from a power source other than a battery, power conservation may not be an issue and therefore the PPG sensor501or second biometric sensor502may be in a constant “on” state (or “sensing state”) such that data collection will begin as soon as the controller505determines that human tissue is present or that the relevant inputs are present for biometric sensor502. In other embodiments, a third sensor may be used in one or both devices such as a proximity sensor that detects the presence of a user. In that case, the proximity sensor output may be used as a trigger input to the controller505to initiate the data collector503to collect biometric sensor data from the PPG sensor501and the second biometric sensor502. Another approach is a user initiated authentication where the user invokes a user interface on one of the devices to manually initiate the authentication process. Any other approach to initiation of the authentication process may be used in the various embodiments.

The flowchart ofFIG. 6provides a method of operation of the multimodal authentication system shown inFIG. 5, in accordance with an embodiment. The method of operation begins and in operation block601, the data collector503obtains first biometric data using PPG sensor501, and obtains second biometric data using the second biometric sensor502. In operation block603, the controller505uses the transceiver507to wirelessly transmit the first biometric data and the second biometric data to the second device520over the wireless link530. The controller511receives the biometric data and passes it to the authentication data processing logic515. In operation block605, the authentication data processing logic515extracts the bulk absorption features from the first biometric data and generates a first biometric template. In operation block607, the authentication data processing logic515extracts the second biometric features from the second biometric data and generates a second biometric template. In operation block609, the authentication data processing logic515compares the first biometric template and the second biometric template to respective corresponding enrolled biometric templates contained in the database513. In decision block611, the authentication data processing logic515determines whether the first biometric template and the second biometric template both match the database. If yes, then in operation block615, the authentication data processing logic515provides the authentication output signal517and the method of operation terminates. If a match with the database is not found in decision block611, then in operation block613access is denied and the method of operation also terminates.

Another embodiment of a multimodal biometric authentication system is illustrated inFIG. 7. A first device710includes a PPG sensor701and a second biometric sensor702which are similar to the sensors in theFIG. 5system. The biometric sensors are operatively coupled to data collection and processing logic703. The data collection and processing logic703may be considered as an integration of the data collector503features and the authentication data processing logic515features into a single logic component.

The data collection and processing logic703is further operatively coupled to controller705which is further operatively coupled to transceiver707. The second device720includes a transceiver709operative to communicate with the first transceiver707by wireless link630. The transceiver709is operatively coupled to controller711and the controller711is operatively coupled to database713. The controller711is operative to provide the authentication output signal717. The transceivers and controllers inFIG. 7are also similar to the transceivers and controllers in theFIG. 5system.

The flow chart ofFIG. 8provides a method of operation of the multimodal authentication system shown inFIG. 7, in accordance with an embodiment. The method of operation begins and in operation block801, the data collection and processing logic703obtains first biometric data using the PPG sensor701and obtains second biometric data using the second biometric sensor702. In operation block803, the data collection and processing logic703extracts bulk absorption features from the first biometric data and generates a first biometric template. In operation block805the data collection and processing logic703extracts second biometric features from the second biometric data and generates a second biometric template. In operation block807, the controller705communicates with the transceiver707and wirelessly transmits the first biometric template and the second biometric template to the second device720using the wireless link630. In operation block809, the second device controller711compares the first biometric template and the second biometric template to respective corresponding enrolled biometric templates in the database713. In decision block811, if the controller711finds that the first biometric template and the second biometric template match the database713, then in operation block815, the controller711provides the authentication output signal717and the method of operation terminates. If a match is not found in decision block811, then the controller711denies access as shown in operation block813and the method of operation also terminates.

Another embodiment of a multimodal biometric authentication system is illustrated in the block diagram ofFIG. 9. The multimodal biometric authentication system includes a first device910and a second device920. The first device910includes a transceiver907operatively coupled to a controller905which is further operatively coupled to data collection and processing logic903. The data collection and processing logic903is operatively coupled to a biometric sensor901and to a database913. The biometric sensor901may be a PPG sensor or some other biometric sensor. The second device920includes a transceiver909that is operative to communicate with transceiver907using the wireless link930. The transceiver909is operatively coupled to a controller911which is further operatively coupled to data collection and processing logic915. The data collection and processing logic915is operatively coupled to the biometric sensor902and to the database917. The biometric sensor902may also be one or the other of a PPG sensor or some other biometric sensor. The transceivers, controllers and data collection and processing logic components are similar to those discussed with respect to theFIG. 5andFIG. 7systems. In theFIG. 9system, the first device910and the second device920each independently evaluate a respective biometric corresponding to the type of biometric sensor incorporated into the device. At least one of the two devices incorporates a PPG sensor as its biometric sensor. The other device may have either some other type of biometric sensor, or a second PPG sensor that operates in a different spectral range from that of its partner device.

Operation of the multimodal biometric authentication system shown inFIG. 9is best understood with respect to the flowchart ofFIG. 10. The method of operation begins and in operation block1001, the data collection and processing logic903obtains first biometric data using biometric sensor901which may be a PPG sensor. The second device data collection and processing logic915likewise obtains second biometric data from biometric sensor902which may be another type of biometric sensor or may be a second PPG sensor with a different spectral range. In operation block1003, in an embodiment in which the biometric sensor901is a PPG sensor, the data collection and processing logic903extracts bulk absorption features from the first biometric data and generates a first biometric template. In operation block1005, the data collection and processing logic903compares the first biometric template to at least one enrolled biometric template in the database913. In decision block1007, the data collection and processing logic903checks whether the biometric template matches an enrolled template in the database913. If a match with the database913found in decision block1007, then in operation block1011the controller905communicates with the transceiver907and wirelessly transmits a first authentication output signal to the second device920using the wireless link930.

However if a match with the database913is not found in decision block1017, then in operation block1009, the first device910wirelessly transmits an access denied signal to the second device using the wireless link930, and the method of operation terminates. If in operation block1011, the authentication output signal is transmitted to the second device920then the method of operation proceeds to operation block1013. In operation block1013, the second device data collection and processing logic915extracts second biometric features from the second biometric data and generates a second biometric template. As shown in operation block1015, the data collection and processing logic915then compares the second biometric data to at least one enrolled biometric template in the database917. In decision block1017, the data collection and processing logic915checks whether the second biometric template matches an enrolled template contained in the database917. If a match is found in decision block1017, then in decision block1019the data collection and processing logic915checks whether or not an authentication output signal has been received from the first device910. If the authentication output signal was received in operation block1011and verified in decision block1019, then the data collection processing logic915provides an authentication output signal919as shown in operation block1021and the method of operation terminates.

If the second biometric template does not match an enrolled template contained in the database917, then the data collection and processing logic915issues the access denied signal as shown in operation block1023and the method of operation terminates. Likewise, if an authentication output signal was not received from first device in decision block1019the access denied command is also issued in operation block1023which terminates the method of operation. In other words, the multimodal authentication system requires that the biometrics taken by both the first device910and the second device920match enrolled biometric templates in their respective databases.

Another embodiment of a multimodal biometric authentication system is illustrated inFIG. 11. A device1100includes a PPG sensor1109, audio equipment1105that includes at least one microphone and that is operatively coupled to a voice recognition engine1103, and data collection and processing logic1107. The data collection and processing logic1107and the voice recognition engine1103are both operatively coupled to a controller1101. The controller1101is operative to issue an authentication output signal1111to a protected system1119which in response, grants a user access. The audio equipment1105may include, among other things, at least one microphone, at least one speaker, signal amplification, analog-to-digital conversion/digital audio sampling, echo cancellation, and other audio processing, etc., which may be applied to one or more microphones and/or one or more speakers of the device1100.

The protected system1119may be a system present on the device1100or may be an external system that is physically external from the device1100in some embodiments. The device1100also includes a display1113which is operatively coupled to the controller1101, and other user interfaces1115which are also operatively coupled to the controller1101. The other user interfaces115may include, but are not limited to, a keypad, touch sensors, a gyroscope and accelerometer (which may be separate or integrated in a single package), mouse, buttons, etc. The controller1101is operatively coupled to a non-volatile non-transitory memory1117.

The memory1117includes executable code1121for the voice recognition engine (VRE)1103, data collection and processing executable code1125for the data collection and processing logic1107, various voice print templates1123(also referred to as “voice models”) and PPG templates1127which are biometric templates. The voice print templates1123and the PPG templates1127are enrolled biometric template databases that contain biometric templates of at least one user that have been enrolled through a biometric enrollment process and such enrollment processes are well understood.

The voice print templates1123are used by the controller1101to compare a voice print obtained from the voice recognition engine1103and determine if a match exists. Likewise, the PPG templates1127contained in memory1117are also used by the controller1101to compare with PPG templates generated by the data collection and processing logic1107to determine whether a match exists. If the controller1101detects a matching voice print from the voice recognition engine1103and matching a PPG template from the data collection and processing logic1107, then the controller1101issues the authentication output signal1111to the protected system1119and thereby grants access.

Operation of the device1100is best understood from the flowchart shown inFIG. 12. The method of operation begins and in operation block1201, the data collection and processing logic1107obtains PPG biometric data using the PPG sensor1109. In operation block1203, the data collection and processing logic1107extracts bulk absorption features from the PPG biometric data and generates a biometric template. In operation block1205, the controller1101compares the biometric template to at least one enrolled biometric template in the PPG templates1127database. In operation block1207, voice recognition engine1103obtains voice data using at least one microphone of the audio equipment1105. In operation block1209, the voice recognition engine1103extracts voice recognition features from the voice data and generates a voice print. In operation block1211, the controller1101compares the voice print to at least one enrolled voice print in the voice print templates1123. In decision block1213, the controller1101checks whether the biometric template and the voice print match their respective databases. If both matches are found in decision block1213, then in operation block1217the controller1101provides the authentication output signal1111and the method of operation terminates with an access grant. If one or both of either the voice print or the biometric template (i.e. bulk absorption features) do not match their respective databases in decision block1213, then the controller1101denies access as shown in operation block1215and the method of operation also terminates.

It is to be understood that the block diagrams provided herein show at least those components necessary to describe the features and advantages of the various embodiments to those of ordinary skill, but that various other components, circuitry, logic, etc. may be present in order to implement various functions and features of the devices such as the example wearable and other mobile devices and that those various other components, circuitry, devices, etc., are understood to be present in the various embodiments by those of ordinary skill.

It is to be understood that information and commands such as, but not limited to, raw biometric sensor data, biometric templates and/or an authentication output signal, sent over the wireless links described herein may be encrypted using any of various suitable data encryption techniques for sending data and/or for communication over wireless links in order to enhance security and prevent middle-man snooping (also referred to as “sniffing”) attacks that attempt to intercept and obtain data over the wireless link.

The various components shown and described in the figures corresponding to multimodal authentication systems may be implemented independently as software and/or firmware executing on one or more programmable processors, and may also include, or may be implemented independently, using ASICs, DSPs, hardwired circuitry (logic circuitry), or combinations thereof. That is, any of the various described controllers, the data collector503, authentication data processing logic515, data collection and processing logic703, data collection and processing logic903, data collection and processing logic915, voice recognition engine1103, and/or data collection and processing logic1107components, may be implemented using an ASIC, DSP, executable code executing on a processor, logic circuitry, or combinations thereof.

Each of the various described database components are implemented in the various embodiments via non-volatile, non-transitory computer readable memory. The memory1117, which is also non-volatile, non-transitory computer readable memory, contains executable instructions or executable code, such as VRE executable code1121and data collection and processing executable code1125, for execution by at least one processor, that when executed, cause the at least one processor to operate in accordance with the functionality and methods of operation herein described. The computer readable memory may be any suitable non-volatile, non-transitory, memory such as, but not limited to, programmable chips such as EEPROMS, flash ROM (thumb drives), compact discs (CDs) digital video disks (DVDs), etc., that may be used to load executable instructions or program code to other processing devices such as mobile devices, including wearable devices, or other devices such as those that may benefit from the features of the herein described embodiments.

Various applications exist for the multimodal biometric systems disclosed herein and these various applications are contemplated by the present disclosure. One such application is to add a PPG sensor having a green spectral range for obtaining a reflective PPG measurement to a medical device having an infrared PPG sensor. The green spectral range PPG measurement may be used to identify or authenticate the patient and to correlate their infrared PPG measurements to their patient records. The patient would have been previously enrolled in the PPG biometric database for the hospital. Another application is a device unlocking feature that requires a PPG biometric on, for example, a smartwatch having a PPG sensor incorporated in the wristband or the case housing, and a second biometric such as a voice print from a second device such as a smartphone. The smartwatch may send an authentication output signal wirelessly to the smartphone to unlock the smartphone if the user's PPG bulk absorption characteristic taken at the smartwatch match and a voice print taken by the smartphone also matches. Other ways of combining two or more biometric features to enhance security and authentication reliability will occur to those of ordinary skill in light of the present disclosure.

While various embodiments have been illustrated and described, it is to be understood that the invention is not so limited. Numerous modifications, changes, variations, substitutions and equivalents will occur to those skilled in the art without departing from the scope of the present invention as defined by the appended claims.