Patent Description:
Image sensors are popular for home applications. Examples include those used for a baby monitor, internet protocol (IP) camera, security camera, and so. Other image sensors include thermal cameras as well as an array of thermal sensors. Expanding the effective applications of image sensors would enhance the popularity. The document <CIT> (<NUM>-<NUM>-<NUM>), discloses the use of multiple sensors to assess the physical state of a user.

The need to expand the application of sensors (for example, thermal sensors) is underscored by an article reported in the Chicago Sun Times in May <NUM> about an Illinois man who died after suffering a heart-related event while driving and crashing his vehicle. The man was driving when he suffered a "heart-related event," lost consciousness, and crashed his vehicle into a utility pole. After crashing into the pole, his car struck another vehicle. Preventive measures addressing such horrific events would certainly be beneficial to the general population.

According to a first aspect of the present invention, there is provided an apparatus for assessing a health condition of a user by supporting biometric data about the user, the apparatus comprising: a thermal sensor interface configured to obtain a thermal signal from a thermal sensor; a radar sensor interface configured to obtain a radar signal from a radar sensor; a processor for executing computer-executable instructions; a memory storing the computer-executable instructions that when executed by the processor cause the apparatus to perform: generating a first biometric vector from the thermal signal, wherein the first biometric vector contains first information about a first biometric feature and wherein the first biometric vector contains third information about a second biometric feature, wherein the first biometric feature and the second biometric feature are different biometric features; generating a second biometric vector from the radar signal, wherein the second biometric vector contains second information about the first biometric feature and wherein the second biometric vector contains fourth information about the second biometric feature; extracting first resultant information about the first biometric feature from the first and second information; generating a first resultant biometric vector containing the first resultant information about the first biometric feature; generating the first resultant biometric vector further containing second resultant information about the second biometric feature, wherein the second resultant information is extracted from the third information and the fourth information; based on a health record of the user, applying a first weight to the first resultant information about the first biometric feature and a second weight to the second resultant information about the second biometric feature, wherein the first weight and the second weight are different; and determining hazard information from the first resultant biometric vector, wherein the hazard information includes a hazard level and a confidence level and wherein the hazard level is indicative of an occurrence of a health event for the user and the confidence level is indicative of a degree of certainty of the hazard level.

According to a second aspect of the present invention, there is provided a method for assessing a health condition of a user, the method comprising: obtaining a thermal signal from a thermal sensor; obtaining a radar signal from a radar sensor; generating a first biometric vector from the thermal signal, wherein the first biometric vector contains first information about a first biometric feature and wherein the first biometric vector contains third information about a second biometric feature, wherein the first biometric feature and the second biometric feature are different biometric features; generating a second biometric vector from the radar signal, wherein the second biometric vector contains second information about the first biometric feature and wherein the second biometric vector contains fourth information about the second biometric feature; extracting first resultant information about the first biometric feature from the first and second information; generating a first resultant biometric vector containing the first resultant information about the first biometric feature; generating the first resultant biometric vector further containing second resultant information about the second biometric feature, wherein the second resultant information is extracted from the third information and the fourth information; based on a health record of the user, applying a first weight to the first resultant information about the first biometric feature and a second weight to the second resultant information about the second biometric feature, wherein the first weight and the second weight are different; and determining hazard information from the first resultant biometric vector, wherein the hazard information includes a hazard level and a confidence level and wherein the hazard level is indicative of an occurrence of a health event for the user and the confidence level is indicative of a degree of certainty of the hazard level.

With another aspect, a thermal signature, which identifies a user, is extracted from a thermal signal. When a health event is detected, an apparatus sends a message indicative of the hazard level about the user to a medical facility.

With another aspect, an apparatus downloads a health record of a user. Biometric features are weighed differently based on the health record.

With another aspect, first and second trained machine learning models may transform thermal signals and radar signals, respectively. The transformed signals are then used to obtain biometric vectors. Moreover, a thermal signature and/or motion vector provided by the first machine learning model to the second machine learning model may assist in transforming the radar signal.

With another aspect, machine learning models may be downloaded by an apparatus from a cloud server. The models may be updated at the cloud server so that updated model information can be received by the apparatus in order to update the downloaded models.

With another aspect, an apparatus provides an assessment of a vehicle driver. The apparatus obtains a thermal signal and a radar signal about the vehicle driver from a thermal sensor and a radar sensor, respectively. The apparatus generates biometric vectors from the thermal and radar signals, extracts resultant information about one or more biometric features, and generates a resultant biometric vector from the resultant information. The apparatus then determines hazard information from the resultant biometric vector when a health event occurs. Based on the hazard information, the apparatus identifies an appropriate action and performs the appropriate action on behalf of the vehicle driver.

With another aspect, an apparatus downloads a chronicle health record and conduct record of a vehicle driver and identifies an appropriate action based on the chronicle health and conduct record.

With another aspect, an apparatus applies a first weight to the first resultant information about a first biometric feature and a second weight to a second resultant information about a second biometric feature based on the chronicle health and conduct record of a vehicle driver.

With another aspect, when a health event about a vehicle driver is detected, an apparatus sends a chronicle health and conduct record to an emergency service to prepare for the arrival of the vehicle driver.

With another aspect, an apparatus uses a thermal sensor for biometric data extraction and tracking for smart home applications. Applications such as health condition analysis, motion estimation (for example, fall estimation or motion trajectory), casual prediction (for example, heart beat is slowing down to a hazardous level), hazard detection (for example, endurance abnormal body position such as laying down on floor, laying sideway on sofa or head down for a long time), learning the profile of individuals, and system adaptation according to individual preferences.

With another aspect, parameters of a thermal sensor may be enhanced to allow as much data to be extracted as possible. Examples include, but not limited to: increasing the number of sensing element (i.e., the resolution), frame rate, sensitivity, and/or signal-to-noise level.

With another aspect, signal processing techniques extract biometric data from the thermal images.

With another aspect, an analytic model is used for hazard prediction and subsequently associated actions taken.

With another aspect, hazard analysis is done by a deep learning model. Actions are taking based on the hazard coefficients with the associated confidence levels estimated from the model.

With another aspect, the model would suggest actions to be taken with the associated confidence levels based on the input data sequence.

With another aspect, the model may be trained to predict the hazard coefficients, and the corresponding actions if necessary, with the corresponding confidence levels based on the events previously occurring.

With another aspect, the model may reside in a cloud server rather than a local processing unit for applications that are less time critical.

With another aspect, parameters of a smart device are configured differently based on a thermal signature of a detected person.

With another aspect, an executed application is changed from a first application to a second application based on a detected condition detected by the first application.

The foregoing summary of the invention, as well as the following detailed description of exemplary embodiments of the invention, is better understood when read in conjunction with the accompanying drawings, which are included by way of example, and not by way of limitation with regard to the claimed invention.

According to the claimed invention, an apparatus detects a health condition of a user that is assessed from a thermal sensor signal and radar sensor signal. One or more resultant biometric vectors are generated from biometric vectors based on the thermal and radar signals, where the resultant biometric vectors contain resultant information about first and second biometric features for a user. Hazard information about the user is obtained from the one or more resultant biometric vectors, where the hazard information is indicative of a health event for the user. Consequently, an appropriate action on behalf of the user may be performed to ameliorate the health condition. The one or more resultant biometric vectors may include additional biometric features and/or a time sequence of the resultant biometric vectors to enhance hazard prediction.

With another aspect of the embodiment, performance indices (for example, resolution, frame rate, and sensitivity) of a thermal sensor or an array of thermal sensors may be increased to support applications such as identification verification, biometric data extraction, and health condition analysis. Prediction may be carried out by monitoring a time sequence of thermal images and consequently an early warning of the health condition may be generated.

With another aspect of the embodiments, the frame rate of a thermal sensor may be increased to a determined level to capture the change in the minor details in the thermal radiation from a human body against time, for example, the detail change in the thermal radiation from human body.

With another aspect of the embodiments, the thermal image of the blood flows through the skin may be converted to a time signal for pulse rate extraction. Further signal processing techniques may be applied to extra biometric data of an individual for analyzing the health condition. An image signal may be processed to identify multiple objects from the content and to track associated biometric data.

With another aspect of the embodiments, an application may determine the position of a human body within the image signal, together with motion tracking from the previous images, for fall detection. Motion estimation may be applied to predict if there is any hazard to the individuals within the image signal.

With another aspect of the embodiments, a profile may be associated to an individual. An apparatus may track and learn the behavior of the individual from the history of image signal. Moreover, the apparatus may adapt when the individual is detected in the scene. For example, the set temperature of the air conditioner in the sitting room may be adapted to an individual's preference when the individual is detected going into the sitting room in the summer time.

With another aspect of the embodiments, the environment temperature can be controlled according to the body surface temperature of individual(s), together with other parameters (such as relative humidity and outside temperature, and so forth) to reach the overall comfort zone through machine learning.

With another aspect of the embodiments, the accuracy of an analysis is determined by the resolution, sampling frequency and sensitivity of a thermal sensor, signal processing techniques in extracting biometric data from the image signals, and analytic/learning algorithms.

With another aspect of the embodiments, applications of thermal sensors may be extended to domestic applications.

With another aspect of the embodiments, the analytic model is composed of a trained model. The model is trained from a database of reference thermal image signals and an associated target vector, which may represent a series of settings for the smart home devices. Reinforcement learning may be deployed to allow the model to adapt to a new target vector. For example, a user may change the temperature setting of a room between summer and winter.

With another aspect of the embodiments, no training is applied to the analytic model but learning from the sequence of target vectors over time that is associated with a thermal signature. For example when a new thermal signature, which is associated with a new user, is detected, a default setting for the smart home devices is applied. When the user changes the setting of individual device, the new setting would be recorded for re-training the model.

<FIG> shows thermal camera <NUM> positioned in room <NUM> in accordance with an embodiment. Camera <NUM> may generate thermal image (thermogram) <NUM> of an individual not explicitly shown.

With some embodiments, thermal camera <NUM> comprises a lens that focuses infrared or far-infrared radiation by objects in view. The focused light is scanned by a thermal sensor, which comprises a plurality of infrared-detector elements (for example, <NUM> by <NUM> pixels). The detector elements may create a very detailed temperature pattern (for example, thermogram <NUM>).

With some embodiments, camera <NUM> may require one-hundredth of a second for the detector array to obtain sensor information to obtain the thermal image. The sensor information may be periodically obtained from several thousand points in the field of view of the thermal sensor to form a sequence of thermal images.

Thermogram <NUM> created by the detector elements of the thermal sensor may be converted into electric impulses. The impulses are then sent to a signal-processing unit (for example, apparatus <NUM> as shown in <FIG>), which may be implemented as a circuit board with a dedicated chip that converts the sensor information into biometric data.

Thermal camera <NUM> may also include a tracking capability so that the direction of camera <NUM> may vary to track a moving object such as person <NUM> moving in room <NUM>.

While <FIG> depicts one thermal sensor, some embodiments may interface with a plurality of thermal sensors. For example, thermal sensor arrays may be positioned in different rooms and/or at entry points of a dwelling.

<FIG> shows apparatus <NUM> interfacing with thermal sensor <NUM> and/or <NUM> through sensor interface <NUM> and smart devices <NUM> and/or <NUM> through smart device interface <NUM> in accordance with an embodiment.

Thermal sensors <NUM> and <NUM> are often used for access control and presence detection. With some embodiments, in order for processor <NUM> to extract biometric data from sensor information, the performance of thermal sensor <NUM> may be extended by increasing the sample frequency (for example, frame rate) of capturing the image signal, identifying and tracking individuals from the image signal, and analyzing detail changes in thermal images against time. Processor <NUM> converts sensor information (signals) to biometric data, such as heart rate, body position, health condition, and so forth. Apparatus <NUM> may also support prediction of future health events may by processing the image signals and/or support system personalization.

With some embodiments, processor <NUM> may process sensor information to detect a thermal signature of a user. When a thermal signature of a particular individual is detected, processor <NUM> may apply the individual's profile (for example, a temperature setting) to smart device <NUM> (for example, an air conditioner).

Processor <NUM> may support one or more health applications that processes and/or analyzes biometric data and may generate notifications about the biometric data to an external entity (for example, a doctor) over communications channel <NUM> via interface <NUM>. As an example, a health application may detect that a user is having a possible heart attack from the biometric data; consequently, an urgent notification is sent to the user's doctor about the event.

With reference to <FIG>, a computing system environment may include a computing device where the processes (for example, process <NUM> shown in <FIG>) discussed herein may be implemented. The computing device includes processor <NUM> for controlling overall operation of the computing device and its associated components, including RAM, ROM, communications module, and first memory device <NUM>. The computing device typically includes a variety of computer readable media. Computer readable media may be any available media that may be accessed by computing device and include both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise a combination of computer storage media and communication media.

Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but is not limited to, random access memory (RAM), read only memory (ROM), electronically erasable programmable read only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device.

Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

In this embodiment, processor <NUM> executes computer-executable instructions stored at memory <NUM> and access profile data stored at memory <NUM>.

With some embodiments, memory devices <NUM> and <NUM> may be physically implemented within a single memory device.

<FIG> shows apparatus 300a that processes information from one or more thermal sensors <NUM> in accordance with an embodiment.

By using a higher quality thermal sensor <NUM> (for example, with a frame rate of at least <NUM> frames per second, resolution of at least <NUM> x <NUM> pixels, good sensitivity, and low noise), biometric data <NUM> may be extracted via appropriate signal processing techniques via analog front end <NUM>, analog to digital convertor (ADC) <NUM>, and feature extractor <NUM>. Biometric data <NUM> may include pulse rate, body surface temperature, temperature distribution pattern, body contour and posture, and so forth. By tracking the variations of biometric data, the health condition of an individual may be analyzed by analyzer <NUM>, and early warning signals <NUM> and <NUM> may be generated by analyzer <NUM> and action generator <NUM>, respectively, by further processing biometric data <NUM>.

An application may utilize a domestic thermal camera installed for fall detection by tracking the change of posture. For example, when the posture changes from upright to horizontal in a short time, a possible fall may be detected and hence an associated alert may be generated. Moreover, a variation of posture, body surface temperature, temperature distribution pattern, and heart rate may be tracked to estimate the hazard level, and associated actions <NUM> can be taken.

Hazard prediction from biometric data <NUM> may be also supported. For example, when one's body surface temperature is continuously dropping and his/her posture is shaking, the chance of a fall may be higher (as indicated by hazard level <NUM>) and hence an alert may be generated (triggered) before a fall can occur.

Block <NUM> may perform signal amplification and non-recursive band-pass filtering, in which the analog signal corresponding to the thermal image is processed for DC offset removal, noise reduction and frequency limited before being processed by ADC <NUM>. (With some embodiments, block <NUM> may comprise a <NUM>-bit ADC with the sampling frequency (for example, <NUM>) being set high enough to capture the details of temperature change of an object.

In the feature extraction block <NUM>, image processing is applied to identify valid objects, track the thermal profile of individual objects over time and extract the parameters from the thermal profile to form a feature vector. Examples of the parameters for the feature vector include period time, variation of the period times, certain time constants within each periodic cycle and their variations over time, etc. The analytic model <NUM> takes in the feature vector and compares it over a trained model. The model is pre-trained with a large set of generic feature vectors using deep learning algorithms, e.g. a deep neural network. Reinforcement learning may be deployed to allow the model to learn from the mistakes. Hazard levels may be provided for the identified objects. In block <NUM>, a list of actions may be pre-defined and may be triggered based on the associated hazard levels.

<FIG> shows apparatus 300b that processes information from one or more thermal sensors <NUM> in accordance with another embodiment. Hazard coefficients <NUM> with associated confidence levels <NUM> are estimated by a model trained using a deep learning model <NUM>, for example, a convolutional neural network with supervised learning. Actions <NUM> are determined from action list <NUM> and may be based on hazard coefficients <NUM> and confidence level <NUM> provided by model <NUM>. Model <NUM> may initially support hazard levels but subsequently identify different hazards with more empirical data such as heart-rate abnormality, body surface temperature drop, fall detection, and so forth.

Model <NUM> in apparatus 300b may also be trained to predict hazards, rather than estimating hazards, based on the training sequence which started from a substantially earlier time.

<FIG> shows apparatus 300c that processes information from one or more thermal sensors <NUM> in accordance with a third embodiment, in which actions <NUM> and associated confidence levels <NUM> are estimated by trained model <NUM>. Again, model <NUM> in apparatus 300c also may be trained to predict any actions needed.

The image processing technique that may be used depends on the system complexity, including the number of thermal sensors, the resolution of each thermal sensor, the list of hazards and actions, the system computation power and memory available, and so forth.

For the embodiments shown in <FIG>, <FIG> and <FIG>, the analytic models may be implemented locally or in a cloud server, depending on criticality of response time.

<FIG> shows process <NUM> that identifies a user from thermal sensor information and applies a corresponding profile in accordance with an embodiment.

Process (application) <NUM> supports human presence detection and thermal signature verification at block <NUM>. If a human object is detected and the thermal signature is matched to a known entity at block <NUM>, all supported smart devices (for example, air conditioner, smart TV, or smart lighting) may be adjusted <NUM> in accordance with the profile database stored at block <NUM>.

If there is any adjustment to the applied profile <NUM>, the adjustment data <NUM> may be sent to a profile adaptation unit <NUM> in which the new settings in the profile may be included. The profile database would be updated <NUM> by profile adaptation unit <NUM> if an adjustment is needed.

To add a new user, profile adaptation unit <NUM> sends the thermal signature of the new user <NUM> to the user identifier unit <NUM> together with the associated profile, which could be a default profile, to the profile database unit <NUM>.

Profile adaptation unit <NUM> may comprise a deep learning model trained using reinforcement learning.

<FIG> shows flowchart <NUM> for sequencing through a plurality of applications, as executed by apparatus <NUM>, in accordance with an embodiment. Apparatus <NUM> may execute one of the plurality of applications depending on a detected condition. For example, a first health application may monitor general health measurements (for example, an amount of activity and temperature) of a user. If one or more of the measurements are abnormal, apparatus <NUM> may initiate different health applications based on the detected condition.

Referring to <FIG>, apparatus <NUM> configures thermal sensors <NUM> and <NUM> in accordance with a first set of sensor parameters at block <NUM> in order to execute a first application at block <NUM>.

If an abnormal condition is detected at block <NUM>, apparatus <NUM> initiates an appropriate application. For example, apparatus may transition to a second application to monitor fall prediction or to a third application to monitor the heart rate of the user at blocks <NUM>-<NUM> and <NUM>-<NUM>, respectively. When executing the second or third applications, apparatus <NUM> may configure thermal sensors <NUM> and <NUM> differently in order to obtain different biometric data.

In another implementation, different configuration parameters may be applied to individual sensor for each application.

In a third implementation, different sets of configuration parameters are applied to the sensors one after another to extract all the biometric data before running the applications.

In a fourth implementation, a most comprehensive set of configuration parameters is used for all sensors and applications. All of the sensors may be set to the best set of configuration, for example but no limited to, highest image resolution, number of bits, frame rate, sensitivity, signal to noise ratio (SNR), computational power, power consumption, and so forth.

<FIG> shows flowchart <NUM> where apparatus <NUM> configures a smart device with one of a plurality of parameter sets based on detected users in accordance with an embodiment. Apparatus <NUM> may monitor sensor data from thermal sensors <NUM> and/or <NUM> to detect thermal signatures of one or more users. For example, thermal sensor <NUM> may be positioned at an entry point of a dwelling. Based on sensor information obtained from sensor <NUM>, apparatus <NUM> may identify users entering and exiting the dwelling. With some embodiments, apparatus may detect either a thermal signature from the front (corresponding to a person entering the dwelling) or from the back (corresponding to the person exiting the dwelling). Based on the detected thermal signatures, a smart device can be configured with different sets of parameters (for example, the temperature setting of an air conditioner).

At block <NUM>, apparatus <NUM> trains to detect thermal signatures of different users from sensor data. For example distinguishing characteristics may be stored at memory <NUM>. When thermal signatures of both users are detected at block <NUM>, only user A at block <NUM>, or only user B at block <NUM>, a smart device may be configured in accordance with a first set of smart device parameters at block <NUM>, a second set at block <NUM>, or a third set at block <NUM>, respectively. With some embodiments, the first set (when both users are detected) may be a compromise between the second and third sets (when only one user is detected). Otherwise (when no users are detected), the smart device may be configured in accordance with a default set of smart device parameters at block <NUM>.

The following capabilities may be supported by the embodiments.

An apparatus uses a thermal sensor for biometric data extraction and tracking for smart home applications. Applications such as health condition analysis, motion estimation (for example, fall estimation), casual prediction (for example, heart beat is slowing down to hazard level), hazard detection (for example, laying down for a long time), learning the profile of individuals, and system adaptation according to individual preferences.

The parameters of a thermal sensor may be enhanced to allow as much data to be extracted as possible. Examples include, but not limited to:.

Signal processing techniques extract biometric data from thermal images.

Analytic model for hazards estimation and subsequently the associated actions taken.

Analytic model for hazards and/or actions prediction.

Model for learning the behaviors of individual(s) to the smart devices according to the biometric data extracted from the thermal sensors.

Configure parameters of a smart device based on different detected people.

Change to a second health application from a first health application based on a detected condition by the first health application. The set of configuration parameters for individual sensors for an active health application may or may not be identical.

Use different set of configuration parameters to extract all biometric data before running the health applications.

Using a single comprehensive set of configuration parameters for all the sensors and health applications.

Obtain thermal sensor data to detect a thermal signature for either the front or the back of a person.

Able to increase the sampling frequency of thermal sensors, including IP cameras, thermal cameras, and thermal sensors, to capture the minor changes of the color content due to thermal radiation from a human body.

Able to increase the resolution and sensitivity of thermal sensors to span the detection range.

With some embodiments, the sets of configuration parameters for all the sensors may be identical, in other words, all sensors can be configured with a most comprehensive set of parameters for all applications. The best sensor configuration may include, but not limited to, highest image resolution, number of bits, frame rate, sensitivity, and signal to noise ratio (SNR).

The following is directed to vehicle operator continuous physical health monitoring embodiments.

Referring back to <FIG>, while embodiments support assessing the health of a person in a room using a thermal sensor, embodiments utilize thermal sensor data to assess the health of a person within other types of confined spaces such as a vehicle. The parameters used may include, but is not limited to, heart rate, breathing rate, body surface temperature, posture (in particular, head position), and the trajectories of these data over time, and so forth.

The physical health condition of a vehicle operator (vehicle driver) may be critical to the safety of the operator, the passengers, and the vehicle itself. The state of the vehicle operator condition could determine the output of a situation should an emergency arises unexpectedly.

With traditional approaches, there are numbers of ways to monitor the physical health of the vehicle operator via wearables devices. However, a wearable device is specific to the individual wearing the device and not to the vehicle and may not ensure that the information or data of the vehicle operator's health is securely monitored during the duration of the vehicle when it is in use.

With an aspect of the embodiments, monitoring of a driver and/or vehicle may be performed in an non-intrusive and accurate manner that is activated all of the time that the vehicle is in operation. Consequently, the health of whoever is driving the vehicle may be assessed. With this approach, biometric information about the driver is utilized for accident prevention, incident alert, critical health warning and postmortem analysis.

<FIG> shows vehicular system <NUM> for continuously monitoring a vehicular operator's physical health in accordance with an embodiment.

In reference to <FIG>, the embodiment obtain thermal sensor data from thermal sensor <NUM> via thermal sensor interface <NUM> as previously discussed.

The thermal sensor <NUM> is typically fitted at a fixed location in front of the vehicle operator (driver), for example, mounted against the top windshield corner in front of the driver.

Processor <NUM> configures the thermal sensor by reference to methods in <FIG>.

Processor <NUM> extracts biometric information contained in sensor data <NUM>. For example, processor <NUM> may continuously monitor the heart rate and head posture about the driver as soon as he sits in the driving seat. In addition, the health record of the driver may be loaded into processor <NUM> via wireless devices <NUM> from a remote database server.

Processor <NUM> may decide addition biometric data are needed based on the health record of the driver. For example if the BMI of the driver exceeds a certain value, the change of heart rate, the change of body surface temperature, and change of head posture over time may also be monitored.

As will be discussed in further detail, processor <NUM> detects one or more current physical conditions about the driver and executes one or more actions to address the detected physical conditions.

Processor <NUM> may report detected physical conditions to the driver, doctor, emergency contact, and so forth via wireless device <NUM> (for example a smartphone) executing an application, initiating a telephone call to <NUM>, generating an e-mail message to a designated person, and so forth.

Processor <NUM> may also initiate an action in response to the detected physical condition. For example, if processor <NUM> determines that the driver is experiencing a heart attack, processor may instruct self-driving interface <NUM> to route the vehicle to the nearest hospital.

As will be further discussed, biometric information may be stored in storage device <NUM> for subsequent analysis about the health condition of the vehicle driver. While storage device <NUM> is shown as a separate device, storage device <NUM> may be integrated within wireless device <NUM>.

<FIG> shows process <NUM> that performs one or more actions based a detected physical condition of a vehicle driver in accordance with an embodiment.

At block <NUM>, processor <NUM> extracts biometric information contained in sensor data <NUM>. Processor <NUM> processes the information conveyed in signal <NUM> to extract measurements for one or more biometric characteristics of the vehicle driver at block <NUM>. Biometric characteristics may include, but are not limited to, heart rate, breathing rate, and deviation from average heart rate (for example, degree of heart beat irregularity).

The measurements of the biometric characteristics may be stored in storage device <NUM> for analysis about the health condition of the vehicle driver at a later time. For example, the stored data may be evaluated by the driver's doctor to determine if medical treatment is needed.

At block <NUM>, process <NUM> obtains the measurements of the biometric characteristics (for example, the vehicle driver's heart rate and breathing rate) and determines whether a health profile applies to the drives. A plurality of health profiles may be specified, where a first health profile maps to normal vital functions of the driver (in other words, no detected health event), a second health profile maps to a heart attack event, a health third profile maps to the driver falling asleep, a fourth health profile maps to excessive alcohol consumption, and so forth.

If an abnormal health is detected based on the determined health profile is detected at block <NUM>, process <NUM> detects whether a particular health event occurred at blocks <NUM>-<NUM>. Based on a particular health event, process <NUM> executes an appropriate action. Exemplary actions include, but are not limited to:.

With an aspect of the embodiments, a processing unit continuously monitors and analyzes the heartbeat of a vehicle driver to generate an alert about any irregularity. The processing unit may use a unique algorithm to provide this capability.

With an aspect of the embodiments, a processing unit may identifying a detected irregularity to correspond to one of a plurality of events about a vehicle driver, including, but not limited to, falling asleep, a heart attack, consuming an excessive amount of alcohol, and so forth.

With an aspect of the embodiments, data about the heartbeat of a vehicle driver may be stored in a storage device. The data may be retrieved at a later time for analyzing whether an abnormal health event occurred.

As previously discussed (for example, apparatus <NUM> as shown in <FIG>), biometric data is captured using thermal sensors <NUM> and <NUM>. However, to enhance the robustness of a biometric system when a thermal sensor signal is blocked (for example, by furniture in smart home applications), one or more RF sensors (a radar sensor) are added to form a hybrid sensing system. While it may be difficult to identify a user only with a radar sensor, a thermal signature of user may be obtained from a thermal sensor array and associated with a radar signal. Consequently, as will be discussed, utilizing both thermal and radar sensors may complement each other to improve the accuracy of hazard/action estimation.

<FIG> shows apparatus <NUM> interfacing with radar sensor <NUM> and thermal sensor <NUM> through radar sensor interface <NUM> and thermal sensor interface <NUM>, respectively.

As will discussed in further detail, radar sensor <NUM> may comprise a transmitter transmitting a radio frequency (RF) signal in the radar spectrum (for example, operating at <NUM> or <NUM>) and one or more receivers detecting reflected radar signals. Apparatus <NUM> may subsequently extract biometric data (for example, breathing rates and motion vectors) from the detected reflected radar signals.

Thermal sensor <NUM> may be used for access control and presence detection. With some embodiments, in order for processor <NUM> to extract biometric data from thermal sensor information, the performance of thermal sensor <NUM> may be extended by increasing the sample frequency (for example, frame rate) of capturing the image signal, identifying and tracking individuals from the image signal, and analyzing detail changes in thermal images against time. Computing device <NUM> converts sensor information (signals) to biometric data, such as heart rate, body position, health condition, and so forth.

As will be discussed in further detail, computing device <NUM> may utilize thermal signatures and associated motion vectors derived from thermal sensor data to assist in processing radar sensor data.

Apparatus <NUM> may also support prediction of future health events may by processing the sensor signals and/or support system personalization.

With some embodiments, computing device <NUM> processes thermal and radar sensor information to detect biometric data about a user. When a thermal signature of a particular individual is detected, computing device <NUM> may apply the individual's profile (for example, a temperature setting) to a smart device (for example, an air conditioner) through output interface <NUM>.

With some embodiments, computing device <NUM> may support a radio frequency (RF) sensor. The RF sensor may operate in the radar spectrum (<NUM>-<NUM>).

Computing device <NUM> may support one or more health applications that processes and/or analyzes biometric data and may generate notifications about the biometric data to an external entity (for example, a doctor) over communications channel <NUM> via interface <NUM>. As an example, a health application may detect that a user is having a possible heart attack from the biometric data; consequently, an urgent notification is sent to the user's doctor about the event. With some embodiments, if the user is driving a vehicle, the health application may stop the vehicle (though output interface <NUM>) and report to emergency health service (though communications interface <NUM>), and/or activate a self-driving function (via output interface <NUM>) to drive the user to a hospital.

Apparatus <NUM> may also interact with cloud server <NUM> to enable computing device <NUM> to access data (for example, a health record of a user) from a remote database. For example, computing device <NUM> may alter a decision based on the health record of the user. In addition, apparatus <NUM> may continuously stream sensor data to cloud server <NUM> for storage or real-time analysis of the health condition of the user.

With reference to <FIG>, a computing system environment includes a computing device where the processes (for example, processes <NUM>-<NUM> shown in <FIG>) discussed herein may be implemented. Each process may correspond to a block of computer readable instructions that is executed by computing device <NUM>. The computing system includes computing device <NUM> for controlling overall operation of the computing device and its associated components, including RAM, ROM, communications module, and first memory device <NUM>. The computing device typically includes a variety of computer readable media. Computer readable media may be any available media that may be accessed by computing device and include both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise a combination of computer storage media and communication media.

With some embodiments, computing device <NUM> executes computer-executable instructions stored at memory <NUM> and access profile data stored at memory <NUM>.

<FIG> shows hybrid sensing system <NUM>. Thermal and radar sensor data are separately captured by thermal sensor <NUM> and radar sensor <NUM>, respectively. Depending on the complexity of the applications, thermal sensor <NUM> may comprise a 32x32 thermopile array, and radar sensor <NUM> may operate at <NUM> with <NUM> transmit and <NUM> receive antennas.

The thermal and radar signals <NUM> and <NUM> are processed through the corresponding analytical models <NUM> and <NUM>, respectively, to obtain biometric vectors with associated confidence levels.

Model <NUM> may be a neural network model pre-trained using pre-processed thermal images and the resulting feature vectors. Model <NUM> may be another neural network model pre-trained using pre-processed RF signals and the resulting feature vectors. Referring <FIG>, model <NUM> may contain models <NUM> and <NUM> as the first stage processing and an additional neural network layers as the second stage processing. The additional neural network layers may be trained with the feature vectors as inputs, and resultant hazard and confidence level as outputs.

As will be discussed, feature vectors <NUM> and <NUM> are obtained from the biometric vectors.

The biometric data extracted from thermal sensor <NUM> may include, but is not limited to, breathing rates, heart rates, body surface temperatures, thermal signatures, and motion vectors. The biometric data extracted from the radar sensor <NUM> may include, but is not limited to, breathing rates and motion vectors. During the sensor data acquisition phase, thermal signature(s) and the associated motion vector(s) <NUM> from thermal sensor model <NUM> may be provided to radar sensor model <NUM> to assist its data analytic processing.

Feature vectors <NUM> and <NUM> are then passed to feature analytic block <NUM>, where vectors <NUM> and <NUM> are compared to obtain resultant feature vector <NUM>. Resultant feature vector <NUM> is then passed to decision logic block <NUM> for risk analysis to provide hazard level <NUM> and corresponding confidence level <NUM>.

The following discusses a hypothetical example for apparatus <NUM>, where biometric data vectors Vr and Vt convey one or more features. Biometric data vectors that convey only one biometric feature are not part of the claimed invention. Moreover, Vt may convey additional features that are not conveyed by Vr.

Biometric data vector Vr from radar sensor model <NUM> may be Vr = [Brb, Crb, Brx, Bry, Crm], where Brb is the measured breathing rate and Crb is the confidence level of the measured breathing rate ranging from <NUM> to <NUM> in an increasing level of confidence. Brx and Bry are the components of the motion vector in X and Y, respectively, with the confidence level Crm. A confidence level of <NUM> is indicative that the measurement is inconclusive.

Concurrently, biometric data vector Vt from thermal sensor <NUM> may be Vt = [Btb, Ctb, Btx, Bty, Ctv], where Btb and Ctb are the measured breathing rate and associated confidence level and Btx, Bty is the motion vector in X and Y with associated confidence level Crv.

Biometric vectors Vr and Vt include a measured value and corresponding confidence level for one or more biometric features. For example, with the examples shown above, Vr and Vt span breathing rate and motion features.

Feature vectors <NUM> and <NUM> are then processed by feature analysis block <NUM>. Exemplary flow chart <NUM> for feature analysis logic block <NUM> is shown in <FIG>. A biometric feature (for example, the breathing rate) of the same type may pass through a decision matrix to obtain resultant feature vector V <NUM>.

For example, when biometric vectors at time t1 are Vr1 = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>] and Vt1 = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>], breathing rate vectors <NUM> and <NUM> are equal to [<NUM>,<NUM>] and [<NUM>,<NUM>], respectively.

Referring to flow chart <NUM> shown in <FIG>, process <NUM> determines a resultant feature vector (V) <NUM>, where V=[B,C], B is the measured feature value, and C is the corresponding confidence level. For example, the biometric feature may be the breathing rate.

Feature vector <NUM> Vt = [Bt, Ct] and feature vector <NUM> Vr = [Br, Cr]. With flow chart <NUM>, Cmin = <NUM>. Cmin is the minimum confidence level for a measured value to be considered. The confidence level may be chosen based on empirical results. It may also be adaptive based on the quality of the sensor signals.

At step <NUM>, if both Ct and Cr are less than Cmin = <NUM>, the results are inconclusive. Otherwise, at step <NUM> process <NUM> will discard the feature vector having a confidence level less than Cmin = <NUM>. In other words at step <NUM>, if Cr < <NUM>, V=Vt. Otherwise, V = Vr.

At step <NUM>, process <NUM> determines Bdiff and Bth, where Bdiff = |Bt - Br| and Bth = <NUM>* (<NUM> * (Bt + Br)).

If Bdiff < Bth, as determined at step <NUM>, then V = [Ba,Cs] at step <NUM> where Ba = <NUM>*(Bt+Br) and Cs = <NUM>*(Ct+Cr).

Otherwise if Bdiff >= Bth, if Ct > Cr, as determined at step <NUM>, V = [Bt,Ct]. Otherwise, V = [Br,Cr].

As an example, let biometric vector Vt = [<NUM>. <NUM>] and Vr = [<NUM>,<NUM>]. Processing Vt and Vr through feature analysis logic block <NUM> in accordance with process <NUM>, one determines the resultant feature vector [<NUM>,<NUM>], corresponding to a breathing rate of <NUM> and a confidence level of <NUM>.

If biometric vectors span more than one feature (for example, breathing rate and motion), each feature may be separately processed by process <NUM>. For example, by processing biometric vectors Vr1 = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>] and Vt1 = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>], the resultant biometric vector feature V would be: V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>].

When only one biometric vector Vr or Vt contains feature information (Bi,Ci) for the ith feature (for example, heart rate), feature analysis logic block <NUM> may use feature information (Bi,Ci) when Ci >= Cmin for constructing the resultant biometric vector.

Referring back to <FIG>, decision logic block <NUM> determines hazard level <NUM> and its associated confidence level <NUM>. For the above example of V<NUM>, since the breathing rate is normal the potential hazard is low with a confidence level, for example, <NUM> (high) although there is no detected movement for the identified subject (user). However, the value of the confidence level may be adjusted based on empirical results associated with an embodiment. Decision logic block <NUM> continuously monitors resultant feature vector <NUM>, as well as its evolvement over time, to determine a potential hazard measure (corresponding hazard warning <NUM>). For example, the resultant biometric vectors of the eight consecutive measurements may be: V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [ <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [ <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], and V<NUM> = [ <NUM>, <NUM>, <NUM>, <NUM>, <NUM>].

Decision logic block <NUM> may determine that the hazard level is high and the confidence level (<NUM>) is medium at the <NUM>th measurement because the breathing rate of the subject is dropping and there is no movement detection. However, decision logic block <NUM> may determine that the hazard level is high and confidence level (<NUM>) is high at the <NUM>th measurement because the breathing level of the subject is dropping rapidly to a hazardous level and there was no movement detected over seven consecutive measurements.

The detection accuracy may be improved if more biometric data (corresponding to additional features) are obtained from sensors <NUM> and/or <NUM>. For instance, heart rate and thermal signature may also be obtained from thermal sensor <NUM>. For example, Vt = [Btb, Ctb, Bth, Cth, Btt, Ctt, Bts1, Cts1, Btx, Bty, Ctv], where Btb, Ctb corresponds to the measured breathing rate, Bth, Cth corresponds to the measured heart rate, Btt, Ctt corresponds to the measured body surface temperature, Bts1, Cts1 corresponds to the thermal signature of an associated entity (for example, registered user1), and Btx, Bty and Ctm corresponds to the motion vector in X and Y with the confidence level Ctm.

Referring to the previous example, assume that biometric vectors are Vr1 = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>] and Vt1 = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>]. The resultant biometric vector (as determined by feature analysis block <NUM> in accordance with process <NUM>) is V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>]. From decision logic block <NUM>, the hazard level would be low with confidence level <NUM> (high) as all biometric data of the identified subject (user1) are normal.

As example, the time sequence of the resultant biometric vectors may be V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [ <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], V<NUM> = [ <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>], and V<NUM> = [ <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>].

With the above time sequence, decision logic block <NUM> may estimate a hazard level as being high with confidence level <NUM> (high) for the identified subject (user1) after the <NUM>th measurement since both the breathing rate and heart rate are dropping together. Moreover, because user1 is detected via thermal signature, more valuable information may be provided, such as reporting the hazard level and confidence level to the registered hospital of user1 and preparing the medical care of user1 before arrival to the hospital.

While not explicitly shown in <FIG>, with some embodiments, hazard prediction may also be done solely by using solely machine learning models <NUM> and <NUM>.

An alternative approach, with the embodiment shown in <FIG>, outputs <NUM> and <NUM> from both sensors <NUM> and <NUM> are directly fed into analytical model <NUM>. Analytical model <NUM> may be an artificial neural network pre-trained with artificial vectors extracted from clinical data. Analytical model <NUM> may be fine-tuned (re-trained) with empirical data directly obtained from the field.

With some embodiments, combinations of thermal sensors and radar sensors may be installed through an associated entity (for example, a house or business) to ensure a desired coverage with optimal system cost in smart home/business applications. For example, a thermal sensor and a radar sensor may be installed in the sitting room as well as in each bed room. However, only thermal sensor arrays may be installed near a toilet and in the kitchen and/or the garage.

With another smart home application, there may be two thermal sensors plus a radar sensor installed in the sitting room. Signals <NUM> and <NUM> from two thermal sensors <NUM> and <NUM> may be pre-processed by signal pre-processing unit <NUM>, as shown in <FIG>, to obtain a resultant thermal sensor signal <NUM>, which may be provided to the systems shown in <FIG> or <FIG>. With some embodiments, signal pre-processing may include high-pass filtering the image signals over time and choosing the one with the higher residual. With some embodiments, signal pre-processing unit <NUM> may include stitching the two thermal sensor signals <NUM> and <NUM> to form a better thermal sensor signal. Alternatively, machine learning models <NUM> and <NUM> (as shown in <FIG>) or machine learning model <NUM> (as shown in <FIG>) may be re-trained for a specific combination of thermal sensors and radar sensors without signal pre-processing.

With some embodiment, a hybrid sensing system may be inside vehicle to increase the accuracy in monitoring the physical health condition of a vehicle operator (vehicle driver) to reduce the chance of false detections. The state of the vehicle operator condition may determine a course of action should an emergency happen.

Referring to <FIG>, hybrid sensing system <NUM> continuously monitors a vehicular operator's physical health for a vehicle application as shown. Hybrid sensing system <NUM> includes radar sensor <NUM> and thermal sensor <NUM>.

Radar sensor <NUM> includes RF transmitter <NUM>, which is typically fitted at a fixed location in front of the vehicle operator (driver) (for example, mounted against the top windshield corner in front of the driver such that the signal from the transmitter is reflected by the driver's body to the receiver without obstructed by any uninterested moving objects, such as the steering wheel). RF transmitter <NUM> generates fixed frequency signal, for example, between <NUM>-<NUM> with a power level between <NUM>-<NUM> watts depending on the RF characteristics of the vehicle.

RF signals received by receivers <NUM>-<NUM> are processed by embedded microcontroller unit (MCU) <NUM> to obtain radar signal <NUM>, which is suitable for a corresponding machine learning model.

Thermal sensor <NUM> is typically physically situated next to radar sensor <NUM> to capture thermal signal <NUM> via thermal sensor array <NUM> from the driver's head as well as the body. Thermal signal <NUM> is processed by embedded MCU <NUM> to output signal <NUM>, which is suitable for a corresponding machine learning model.

Core processor <NUM> executes computer readable instructions stored at device <NUM> to support machine learning models for thermal sensor <NUM> and radar sensor <NUM>, feature analysis block <NUM>, and decision logic block <NUM> as shown in <FIG>. With some embodiments, core processor <NUM> may have a machine learning model implemented for both sensor types as shown in <FIG>.

When core processor <NUM> detects any hazardous physical conditions about the driver, it may execute one or more actions to address the detected physical conditions. The list of actions includes, but not limited to, reporting the detected physical conditions to the driver, doctor, or emergency contact via embedded wireless device <NUM> (for example, an LTE module) installed in the hybrid sensing system <NUM>, initiating a telephone call to <NUM>, and generating an e-mail message to a designated person.

Wireless device <NUM> may allow the health record of the driver to be loaded into core processor <NUM> via the wireless devices from a remote database server. The parametric of the decision logic supported by core processor <NUM> may be altered based on the heath record of the driver.

In addition, wireless device <NUM> may also allow the sensor data to be continuously streamed to cloud server <NUM> for storage or for real-time analyzing the health condition of the driver (if one chooses to execute the analytic model in the cloud instead of locally or to execute the model both locally as well as in the cloud for cross-checking purposes). Cloud server <NUM> may use the data from hybrid sensor system <NUM> to fine-tune the analytic model. Moreover, system <NUM> may train new models based on new sensor combinations. The re-trained or new analytic models may be downloaded from cloud server <NUM> to the core processor <NUM> to continuously improve the accuracy of the analytic models.

Downloaded information from cloud server <NUM> may include a chronicle health record and the conduct record of the user (driver) for reference. For example, if the downloaded conduct record indicates that the user has previously driven while intoxicated, system <NUM> may extract pertinent feature information that may be indicative of intoxication so that appropriate actions can the performed.

Also, system <NUM> may send the record of the driver's conduct to a control center if system <NUM> detects a hazard in order to assist with a decision making process. The chronicle health record, as well as the driver's conduct record, may be sent to an emergency service when a hazard occurs to allow for better preparation when the user arrives at the hospital.

All the downloaded information (for example, the sensor data as well as system outputs) may be temporarily stored in storage device <NUM> (which may also store computer readable instructions as previously discussed). The data may be retained until it is cleared by authorized persons, for example, after the data is backed up at the end of business hours. Moreover, the data may be retrieved under certain conditions by authorized persons, for example, when an accident happens that involves the user.

Core processor <NUM> may store biometric data (for example, sensor data, biometric vectors, and/or resultant biometric vectors) at storage device <NUM>. The stored biometric data may be subsequently retrieved to reconstruct an health event that occurred or to provide data for legal evidence.

Advanced functions may be implemented by core processor <NUM> when computational power is available. For example, if core processor <NUM> determines that the user (driver) is experiencing a heart attack, it may instruct initiate an autonomous driving unit via interface <NUM> to route the vehicle to the nearest hospital.

As previously discussed, a thermal signal (for example, from thermal sensor <NUM>) may be blocked (and consequently not available) in some situations; however, a received RF signal (for example, from radar sensor <NUM>) may be processed by itself to assess the vehicular operator's physical health.

<FIG> shows an embodiment of the processing that may supported by system <NUM>. The sequence of actions is similar to the ones as discussed with <FIG>. Health record <NUM> of the driver and profiles of the hazards to be detected/the associated actions for each hazard <NUM> may loaded into decision logic block <NUM>. By presenting feature vector to decision logic, block <NUM> is able to detect a particular hazard and to perform associated actions <NUM> from presented feature vectors <NUM>.

<FIG> shows flow chart <NUM> for decision logic block <NUM>. Different weightings are applied to the elements (corresponding to different feature information) in the resultant feature vector based on the health record of the driver. For example, if the driver is over certain age (for example, <NUM> years of age) and is obese (for example, <NUM> and <NUM>), more weight can be applied to the change of heart rate over time and to the breathing rhythm. As another example, if the user (driver) has a record of being a careless driver, more weight can be applied to the motion vectors. The weighted feature vector is then mapped to the list of hazards. An exemplary list of hazards may be:.

When a match of the hazard type is found, the list of associated actions may be performed. For example, if the driver is determined to be tired and prone to driving while asleep, the alert system in the vehicle may be activated in order to wake the driver. As another example, if critical biometric data is found to be abnormal (for example, a change of heart rate is abnormal), the alert system may instruct the driver to pull over the vehicle and a message may be sent to the control center to request a backup driver to take over the driving. As another example, if the driver is detected to be unconscious, the alert system in the vehicle may stop the vehicle. Moreover, the alert system may automatically request for emergency health service. If autonomous driving is available with the vehicle, the vehicle may be self-driven to the nearest emergency health service.

Claim 1:
An apparatus for assessing a health condition of a user by supporting biometric data (<NUM>) about the user, the apparatus comprising:
a thermal sensor interface (<NUM>, <NUM>) configured to obtain a thermal signal (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) from a thermal sensor (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>);
a radar sensor interface (<NUM>) configured to obtain a radar signal (<NUM>, <NUM>) from a radar sensor (<NUM>, <NUM>);
a processor (<NUM>, <NUM>, <NUM>, <NUM>) for executing computer-executable instructions;
a memory (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) storing the computer-executable instructions that when executed by the processor cause the apparatus to perform:
generating a first biometric vector from the thermal signal, wherein the first biometric vector contains first information about a first biometric feature and wherein the first biometric vector contains third information about a second biometric feature, wherein the first biometric feature and the second biometric feature are different biometric features;
generating a second biometric vector from the radar signal, wherein the second biometric vector contains second information about the first biometric feature and wherein the second biometric vector contains fourth information about the second biometric feature;
extracting first resultant information about the first biometric feature from the first and second information;
generating a first resultant biometric vector containing the first resultant information about the first biometric feature;
generating the first resultant biometric vector further containing second resultant information about the second biometric feature, wherein the second resultant information is extracted from the third information and the fourth information;
based on a health record of the user, applying a first weight to the first resultant information about the first biometric feature and a second weight to the second resultant information about the second biometric feature, wherein the first weight and the second weight are different; and
determining hazard information from the first resultant biometric vector, wherein the hazard information includes a hazard level (<NUM>, <NUM>) and a confidence level (<NUM>, <NUM>) and wherein the hazard level is indicative of an occurrence of a health event for the user and the confidence level is indicative of a degree of certainty of the hazard level.