Patent Description:
Occupancy detection, including presence detection and detection of the number of people occupying a space (also referred to herein as "head counting"), is an important function in smart homes and buildings. In particular, occupancy detection enables efficiency improvements in smart heating, ventilation and air conditioning (HVAC) systems, improved smart evacuation in emergency situations, discovering intruders or abnormal patterns of occupants' behaviors for security systems, and other improvements.

Some conventional occupancy detection systems rely on heat or infrared sensors to detect the occupancy of a space. However, these sensors are limited to their particular field of view, and may require a number of sensors for large or complex areas. Moreover, heat or infrared sensors can generate false positives due to heat sources, for example appliances, or areas subject to sunlight.

Some other conventional systems use radar and sonar sensors, chemosensors, or video cameras. However, these systems have limited detection distance, may have difficulty in detecting a static person, and/or may require significant computational complexity. Moreover, in particular with video sensors, there may be privacy concerns for occupancy detection using video signals in a dwelling or other private locations.

Another form of occupancy detection uses audio sensors to determine whether a space is occupied. However, using audio systems to determine occupancy within a space is difficult in some environments, conditions, and when there is background noise present in the space. For example, presence detection in an open office environment or in the presence of TV, radio, or other background audio noise is more difficult than in a quiet single room environment. Moreover, conventional audio-based occupancy detection systems require multiple sensors to accurately determine presence or head count, which increases energy consumption, upfront cost, and computational complexity for operating the system.

What is needed therefore is an improved occupancy detection system.

<CIT> provides a sensing system for use with at least one sensor having at least two defined states.

Non patent literature "<NPL>, provides voice controlled interactive smart speakers with verifying the presence of a user.

In the present invention, a method of detecting occupancy in an area comprises obtaining, with a processor, an audio sample from an audio sensor and determining, with the processor, feature functional values of a set of selected feature functionals of statistical operations from the audio sample, as defined in independent claim <NUM>. The determining of the feature functional values comprises extracting audio low-level descriptive (LLD) features from the audio sample using a frame-level sliding window with no overlap, partitioning the extracted audio LLD features into segment features determined by aggregating feature frames in segments with a fixed length in time and a shift of one frame, wherein the segments overlap one another such that each segment is calculated shifted by one frame, and determining the feature functional values of the determined segment features. The method further includes selecting audio LLD features that contain information most relevant of occupancy of the area from the determined feature functional values based on a correlation or independence of the selected feature functionals of the statistical operations. The method further includes determining, with the processor, occupancy in the area using a classifier based on the determined feature functional values of a set of selected audio LLD features.

In some embodiments of the method, the classifier is a decision tree classifier.

In a further embodiment, the decision tree classifier has maximum depth of between <NUM> and <NUM>. In another embodiment, the decision tree classifier has a maximum depth of five.

In yet another embodiment of the method, the set of feature functionals includes between <NUM> and <NUM> feature functionals. In another embodiment, the set of feature functionals includes between <NUM> and <NUM> feature functionals. In one particular embodiment, the set of feature functionals includes <NUM> feature functionals.

The audio LLD features, in some embodiments, include one or more of envelope dynamic range, zero crossing rate, energy, brightness, spectral variance, spectral roll off, spectral flux, at least one MFCC coefficient, a delta of at least one MFCC coefficient, and a delta-delta of at least one MFCC coefficient. In further embodiments, the set of feature functionals includes at least one of the group consisting of: mean, median, standard deviation, absolute integral, minimum, maximum, dynamic range, dominant-frequency, and entropy, determined for each of the audio LLD features.

In one embodiment, the set of feature functionals includes at least two selected from the group consisting of: maximum MFCC-<NUM>; mean energy; dynamic range of envelope dynamic range; mean of brightness; dynamic range of brightness; median of brightness; entropy of MFCC-<NUM> delta; standard deviation of spectral flux; entropy of MFCC-<NUM>; standard deviation of envelope dynamic range; entropy of envelope dynamic range; absolute integral of MFCC-<NUM> delta; entropy of zero crossing rate; absolute integral of brightness; entropy of spectral roll off; entropy of brightness; entropy of spectral flux; entropy of spectral variance; entropy of MFCC-<NUM>; entropy of MFCC-<NUM> delta; entropy of MFCC-<NUM> delta-delta; entropy of MFCC-<NUM>; entropy of energy; entropy of MFCC-<NUM>; and entropy of MFCC-<NUM> delta.

In yet another embodiment, the set of feature functionals includes at least two selected from the group consisting of: maximum MFCC-<NUM>; mean energy; dynamic range of envelope dynamic range; mean of brightness; dynamic range of brightness; median of brightness; entropy of MFCC-<NUM> delta; standard deviation of spectral flux; entropy of MFCC-<NUM>; standard deviation of envelope dynamic range; entropy of envelope dynamic range.

In some embodiments of the method, the set of selected feature functionals and the classifier are learned in a machine-learning training process.

In the present invention, a system for determining occupancy in an area comprises at least one audio sensor configured to record an audio sample in the area and a processor, as defined in independent claim <NUM>. The processor is configured to execute programmed instructions stored in a memory to obtain the audio sample from the audio sensor, determine feature functional values of a set of selected feature functionals of statistical operations from the audio sample, select audio LLD features that contain information most relevant of occupancy of the area from the determined feature functional values based on a correlation or independence of the selected feature functionals of the statistical operations, and determine occupancy in the are using a classifier based on the determined feature functional values of a set of selected audio LLD features. The determining of the feature functional values comprises extracting audio low-level descriptive (LLD) features from the audio sample using a frame-level sliding window with no overlap, partitioning the extracted audio LLD features segment features determined by aggregating feature frames in segments with a fixed length in time and a shift of one frame, wherein the segments overlap one another such that each segment is calculated shifted by one frame, and determining the feature functional values of the determined segment features.

In one embodiment of the system, the classifier is a decision tree classifier.

In another embodiment, the decision tree classifier has maximum depth of between <NUM> and <NUM>.

In a further embodiment according to the disclosure, the set of feature functionals includes between <NUM> and <NUM> feature functionals.

In some embodiments of the system, the audio LLD features include one or more of envelope dynamic range, zero crossing rate, energy, brightness, spectral variance, spectral roll off, spectral flux, at least one MFCC coefficient, a delta of at least one MFCC coefficient, and a delta-delta of at least one MFCC coefficient; and the set of feature functionals include at least one of the group consisting of: mean, median, standard deviation, absolute integral, minimum, maximum, dynamic range, dominant-frequency, and entropy, determined for each of the audio LLD features.

In yet another embodiment, the set of feature functionals includes at least two selected from the group consisting of: maximum of MFCC-<NUM>; mean of energy; dynamic range of envelope dynamic range; mean of brightness; dynamic range of brightness; median of brightness; entropy of MFCC-<NUM> delta; standard deviation of spectral flux; entropy of MFCC-<NUM>; standard deviation of envelope dynamic range; entropy of envelope dynamic range; absolute integral of MFCC-<NUM> delta; entropy of zero crossing rate; absolute integral of brightness; entropy of spectral roll off; entropy of brightness; entropy of spectral flux; entropy of spectral variance; entropy of MFCC-<NUM>; entropy of MFCC-<NUM> delta; entropy of MFCC-<NUM> delta-delta; entropy of MFCC-<NUM>; entropy of energy; entropy of MFCC-<NUM>; and entropy of MFCC-<NUM> delta.

For the purposes of promoting an understanding of the principles of the embodiments described herein, reference is now made to the drawings and descriptions in the following written specification.

Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order than the described embodiment. Various additional operations may be performed and/or described operations may be omitted in additional embodiments.

The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the disclosure, are synonymous. As used herein, the term "approximately" refers to values that are within ±<NUM>% of the reference value.

As used herein, the term "presence detection" refers to detecting whether any individuals are present in an area, and the term "head counting" refers to detecting the quantity of individuals in an area. As used herein, the terms "detect occupancy" and "occupancy detection" can refer to either presence detection or head counting.

<FIG> schematically illustrates an exemplary embodiment of an occupancy detection system <NUM> that detects occupancy, for example detecting the presence of at least one individual and/or detecting the quantity of people, in an area <NUM>. The area <NUM> may be a single room, a plurality of rooms, a desired portion of a single room or plurality of rooms, an outdoor area, any other area in which it is desired to detect occupancy, or any combination of the above. The occupancy detection system <NUM> is configured to monitor characteristics of one or more audio signals obtained from the area <NUM> and to determine whether the area <NUM> is occupied, or how many people occupy the area <NUM>. In particular, the system <NUM> may utilize a machine learning model to classify feature functionals of the detected audio to determine occupancy of the area <NUM>.

The occupancy detection system <NUM> includes a sensor package <NUM> that includes a processor <NUM> operably connected to a memory <NUM>, and a single audio sensor <NUM>, which can, for example, be a microphone or other suitable audio receiving device. In one embodiment, the audio sensor <NUM> may be a MEMS audio sensor formed as an application-specific integrated circuit (ASIC). The audio sensor <NUM> is configured to sense sound pressure waves and convert the sound pressure waves into a digital or analog electronic signal. The electronic signal is transmitted from the audio sensor <NUM> to the processor <NUM> via wired or wireless communication. In some embodiments, the system <NUM> may also include a display <NUM> operably connected to the processor <NUM> and configured to inform a user as to whether the area <NUM> is occupied and/or how many people occupy the area <NUM>.

In some embodiments, the sensor package <NUM> may be part of another electronic device. For instance, in one embodiment, the sensor package <NUM> may be integrated in a computer, a smart home hub, an alarm controller, a portable electronic device such as a cellular telephone, a tablet, a smart watch, or the like. Moreover, in such an embodiment, the processor <NUM> and memory <NUM> may be the processor and memory used in the electronic device for general functioning of the electronic device, while the audio sensor <NUM> may be the microphone integrated in the electronic device. In other embodiments, however, the sensor package <NUM> may be a dedicated sensor package. In further embodiments, the processor <NUM>, memory <NUM>, audio sensor <NUM>, and/or the display <NUM> may be separate, while the other components may be integrated in an electronic device.

As discussed in more detail below, the processor <NUM> is configured to process the audio signals and to use a classifier model on the detected audio signals to determine the occupancy in a room. It will be recognized by those of ordinary skill in the art that a "processor" includes any hardware system, hardware mechanism or hardware component that processes data, signals or other information. The processor <NUM> may include a system with a central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems.

The memory <NUM> is configured to store program instructions that, when executed by the processor <NUM>, enable the sensor package <NUM> to perform various operations described below, including determining the occupancy in the area <NUM>. The memory <NUM> may be of any type of device capable of storing information accessible by the processor <NUM>, such as write-capable memories, read-only memories, or other computer-readable mediums.

In particular, the memory <NUM> is configured to store program instructions corresponding to at least one machine learning model, in particular to an occupancy classification model and classification parameters thereof. The processor <NUM> is configured to utilize the occupancy classification model to extract features from the audio signal or signals and to classify whether the area is occupied and/or how many individuals occupy the area. As used herein, the term "machine learning model" refers to a system or set of program instructions and/or data configured to implement an algorithm or mathematical model that predicts and provides a desired output based on a given input. It will be appreciated that parameters of a machine learning model are not explicitly programmed or the machine learning model is not necessarily designed to follow particular rules in order to provide the desired output for a given input. Instead, the machine learning model is provided with a corpus of training data from which the processor identifies or "learns" patterns and statistical relationships or structures in the data, which are generalized to make predictions with respect to new data inputs. The classification parameters include a plurality of values for parameters of the occupancy classification model which were learned during a training process.

While the embodiment of <FIG> includes only one sensor package <NUM> having only one audio sensor <NUM>, the reader should appreciate that in other embodiments, as illustrated in <FIG>, the system <NUM> may include two, three, or any suitable number of sensor packages <NUM> or audio sensors <NUM> as desired for a particular application.

<FIG> illustrates an occupancy detection system 100A that includes a plurality of sensor packages <NUM>, two of which are shown in <FIG>. The reader should appreciate, however, that the occupancy detection system 100A may include any desired number of sensor packages 110A. Each sensor package 110A may be configured similar to the sensor package <NUM> illustrated in <FIG>, such that each sensor package 110A may include a processor and an audio sensor. In some embodiments, the sensor package 110A may also include a wireless data transceiver (not shown), and/or a battery (not shown) so as to enable the sensor packages 110A to be easily positioned at desired locations within the area <NUM>. In further embodiments, the sensor packages 110A may include only an audio sensor, only an audio sensor and a transceiver, only an audio sensor and a battery, or only an audio sensor, a transceiver, and a battery.

The occupancy detection system 100A also includes a system controller <NUM>, which communicates with the sensor packages 110A to obtain digital or analog signals from the sensor packages 110A corresponding to the audio signals received by the respective audio sensors. The system controller <NUM> is then configured to determine the occupancy of the area <NUM>, as discussed in detail below. The system controller may be located in the area <NUM>, or, as illustrated in <FIG>, may be located outside the area. The system controller <NUM> may be connected to or integrated in, for example, a security system controller, a building emergency control unit, an HVAC controller, a computer, a portable electronic device, or another desired controller. In an alternative embodiment, the sensor packages 110A may be connected to one another for data transmission via a wired or wireless connection and configured such that the controller of one of the sensor packages 110A performs the determination of the occupancy. In some such embodiments, a system controller <NUM> may be omitted from the sensor system 100A.

The system controller <NUM> includes a processor <NUM> operably connected to a memory <NUM>, a transceiver <NUM>, and, in some embodiments, a display <NUM>. The transceiver <NUM> includes for example, one or more of a Wi-Fi® transceiver, a ZigBee® transceiver, a Z-Wave® transceiver, a Bluetooth® transceiver, a wireless telephony transceiver, and RF transceiver, or another transceiver suitable to send and receive communication signals to and from the sensor packages 110A.

It will be recognized by those of ordinary skill in the art that a "processor" includes any hardware system, hardware mechanism or hardware component that processes data, signals or other information. The processor <NUM> may include a system with a central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems.

The memory <NUM> may be of any type of device capable of storing information accessible by the processor <NUM>, such as write-capable memories, read-only memories, a memory card, ROM, RAM, hard drives, discs, flash memory, or other computer-readable medium. The memory <NUM> is configured to store program instructions that, when executed by the processor <NUM>, enable the controller <NUM> to perform various operations described elsewhere herein, communicating with the sensor package 110A to receive the audio signal and classify the occupancy of the area using a machine learning model.

In particular, the memory <NUM> is configured to store program instructions corresponding to at least one machine learning model, in particular to an occupancy classification model and classification parameters thereof. The processor <NUM> is configured to utilize the occupancy classification model to extract features from the audio signal or signals and to classify whether the area is occupied and/or how many people occupy the area.

<FIG> illustrates a machine learning process <NUM> for training a machine-learning occupancy detection system such as the occupancy detection systems <NUM>, 100A of <FIG> to detect the occupancy of an area. In the description of the methods, statements that a method is performing some task or function refers to a controller or general purpose processor executing programmed instructions stored in non-transitory computer readable storage media operatively connected to the controller or processor to manipulate data or to operate one or more components in the occupancy detection systems <NUM>, 100A to perform the task or function. Particularly, the processor <NUM> of the sensor package <NUM> or 110A or the processor <NUM> of the system controller <NUM> described above may be such a controller or processor. Alternatively, the controller or processor may be implemented with more than one processor and associated circuitry and components, each of which is configured to form one or more tasks or functions described herein. It will be appreciated that some or all of the operations the method can also be performed by a remote server or cloud processing infrastructure. Additionally, the steps of the methods may be performed in any feasible chronological order, regardless of the order shown in the figures or the order in which the steps are described.

The process <NUM> begins by obtaining an audio sample (block <NUM>). In one embodiment, the audio sample is sensed by a single audio sensor, for example the audio sensor or microphone <NUM>. In another embodiment, the audio sample is sensed by a plurality of audio sensors <NUM> that are adjacent to one another or spread out over the area in which the occupancy detection determination is performed. The processor communicates with the one or more audio sensors to receive a time series of acoustic values corresponding to the detected audio in the area. The processor obtains the sensed audio sample from the audio sensor(s) via direct connection or via communication over a wired or wireless network.

Next, the method proceeds with extracting features from the audio sample (block <NUM>). The processor extracts audio low-level descriptive (LLD) features using a frame-level sliding window with no overlap, as summarized in Table <NUM>. The frame lengths from which the LLD features are extracted may be from approximately <NUM> to approximately <NUM>. In some embodiments, the length of the frames is dependent on the features detected in the audio signal. In another embodiment, the LLD feature frame lengths are between approximately <NUM> and approximately <NUM>. In one particular embodiment, the LLD feature frame lengths are approximately <NUM>.

The LLD features extracted from the audio sample can be grouped into three main categories: time domain features, frequency domain features, and cepstral domain features. The time domain features can include, for example, the envelope dynamic range (i.e. the range of the temporal envelope of the signal) and zero crossing rate (i.e. the number of time-domain zero crossings of the signal within a processing frame, which is indicative of the frequency of signal amplitude sign change), both measured in a single dimension. The frequency domain features can include, for example, energy of the signal (i.e. the summation of the square power of the signal), brightness (i.e. the measure of high-frequency content in the signal, measured using the spectral centroid, or the weighted mean of the frequencies, of the signal), spectral variance (i.e. the statistical variance of the frequency spectrum), spectral roll-off (the frequency under which a specified N percentile of the total energy of the power spectral distribution is contained; useful for distinguishing voiced speech from unvoiced noise), and spectral flux of the audio signal (represented by, for example, a two-norm of the frame-to-frame spectral amplitude difference vector, which defines the amount of frame-to-frame fluctuation in time), again measured in a single dimension.

The cepstral domain features mel-frequency cepstral coefficients (MFCC) and their differential (also referred to as "delta" or "d") and acceleration (also referred to as "delta-delta" or "dd") coefficients. MFCC's are coefficients that are commonly used in the art to enable automated frequency detection to interpret frequency differences more like the human ear. In some embodiments, the frames used for the MFCC's are, for example, between approximately <NUM> to approximately <NUM>, between approximately <NUM> to approximately <NUM>, or approximately <NUM>. In the embodiments described below, the MFCC's are calculated with <NUM> filter banks. The reader should appreciate, however, that, in other embodiments, MFCC's are calculated using between <NUM> and <NUM> mel-frequency filter banks. In certain embodiments, only coefficients of the lower <NUM>-<NUM> mel-frequency filter banks are kept, while the coefficients of the higher filter banks may be discarded to arrive at the MFCC's for each frame. In other embodiments, the coefficients of the lower <NUM> mel-frequency filter banks are retained, with the remaining filter banks discarded.

The process <NUM> continues by partitioning the features into segments with a fixed length in time and a shift of one frame (block <NUM>). Thus, the feature frames are aggregated over the segment length to determine the segmented features. The segments may have length of, for example, between <NUM> second and <NUM> seconds. In one embodiment, the segments have length of between <NUM> seconds and <NUM> seconds. In one particular embodiment, the features are partitioned into <NUM> second segments. In other embodiments, the features are partitioned into different segment lengths depending on the feature functionals applied to the particular features. Furthermore, in some embodiments, the processor analyzes different segment lengths for the features to investigate the optimum time window for the particular occupancy detection task. The optimum time window may be based on the features in the audio signal and/or the features of the area in which the occupancy detection is performed.

The segments overlap one another such that each segment is calculated shifted by one feature frame.

Next, the process <NUM> proceeds with the processor applying functionals to the determined LLD features and the respective delta and acceleration coefficients of the LLDS for each segment (block <NUM>). As illustrated in Table <NUM>, the functionals applied are statistical operations including, for example, one or more of determining the mean, median, standard deviation, absolute integral, minimum, maximum, dynamic range, dominant-frequency, or entropy (determined, for example, using the Shannon entropy equation) of the determined and segmented LLDs. The processor may be configured to determine every functional of every LLD feature as the determined feature functionals. Alternatively, the processor may be configured to determine a limited number of feature functionals to reduce computational resources necessary for the determination of the feature functionals.

The process <NUM> continues with the processor selecting feature functionals that contain information most relevant of occupancy of an area from the determined feature functionals (block <NUM>). In particular, the controller analyzes the contribution of different audio feature types determined from the LLDs for the classification accuracy. Since there may be a large number of possible audio features in the audio sample, the selecting of the features includes performing transformations of the features or selecting only a subset of the features to analyze. Reducing the number of features for classification improves the speed and reduces the complexity of the computation. In particular, the feature selection methods pool together the most relevant and uncorrelated features and define the effect of each feature in presence detection and head counting tasks.

Feature selection as an automatic method to select the most relevant features to the modeling problem has many benefits, such as improving the performance, providing a faster and simpler model that requires reduced computational resources, and allowing better understanding of the data and its underlying process. Different feature selection methods put more emphasis on one aspect than others. In some embodiments, the feature selection may include univariate chi-squared (χ<NUM> or Chi2) statistical analysis and/or least absolute shrinkage and selection operator (LASSO) statistical, in particular LASSO using the ℓ1 penalty. Both of these feature selection methods provide simple, quick, and effective feature selection.

In the feature selection, the correlation or independence of the various different feature functionals are determined using chi-squared and/or LASSO analysis. The processor determines the correlation between different feature functionals and the correlation between the features and known results (i.e. known presence or head count in the area). Feature functionals that exhibit low correlation with the known results are removed (i.e. not selected) because these feature functionals do not contain sufficient information relevant to the presence or head count determination. Conversely, features that are strongly correlated with the known results are retained. Additionally, features that are strongly correlated with one another may be discarded such that only one of the strongly correlated features remains. This enables the presence detection or head counting process to be performed with fewer feature calculations, and therefore less computational resources are necessary for the process.

The feature extraction may include more than <NUM> feature functionals that can be extracted from the data. However, many of these feature functionals are correlated with one another, or are uncorrelated with the presence detection and/or head count. In the feature selection step, the processor is configured to rank the best feature functionals for determining presence or head count. In one particular embodiment, <NUM> feature functionals are selected. In a further embodiment, the processor selects only the <NUM> best feature functionals (i.e. the feature functionals exhibiting high correlation with the known results, while limiting feature functionals correlated with one another to only one of the correlated feature functionals) in the feature selection step.

The number of feature functionals selected may vary in different embodiments based on whether the system is configured for presence detection or for head counting. In particular, accurate head counting may require more features than presence detection, since head counting requires determining not only if people are present, but the quantity of people present in the area.

Finally, the process concludes by classifying the selected features (block <NUM>). In some embodiments, the controller <NUM>, <NUM> generates a decision tree classifier, which advantageously has a fast inference time, is simple to interpret, and is computationally efficient. A decision tree is a type of supervised machine learning in which the controller continuously splits the data along decision nodes until arriving at a leaf, or final outcome. In other embodiments, depending for example on the amount of training data available, the computational resources available and the required online factor, other classifiers such as support vector machine, deep neural networks, etc. may be used in place of the decision tree classifier.

In the process <NUM>, the decision tree may be determined using a recursive binary splitting procedure, for example using greedy splitting and/or Gini impurity decision criterion. The decision tree may be configured for a number of parameters. In one embodiment, the decision tree classifier may be configured with a prespecified depth, minimum size for split, minimum leaf size, etc. In addition, in certain embodiments, an ensemble decision tree, which combine a plurality of independently generated decision trees (i.e. multiple "estimators") using the audio data to generalize the classifier, which can in some embodiments improve the robustness of the process <NUM>.

In various embodiments, the decision tree may be generated with, for example, between <NUM> and <NUM> estimators, a maximum depth of between <NUM> and <NUM>, a minimum size for split of between <NUM> and <NUM>, and a minimum leaf size of between <NUM> and <NUM>. In one particular embodiment, the decision tree is generated using one estimator, maximum depth of <NUM>, minimum size for split of <NUM>, minimum leaf size of <NUM>, and Gini impurity decision criterion. The reader should appreciate, however, that in other embodiments any desired values may be used for the number of estimators, maximum depth, minimum size for split, and minimum leaf size.

In at least one embodiment, the training process is performed on an external device, such as a server (not shown), and the resulting classification parameters are provided to the occupancy detection system <NUM>, 100A for storage in the memory <NUM>, <NUM> and subsequent usage. In such embodiments, the system may be easily adapted to a variety of different uses with reduced outlay and installation cost.

In another embodiment, the training process is performed as a system calibration when the occupancy detection system <NUM>, 100A is installed and the training data is then stored in the memory <NUM>, <NUM> for subsequent use. The occupancy detection system machine learning algorithm is therefore tailored to the specific area in which the system is installed. In such embodiments, a high degree of accuracy is obtainable since the classification parameters are based on the specific characteristics of the area in which the system is installed.

<FIG> illustrates a flow chart of a process <NUM> for determining occupancy in an area. The process <NUM> refers to a processor, for example the processor <NUM> or <NUM> executing programmed instructions stored in a memory, for example memory <NUM> or <NUM>, to perform the functions described below to detect the occupancy of an area. In the description of the methods, statements that a method is performing some task or function refers to a controller or general purpose processor executing programmed instructions stored in non-transitory computer readable storage media operatively connected to the controller or processor to manipulate data or to operate one or more components in the occupancy detection systems <NUM>, 100A to perform the task or function. Particularly, the processor <NUM> of the sensor package <NUM> or 110A or the processor <NUM> of the system controller <NUM> described above may be such a controller or processor. Alternatively, the controller or processor may be implemented with more than one processor and associated circuitry and components, each of which is configured to form one or more tasks or functions described herein. It will be appreciated that some or all of the operations the method can also be performed by a remote server or cloud processing infrastructure. Additionally, the steps of the methods may be performed in any feasible chronological order, regardless of the order shown in the figures or the order in which the steps are described. By way of example, in some embodiments, the process <NUM> may be performed by a computer, a smart home hub, an HVAC controller, an alarm controller, a portable electronic device such as a cellular telephone, a tablet, or a smart watch, or the like.

In some embodiments, the process <NUM> begins by calibrating or training the system (block <NUM>). The training may, for example, be performed using the process <NUM> of <FIG>. The selected features, functionals, and/or classifier, for example the decision tree, from the calibration or training of the system are stored in the memory, e.g. memory <NUM> or <NUM>, associated with the processor. In other embodiments, the system may be pre-programmed with the selected features, functionals, and/or classification algorithm such that some or all of the calibration or training of the machine-learning occupancy detection system is not necessary. In one embodiment, between <NUM> and <NUM> feature functionals are selected. In another embodiment, between <NUM> and <NUM> feature functionals are selected. In one particular embodiment, <NUM> feature functionals are selected.

The process then proceeds with the processor obtaining an audio sample (block <NUM>) using an audio sensor, for example a microphone. In one embodiment, the audio sample is sensed by a single audio sensor, for example the audio sensor or microphone <NUM>. In some embodiments, the audio sample is sensed by a computer, a smart home hub, an alarm controller, a portable electronic device such as a cellular telephone, a tablet, or a smart watch, or the like. In another embodiment, the audio sample is sensed by a plurality of audio sensors that are adjacent to one another or spread out over the area in which the occupancy detection determination is performed. The processor communicates with the one or more audio sensors to receive a time series of acoustic values corresponding to the detected audio in the area. The processor obtains the sensed audio sample from the audio sensor(s) via direct connection or via communication over a wired or wireless network.

Next, the processor determines the selected feature functionals from the audio sample (block <NUM>). The processor <NUM> or <NUM> extracts the audio LLD features, segments the extracted features, and determines the feature functionals in a similar manner as described above in the process <NUM> of <FIG>. The frame lengths from which the LLD features are extracted may be from approximately <NUM> to approximately <NUM>. In some embodiments, the length of the frames is dependent on the features detected in the audio signal. In another embodiment, the LLD feature frame lengths are between approximately <NUM> and approximately <NUM>. In one particular embodiment, the LLD feature frame lengths are approximately <NUM>.

As in the process <NUM> described above, the segments may have length of, for example, between <NUM> second and <NUM> seconds. In one embodiment, the segments have length of between <NUM> seconds and <NUM> seconds. In one particular embodiment, the features are partitioned into <NUM> second segments. In other embodiments, the features are partitioned into different segment lengths depending on the feature functionals applied to the particular features. Furthermore, in some embodiments, the processor analyzes different segment lengths for the features to investigate the optimum time window for the particular occupancy detection task. The optimum time window may be based on the features in the audio signal and/or the features of the area in which the occupancy detection is performed. The segments overlap one another such that each segment is calculated shifted by one feature frame.

In contrast to the training process <NUM> described above, the process <NUM> for determining occupancy is limited in the number of feature functionals determined. The reader should appreciate, however, that any desired number of feature functionals may be used depending on the desired accuracy and computational resources available. Additionally, the selected features may vary based on the data received from executing the machine learning models. As discussed above with regard to the training and calibration process <NUM>, and as will be explained with reference to experimental results below, the selected feature functionals may be those feature functionals that provide the greatest amount of information related to the presence and/or head count in the area.

Table <NUM> lists <NUM> feature functionals determined to include information relevant to head counting from an experimental training process discussed in detail below. In one embodiment, all <NUM> feature functionals from the table are selected. In another embodiment, between <NUM> and <NUM> of the feature functionals from Table <NUM> are selected. In another embodiment, <NUM> of the feature functionals from Table <NUM> are selected. In some embodiments, the number of selected feature functionals selected may be selected from Table <NUM> in descending order (i.e. an embodiment with <NUM> selected feature functionals may use feature functionals <NUM>-<NUM> in Table <NUM>).

The process <NUM> continues with the processor <NUM> or <NUM> determining the occupancy of the area based on the feature functionals using a classifier (block <NUM>). As discussed above, the classifier may be developed using a machine learning model such as the machine-learning training process of <FIG>. The determination of the occupancy may be performed using a decision tree.

In one particular embodiment, the decision tree used may have, for example, a depth of between <NUM> and <NUM>. In one particular embodiment, the decision tree used has maximum depth of <NUM>. The reader should appreciate, however, that in other embodiments any desired values may be used for the maximum depth and other parameters of the decision tree.

The decision tree classifier determines, based on the input audio data as segmented and analyzed via feature functionals, what the occupancy of the area is likely to be. The decision tree output may be, in one embodiment, probability of a certain number of people present in the area, or probability that any individuals are present in the area. In another embodiment, the decision tree output may be a value of the determined number of individuals present in the area, or presence or lack of presence in the area.

The method <NUM> continues by generating an output based on the determination of occupancy (block <NUM>). The output may, in one embodiment, be a perceptible output depending on the occupancy determination made using the classifier depicted on the display <NUM> or <NUM>. The perceptible output may include an indication of whether presence is detected in the area or whether no presence is detected in the area based on the determination made using the classifier. In another embodiment, the perceptible output may be an indication on the display of the quantity of people in the area based on the determination made using the classifier. In other embodiments, the perceptible output may be an audible indicator, such as an alert or alarm, or light indicator.

In a further embodiment, the output is an electronic signal transmitted to another electronic device or stored in a memory or the memory <NUM>, <NUM>. For example, the output may be an electronic signal output to a computer, a smart home hub, an HVAC controller, an alarm controller, a portable electronic device such as a cellular telephone, a tablet, or a smart watch, or the like. The received output may cause the electronic device to execute programmed instructions to, for example, activate an alarm, operate HVAC systems, activate or deactivate lights, or perform other automated functions.

The disclosed systems <NUM>, 100A and processes <NUM>, <NUM> provide a number of improvements to computer and occupancy detection technology by affording an efficient and cost-effective way to increase occupancy detection performance over conventional systems. The system <NUM> and processes <NUM>, <NUM> enable detection of occupancy and head counting in realistic scenarios using only audio signals collected from the environment. Audio processing is generally less computationally intensive than other occupancy detection technologies, for instance video processing, and therefore the disclosed audio based occupancy detections system requires less computational resources as compared to conventional occupancy detection systems. Furthermore, the use of audio signals enables the system <NUM> to be easily accessible in a variety of different environments and scenarios.

In addition, using audio signals improves the accuracy of the occupancy detection in different applications as compared to conventional systems. Using only audio signals is more desired in some applications because the audio detection is considered to be less intrusive to privacy as compared to other conventional occupancy detection systems and methods such as, for example, video occupancy detection. Additionally, the disclosed system and process provides excellent coverage of a room as compared to conventional occupancy detection systems that have a limited field of view (e.g. infrared or video based systems) or are constrained based on the position of the sensor.

Moreover, the occupancy detection used in the disclosed system and process enables determination of a limited number of features and feature functionals that are likely to provide information on whether an area is occupied and/or how many people occupy the area. As a result, the required computational resources, and therefore the energy costs of the system are reduced. The occupancy detection process can therefore be performed on devices with reduced computational power as compared to conventional methods. For instance, the occupancy detection process may be performed on a portable electronic device, such as a cellular telephone, tablet, or smart watch.

Two experiments were performed using the system <NUM> and processes <NUM>, <NUM> in a simulated living room environment and in a simulated office environment. In both experiments, functionals were applied with time windows of <NUM> seconds. In addition, a leave-one-recording-out cross-validation method was used in the experiments as an evaluation technique. Consequently, <NUM>-fold and <NUM>-fold cross-validation were used in the living room and single office scenarios, respectively. Finally, classification accuracy was used as the performance measurement.

The audio samples utilized in the first experiment were drawn from a multisensory occupancy detection corpus that includes recordings collected in a single office environment (representative of a commercial scenario), while the audio samples from the second experiment were drawn from a simulated living room environment (representative of a residential scenario). The audio data was recorded in <NUM> using a MEMS acoustic sensor manufactured by Robert Bosch GmbH with the model number AKU151. It is noted that the sensor used is specifically designed for use in space-constrained consumer electronic devices, for example portable electronic devices. The number of people present at a given time in the experimental environments was available throughout the entire corpus based on video recordings to verify the accuracy of the occupancy detection.

The single office data was acquired in a small, enclosed room including one fully equipped office work place. The recordings were carried out over seven days during daytime on workdays. The recorded audio data mostly includes the regular office work of a single person, including phone conversations and longer meetings with a second colleague. Additionally, in order to increase the data variety and balance the presence ratio, data was recorded on a day off and over one night. Approximately <NUM> hours of audio data were collected form the single office scenario. The percent of sampled time of each head count in the single office scenario is illustrated in <FIG>.

The living room data was acquired in a larger lab room that was furnished as a simple living room setting. Data was recorded over six sessions, each session following a predefined protocol with activities in varying order. The following activities were carried out: watching TV, reading newspaper, talking, and playing a card game. The number of people present in the room as well as an approximate length of each activity was defined in the protocol. Approximately <NUM> hours of audio data were collected from the living room scenario. The percent of sampled time of each head count in the simulated living room is illustrated in <FIG>.

The classifier performance on the full feature set in the living room environment is shown in <FIG> and <FIG>. <FIG> illustrates a plot of the decision tree performance with respect to maximum depth with one estimator, minimum size for split of <NUM>, minimum leaf size of <NUM>, and Gini impurity decision criterion. As observed in the experimental embodiment, using a deeper tree may result in an over-fitted model and, further, the deeper tree increases the computational complexity of the model.

<FIG> illustrates the boosted decision tree classification performance with respect to the number of estimators, with the other parameters of the model kept the same. Based on this figure, in the experimental embodiment, performance gain by using the ensemble method is minimal. At the end, a simple decision tree with maximum depth of five and only one estimator is chosen as the baseline system for the experimental embodiments due to the performance and computational cost trade-off. Because the experiments depicted in <FIG> and <FIG> required lengthy computations, the same experiments were not repeated for the single office environment on the full feature set, and instead the same decision tree parameters were used for both the living room and single office environments.

In the next experiments, the feature variable informational contributions in head counting tasks were analyzed using Chi2 and LASSO methods in the living room scenario. <FIG> shows the system accuracy with respect to number of features chosen via the Chi2 method. As seen in the <FIG>, out of hundreds of feature functionals, choosing only two feature functionals that have the highest Chi2 value between the feature variables and known results improves the performance up to <NUM>% absolute value and greatly increases the speed of the classifier.

<FIG> depicts the classifier accuracy with respect to features as selected using the LASSO method in living room scenario. The results of the LASSO selected feature functionals are generally in agreement with the results from Chi2 feature selection. Given the similar accuracy trend between Chi2 and LASSO methods in the living room scenario, the experiments were continued using LASSO feature functional selection for the single office environment. As illustrated in <FIG>, selecting a relatively low number of feature functionals provides a reasonable classification performance in the single office scenario as well. The reader should note that the results from LASSO and Chi2 feature functional selection methods are in agreement with the decision tree classification results in <FIG> and <FIG>, in which an only one- or two- layer decision tree can yield accurate classification results.

Table <NUM> summarizes the top eighteen most relevant feature functionals determined for both the living room and single office environments. <FIG>, respectively, depict the LASSO coefficients for each of the feature functionals in the order of Table <NUM> for the living room and single office scenarios, respectively. As shown in Table <NUM>, the most important feature chosen in both experimental environments was an energy term (it is noted that the first MFCC coefficient is a representation of the audio signal energy representation as well). This result is expected heuristically. However, two different functionals, namely maximum and mean, were selected as the functional with the highest LASSO coefficient for the living room and single office scenarios, respectively. Also, it appears from the selected features in the experimental embodiments that the time and frequency audio features are considered more relevant in the experimental head counting scenarios as compared to the cepstral (MFCC) features.

To compare the best feature functionals across both environments, head count performance vs. attributes was then studied, as illustrated in <FIG> for the living room and single office environments, respectively. In these figures, the x-axis represent the attributes exploited in each experiment. #-LR and #-SO, respectively, represent the feature number selected in the living room (LR) and single office (SO) scenarios from Table <NUM>. Moreover, the + sign at the beginning of the x-axis label shows the attribute accumulation from the previous step. For example, the first point in the plot (<NUM>-LR) represents the accuracy when using the first best feature in living room scenario (Max-mfcc <NUM>), <NUM>-LR+<NUM>-SO shows the performance when using first feature of both environments (Max-mfcc1 and Mean-Energy), +<NUM>-LR represents the accuracy when using <NUM> dimensional features <NUM>-LR, <NUM>-SO and <NUM>-LR, and so on. Based on both plots, <NUM> dimensional features (<NUM>-LR point on the x-axis) was chosen as the final set, with <NUM>% and <NUM>% head count accuracy for living room and single office environments, respectively.

<FIG> and <FIG> illustrate a decision tree algorithm determined from the data of the experimental living room scenario. In the flow chart of <FIG>, the subscript for the X splitting variable is the feature functional number from Table <NUM> above minus one (i.e. X<NUM> refers to feature <NUM> - Maximum of MFCC1; X<NUM> refers to feature <NUM> - mean of energy; etc.). The "value" array has four values, corresponding to actual (known) head counts of [<NUM>, <NUM>, <NUM>, <NUM>] in the experimental living room scenario.

The reader should appreciate that the decision tree of <FIG> and <FIG> is an experimental example only. The nodes, leaves, and splits used in other embodiments may vary depending on a number of factors, including the size of the area, usage of the area, selected feature functionals, desired processing power used, desired accuracy, desired use of the head counting data, etc..

Moreover, as illustrated in <FIG>, confusion matrices of head count task for the two environments were calculated using the <NUM>-dimensional final feature set. As seen in <FIG>, presence detection has a high performance rate using only a single audio signal, with <NUM>% accuracy for living room scenario and <NUM>% accuracy for the single office scenario. As a result, it is evident that an audio sensor can be used confidently as the first step of presence detection. In some instances, additional sensors, for example additional audio sensors, may be used in the calculation once presence is detected to increase the head count accuracy without overly increasing the computational requirements. This way, use of both computational and energy resources can be optimized without sacrificing detection accuracy.

In addition, the overall head count accuracy using only a subset of features provided accurate classification performance, with <NUM>% and <NUM>% classification accuracy for living room and single office environments, respectively. Based on the confusion matrices, performance in single office environment is accurate for <NUM>, <NUM> and <NUM> head counts. Thus, the disclosed occupancy detection system, used with the experimental parameters, provides accurate occupancy detection results with reduced cost and computational resource requirements as compared to conventional systems.

Finally, different segment lengths (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> seconds) were extracted to investigate the effects of using different time windows in the feature functional determinations. The experiments suggest that a window length of <NUM> seconds improves the accuracy in the single office scenario from <NUM>% to <NUM>% for the head counting task and from <NUM>% to <NUM>% for presence detection. In living room scenario, using a time window of <NUM> seconds improved performance from <NUM>% to <NUM>% for head counting and from <NUM>% to <NUM>% for presence detection. The results illustrate that a longer time window is advantageous for head counting as compared to other audio analytic tasks, such as automated speech recognition ("ASR"), emotion recognition, etc., which use a shorter time window in the range of tens of milliseconds to several seconds.

The system performance on three other environments using the <NUM>-dimension final features and <NUM> second segments. The system gains <NUM>%, <NUM>% and <NUM>% accuracy in open office (<NUM>-way), bedroom (<NUM>-way), and meeting room (<NUM>-way) environments. These results illustrate that the feature functionals selected from the single office and living room scenarios are also applicable for scenarios outside of the training set. As a result, the disclosed occupancy detection system is accurate for a variety of scenarios using the selected feature functionals and classifier model from the experimental results.

Claim 1:
A method of detecting occupancy in an area comprising:
Obtaining (<NUM>), with a processor (<NUM>), an audio sample from an audio sensor (<NUM>);
determining, with the processor (<NUM>), feature functional values of a set of selected feature functionals of statistical operations from the audio sample, the determining of the feature functional values comprising:
extracting (<NUM>) audio low-level descriptive, LLD, features from the audio sample using a frame-level sliding window with no overlap;
partitioning (<NUM>) the extracted audio LLD features into segment features determined by feature frames aggregated over segments with a fixed length in time and a shift of one frame, wherein the segments overlap one another such that each segment is calculated shifted by one frame; and
determining (<NUM>) the feature functional values of the determined segment features;
selecting (<NUM>) audio LLD features that contain information most relevant of occupancy of the area from the determined feature functional values, based on a correlation or independence of the selected feature functionals of the statistical operations; and
determining, with the processor (<NUM>), occupancy in the area using a classifier based on the determined feature functional values of a set of selected audio LLD features.