Patent Description:
There exist several types of systems that can detect the presence of an intruder in an indoor environment. They are mostly based on motion sensors (e.g. infrared or magnetic sensors), contact sensors mounted to windows and doors or glass break sensors - either vibration based and directly applied to each glass pane of a window or door - which can supervise several windows in a room. Image sensors and cameras are also an option.

The signal measured at a barometric pressure sensor or at a microphone sensor can be seen as a simpler (less installation effort) and cheaper alternative to the approaches mentioned above which however makes use of advance machine learning techniques and neural network architectures to boost performance over more conventional methods.

As far as the specific detection algorithm is concerned, a simple state-of-the art approach based on a combination of sensor signals (including a barometer) and thresholds is disclosed in <CIT>. Due to its simplicity, this approach has the drawback of not being able to distinguish between different types of events (such as opening or tilting of a window or window crashing) and is prone to generate a too high number of false alarms.

A more complex approach consists of applying a Hidden Markov Model to the collected pressure sensor signals. This method requires the identification of the different states and transition probabilities from one state to the other which turns out to be quite challenging if different type of opening events needs to be classified (window open/tilted, door open, etc.) and the use of a simpler probabilistic model becomes thus more attractive.

As far as sensor fusion is concerned, a simple combination of pressure and microphone sensors has been also proposed in <CIT>, where a majority rule is applied to the final output of a basic threshold-based algorithm running on signals of a pressure sensors and on signals of a microphone.

<CIT> discloses collecting air pressure signals from a pressure sensor and sound signals from a sound sensor, at a building opening. The pressure peaks occurring in the air pressure signals are sensed. The sound pulses occurring in the sound signals are sensed. The joint occurrence of a pressure peak in the air pressure signals and a sound pulse in the sound signals, which is indicative of an opening/ closing event of the building opening, is detected.

TWI645376 B discloses a security system of an indoor space. It includes a pressure sensing module that transmits a barometric pressure to a processing module. It also includes a sound module that transmits values of sound of the indoor space and frequency to the processing module. The processing module compares the received barometric pressure and sound and frequency values with preset thresholds to determine the closed state of the doors and windows.

Such an approach, despite its simplicity, does not make an efficient and judicious use of the two available streams and compromises classification performance.

Disclosed is a classification system for classifying an external impact on a window or on an access opening of an enclosed structure. The classification system comprises:.

The classification system according to the disclosure is based on a multiple-stream method, which combines data from sensors of different nature in a common network architecture to enhance each stream with the knowledge from the other stream at an early stage of the processing flow.

The term "classifying an external impact on a window or on an access opening of an enclosed structure" refers to a detection of a nature of an externally induced event having an influence on the window or on the access opening of the enclosed structure. The classes may comprise:.

The enclosed structure may be for example a building, a portion of a building, e.g. a room, a vehicle or a part of a vehicle, such as a passenger compartment. The access opening may be a door, a luggage compartment lid or the like.

The term processor refers to an electronic device configured for specific task. A processor may comprise hardware or a combination of hardware and software. Different processors may share hardware components and/or software components.

The audio signal feature processor is configured for transforming the audio signal into audio feature vectors, wherein the audio feature vector comprises values of features which are characteristic for the audio signal. Some of the features may correspond to the absolute value of the mean deviation of the respective audio signal, to the phase of the mean deviation of the respective audio signal, to the absolute value of the standard deviation of the respective audio signal or to the phase of the standard deviation of the respective audio signal. Also considered may be the maximum, the minimum, dynamic moments, the dynamic moments of the first derivative or meta features, like kurtosis, skewness or quantile, of the audio signal.

In the same way, the pressure signal feature processor is configured for transforming the pressure signal into pressure feature vectors, wherein the pressure feature vector comprises values of features which are characteristic for the pressure signal. Some of the features may correspond to the absolute value of the mean deviation of the respective pressure signal, to the phase of the mean deviation of the respective pressure signal, to the absolute value of the standard deviation of the respective pressure signal or to the phase of the standard deviation of the respective pressure signal. Also considered may be the maximum, the minimum, the dynamic moments, the dynamic moments of the first derivative or meta features, like kurtosis, skewness or quantile, of the pressure signal.

The feature combining processor combines the audio feature vector and the pressure feature vector in order to produce an audio feature and pressure feature vector, which 41w is fed to the classification processor.

In particular, the audio feature and pressure feature vector is fed to an input layer of the first machine learning algorithm and the classification output is based on an output of the first machine learning algorithm.

The term "machine learning algorithm" refers to all statistical models that computer systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. The structure of the machine learning algorithm usually is predefined. However, the parameters of the machine learning algorithm usually are learned during a preoperational learning phase and/or during the operational phase.

It has to be noted that the proposed techniques are flexible enough to be able to incorporate further sensor signals such as sensor signals from radar or environmental sensors in addition to the signals from the barometric pressure sensor and the microphone as discussed herein.

The core of the disclosure is a mechanism for the classification of signals from multiple sources based on fusing the different streams at an early stage of the processing flow. Specifically, the signals are measured at a microphone and at a pressure sensor with the final goal of recognizing an impact on a window or on an access opening of an enclosed structure. The proposed classification system comprises:.

The combination of the features of the audio signal and the features of pressure signal before the features are fed to the first machine learning algorithm provide a higher classification accuracy compared to solutions in which the features of the audio signal and the features of the pressure signal are classified independently from each other.

According to some embodiments the audio signal pre-processor comprises an audio signal normalization processor configured for normalizing the audio signal, and/or the pressure signal pre-processor comprises a pressure signal normalization processor configured for normalizing the pressure signal. Such features further improve the classification accuracy.

According to some embodiments the audio signal pre-processor comprises an audio signal filter configured for filtering the audio signal, and/or the pressure signal pre-processor comprises a pressure signal filter configured for filtering the pressure signal. Each of the filters may, for example, be implemented as a high pass filter or a noise filter. Such features further improve the classification accuracy.

According to some embodiments the audio signal pre-processor comprises an audio signal outlier removing processor configured for removing outliers from the audio signal, and/or the pressure signal pre-processor comprises a pressure signal outlier removing processor configured for removing outliers from the pressure signal. Such features further improve the classification accuracy.

According to some embodiments the audio signal pre-processor comprises an audio signal converter processor configured for providing the pre-processed audio signal as two-dimensional pre-processed audio signal time-frequency maps, and/or the pressure signal pre-processor comprises a pressure signal converter processor configured for providing the pre-processed pressure signal as two-dimensional pre-processed pressure signal time-frequency maps. Each of the signal converter processors may be implemented as a mel-spectrogram algorithm. These features simplify the subsequent determination of features and, thus, improve the classification accuracy
According to some embodiments the audio signal feature processor is configured for calculating at least a first portion of the audio feature vector by using one or more first explicit formulas, and/or the pressure signal feature processor is configured for calculating at least a first portion of the pressure feature vector by using one or more second explicit formulas. In such embodiments at least a first portion of each of the feature vectors is calculated by predefined formulas which limits the computational effort compare to embodiments which are solely using adaptive algorithms for calculating each of the feature vectors.

According to some embodiments the first machine learning algorithm is implemented as a first artificial neural network, wherein the classification processor comprises a softmax processor, which is arranged at an output side of the first artificial neural network, and which outputs the classification output.

It has been found that an artificial neural network is more suitable for the classification processor than other machine learning algorithms such as decision trees or the like. The softmax processor is capable of executing a softmax function. The softmax function is used as the final layer of the neural network-based classification processor. In particular, the combination of an artificial neural network and the softmax processor limits the computational effort and provides a high accuracy.

According to some embodiments the first artificial neural network comprises at least one first convolution processor. A convolution processor is a processor which acts as a convolutional neural network The name "convolutional neural network" indicates that the network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

According to some embodiments the first artificial neural network comprises at least one first convolution batch normalization processor. A convolution batch normalization processor is a processor capable of executing a batch normalization algorithm. Batch normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks.

The first convolution processor and the first convolution batch normalization processor may be arranged parallelly, wherein the outputs of the processors are summed up in order to provide the output of the first artificial neural network. These features limit the computational effort and provide a high accuracy.

According to some embodiments the first artificial neural network comprises at least one first rectified linear unit processor. In the context of artificial neural networks, the rectifier is an activation function defined as the positive part of its argument. It has been demonstrated to enable better training of deeper networks, compared to the other activation functions.

According to some embodiments the classification processor comprises at least one first pooling layer processor, which is arranged at the output side of the first artificial neural network. A pooling layer processor layer shapes its inputs to a 1D array, which is suitable for a subsequent softmax layer.

According to some embodiments the classification processor comprises at least one second rectified linear unit processor, which is arranged at the output side of the first artificial neural network. The configuration of the second rectified linear unit processor is similar to the configuration of the first rectified linear unit processor.

According to some embodiments the audio signal feature processor is configured for executing a second machine learning algorithm, wherein at least a second portion of the audio feature vector is provided at an output layer of the second machine learning algorithm, and/or the pressure signal feature processor is configured for executing a third machine learning algorithm, wherein at least a second portion of the pressure feature vector is provided at an output layer of the third machine learning algorithm. In such embodiments the calculation of each of the feature vectors is not solely rely on predefined algorithms, instead the calculation relies at least partially on machine learning algorithms. By these features the calculation of the feature vectors may be adapted to the specific environment of the classification system so that the classification accuracy may be improved.

According to some embodiments the second machine learning algorithm is implemented as a second artificial neural network, and/or the third machine learning algorithm is implemented as a third artificial neural network. The second artificial network and/or the third artificial network may be implemented similarly to the first artificial network.

According to some embodiments the second artificial neural network comprises at least one second convolution processor, and/or the third artificial neural network comprises at least one third convolution processor. The second convolution processor and/or the third convolution processor may be implemented similarly to the first convolution processor.

According to some embodiments the second artificial neural network comprises at least one second convolution batch normalization processor, and/or the third artificial neural network comprises at least one third convolution batch normalization processor. The second convolution batch normalization processor and/or the third convolution batch normalization processor may be implemented similarly to the first convolution batch normalization processor.

According to some embodiments the audio signal feature processor comprises at least one second pooling layer processor at an output side of the second artificial neural network, and/or the pressure signal feature processor comprises at least one third pooling layer processor at an output side of the third artificial neural network. The second pooling layer processor and/or the third pooling layer processor may be implemented similarly to the first pooling layer processor.

According to some embodiments the audio signal feature processor comprises at least one third rectified linear unit processor at an output side of the second artificial neural network, and/or the pressure signal feature processor comprises at least one fourth rectified linear unit processor at an output side of the third artificial neural network. The third rectified linear unit processor and/or the fourth rectified linear unit processor may be implemented similarly to the first rectified linear unit processor.

According to some embodiments the audio signal feature processor comprises at least one fourth convolution processor at an output side of the second artificial neural network, and/or the pressure signal feature processor comprises at least one fifth convolution processor at an output side of the third artificial neural network. The fourth convolution processor and/or the fifth convolution processor may be implemented similarly to the first convolution processor.

According to some embodiments the feature combining processor is configured for producing the audio feature and pressure feature vector by producing a concatenation or a sum or an elementwise multiplication of the audio feature vector and the pressure feature vector.

With Fa being the audio feature vector and Fp being the pressure feature vector out of the two sensor streams - learned or explicitly extracted- and assuming they have been reduced to compatible sizes in the pre-processing and feature extraction steps, we could have various combinations. Possible options for producing the audio feature and pressure feature vector are:.

According to some embodiments the feature combining processor is configured for producing the audio feature and pressure feature vector by producing a weighted concatenation or a weighted sum or a weighted elementwise multiplication of the audio feature vector and the pressure feature vector, wherein, by a weight computation processor, a weight for the audio feature vector and a weight for the pressure feature vector is determined depending on the audio feature vector and/or the pressure feature vector.

With Fa being the audio feature vector and Fp being the pressure feature vector of the two sensor streams - learned or explicitly extracted- and assuming they have been reduced to compatible sizes in the pre-processing and feature extraction steps, we could have various combinations. Possible options are:.

The weights allow taking into account that, under certain circumstances, the features of the audio signal may be more distinctive (chararcteristic) than the features of the pressure signal or vice versa. Thus, the classification accuracy may be improved by allocating a higher weight to the features of the audio signal than to the features of the pressure signal in case that the features of the audio signal are more characteristic than the features of the pressure signal and by allocating a higher weight to the features of the pressure signal than to the features of the audio signal in case that the features of the audio signal are less characteristic than the features of the pressure signal. In this case the weights are, at least partially, determined by the use of a predefined algorithm, which limits the computational effort.

According to some embodiments the feature combining processor is configured for producing the audio feature and pressure feature vector by producing a weighted concatenation or a weighted sum or a weighted elementwise multiplication of the audio feature vector and the pressure feature vector, wherein, by a weight computation processor, a weight for the audio feature vector and a weight for the pressure feature vector is determined depending on a hidden state of the first artificial neural network. In this case the weights are, at least partially, determined by the use of learned data from the first artificial network, which further improves the classification accuracy.

According to some embodiments the feature combining processor is configured for adding a first offset vector to the audio feature vector and/or a second offset vector to the pressure feature vector before producing the audio feature and pressure feature vector, wherein the first offset vector and/or the second offset vector are determined depending on the audio feature vector and/or the pressure feature vector. In this case the offsets are, at least partially, determined by the use of a predefined algorithm, which limits the computational effort. Such features improve the classification accuracy.

According to some embodiments the feature combining processor is configured for adding a first offset vector to the audio feature vector and/or a second offset vector to the pressure feature vector before producing the audio feature and pressure feature vector, wherein the first offset vector and/or the second offset vector are determined depending on a hidden state of the first artificial neural network. In this case the offsets are, at least partially, determined by the use of learned data from the first artificial network, which further improves the classification accuracy. Such features improve the classification accuracy.

Further disclosed is a method for classifying an external impact on a window or on an access opening of an enclosed structure. The method comprises the steps of:.

Further disclosed is a Computer program for implementing, when running on a computer, a method as disclosed herein.

Preferred embodiments of the invention are subsequently discussed with respect to the accompanying drawings, in which:.

Equal or equivalent elements or elements with equal or equivalent functionality are denoted in the following description by equal or equivalent reference numerals.

In the following description, a plurality of details is set forth to provide a more thorough explanation of embodiments of the present disclosure. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail in order to avoid obscuring embodiments of the present disclosure. In addition, features of the different embodiments described hereinafter may be combined with each other, unless specifically noted otherwise.

<FIG> shows a schematic view of a first embodiment of a classification system <NUM> for classifying an external impact on a window or on an access opening of an enclosed structure according to the disclosure. The classification system comprises:.

According to some embodiments the audio signal pre-processor <NUM> comprises an audio signal filter configured for filtering the audio signal AS, and/or the pressure signal pre-processor <NUM> comprises a pressure signal filter configured for filtering the pressure signal PS.

According to some embodiments the audio signal pre-processor <NUM> comprises an audio signal outlier removing processor configured for removing outliers from the audio signal AS, and/or the pressure signal pre-processor <NUM> comprises a pressure signal outlier removing processor configured for removing outliers from the pressure signal PS.

According to some embodiments the audio signal feature processor <NUM> is configured for calculating at least a first portion of the audio feature vector AFV by using one or more first explicit formulas, and/or the pressure signal feature processor <NUM> is configured for calculating at least a first portion of the pressure feature vector PFV by using one or more second explicit formulas.

According to some embodiments the feature combining processor <NUM> is configured for producing the audio feature and pressure feature vector APV by producing a concatenation or a sum or an elementwise multiplication of the audio feature vector AFV and the pressure feature vector PFV.

In a further aspect the disclosure refers to a method for classifying an external impact on a window or on an access opening of an enclosed structure. The method comprises the steps of:.

In a further aspect the disclosure refers to a computer program for implementing, when running on a computer, a method as discussed herein.

This disclosure deals with the intelligent combinations - sensor fusion - pressure signals PS of a barometric pressure sensor <NUM> and audio signals AS of a microphone <NUM> for the identification of specific types of events such as a window or house door being opened or closed or the break of a window glass. In the context of smart home and home scene analysis the identification of such events can be used for instance to detect that a burglar is entering a home or house. Similar events are also relevant for other enclosed environment like cars.

It has been observed that the pressure signal PS measured at a barometric pressure sensor <NUM> has a characteristic pattern depending on whether a window or a door in the home, where the barometric pressure sensor <NUM> is placed, is being opened, tilted, broken or similar. The pattern of the pressure signal PS remains distinctive not only in the affected room but also in the whole home as long as doors inside the home are not tightly closed. Similarly, audio signals AS also exhibit a very distinctive behavior when events like slamming, crashing or hitting a window happens near the microphone <NUM>. Nevertheless, the specific patterns recorded at the microphone <NUM> and at the barometric pressure sensor <NUM> have different characteristics that - depending on the event - can be more or less pronounced.

According to <FIG> the audio signal AS and the pressure signal PS are first separately pre-processed by the audio signal pre-processor and the <NUM> pressure signal pre-processor <NUM>. This includes, where needed, high pass or noise filtering, signal normalization, outliers' elimination, and so on but also the generation of <NUM>-dimensional time-frequency maps (e.g. a feature map).

As a second step, the pre-processed audio signal PAS is fed to the audio signal feature processor <NUM> and the pre-processed pressure signal PPS is fed to the feature pressure signal feature processor <NUM>. The signals PAS and PPS are still kept separate to enable a more appropriate and signal specific learning of the discriminative properties of the multiple stream data. The extracted audio feature vector AFV and the extracted pressure feature vector PFV could be the explicit calculation of a certain metric (such as a derivative, a phase, a mean value, etc.) or could be implicitly learned through a separate neural network.

At this point, the extracted audio feature vector AFV and the extracted pressure feature vector PFV are fused or combined together in order to calculate an audio feature and pressure feature vector APV. This operation can have different implementations, ranging from a simple feature concatenation to a more complex weighted combination with weights (and offsets) being learned with a separate attention mechanism and thus a separate back-propagation process. The choice between the different combination methods will depend on the specific structure the subsequent structure, on its convergence properties and on the nature of the data (and extracted features).

Now, the computed audio feature and pressure feature vector APV is sent to the classification processor <NUM>, which can be implemented as general machine learning classification algorithm or more conveniently as a neural network followed by a softmax layer. The specific network could be a fully combined network or an additional convolutional block depending on what features have been extracted and how they have been combined in the previous steps.

<FIG> shows a schematic view of a second embodiment of a classification system <NUM> for classifying an external impact on a window or on an access opening of an enclosed structure according to the disclosure.

According to some embodiments the audio signal pre-processor <NUM> comprises an audio signal normalization processor <NUM> configured for normalizing the audio signal AS, and/or the pressure signal pre-processor <NUM> comprises a pressure signal normalization processor <NUM> configured for normalizing the pressure signal PS.

According to some embodiments the audio signal pre-processor <NUM> comprises an audio signal converter processor <NUM> configured for providing the pre-processed audio signal PAS as two-dimensional pre-processed audio signal time-frequency maps.

According to some embodiments the first machine learning algorithm is implemented as a first artificial neural network <NUM>, wherein the classification processor <NUM> comprises a softmax processor <NUM>, which is arranged at an output side of the first artificial neural network <NUM>, and which outputs the classification output CO.

According to some embodiments the audio signal feature processor <NUM> is configured for executing a second machine learning algorithm, wherein at least a second portion of the audio feature vector AFV is provided at an output layer of the second machine learning algorithm.

According to some embodiments the second machine learning algorithm is implemented as a second artificial neural network <NUM>.

According to some embodiments the second artificial neural network <NUM> comprises at least one second convolution processor <NUM>.

According to some embodiments the second artificial neural network <NUM> comprises at least one second convolution batch normalization processor <NUM>.

According to some embodiments the audio signal feature processor <NUM> comprises at least one second pooling layer processor <NUM> at an output side of the second artificial neural network <NUM>.

The first pre-processing step may consist for both the audio signal AS and the pressure signal PS in data normalization. Afterwards, the mel scaled spectrogram may be calculated for the audio signal AS. The audio spectrogram may comprise <NUM> mel coefficients with a windows size of <NUM> and an overlap of <NUM>. Clearly, other combinations of window size, overlap and number of mels are possible.

For the pressure signal PS maximum, minimum and mean value may be extracted as metrics, as well as the dynamic moments and the dynamic moments of the first derivative. Other metrics or meta features like kurtosis, skewness, quantile shall also be considered.

As far as the audio signal is concerned, features are learned out of the pre-processed audio signal PAS in the form of an audio time-frequency representation with the second neural network <NUM>. This consists of the combination of a convolution processor <NUM> and a convolution batch normalization processor <NUM>. Here, the convolution batch normalization processor <NUM> indicates a shortcut, where the same pre-processed audio signal PAS going into the convolution processor <NUM> is used as an input for the convolution batch normalization processor <NUM>. This shall improve the convergence of the second artificial network15. A second pooling layer processor <NUM> shapes the output of the second neural network <NUM> to a 1D array.

The calculated pressure features and the trained audio features are then concatenated and fed to a feedforward neural network <NUM> followed by a softmax processor <NUM> that eventually delivers the classification probabilities.

<FIG> shows a schematic view of a third embodiment of a classification system <NUM> for classifying an external impact on a window or on an access opening of an enclosed structure according to the disclosure.

According to some embodiments the audio signal feature processor <NUM> comprises at least one third rectified linear unit processor <NUM> at an output side of the second artificial neural network <NUM>.

According to some embodiments the audio signal feature processor <NUM> comprises at least one fourth convolution processor <NUM> at an output side of the second artificial neural network <NUM>.

In <FIG> the combination of the convolution processor <NUM> and the convolution batch normalization processor <NUM> is followed by a third rectified linear unit processor <NUM> and a fourth convolution processor <NUM>. A second pooling layer processor <NUM> shapes the output of the second convolution processor <NUM> to a 1D array.

<FIG> shows a schematic view of a fourth embodiment of a classification system <NUM> for classifying an external impact on a window or on an access opening of an enclosed structure according to the disclosure.

According to some embodiments the pressure signal feature processor <NUM> is configured for executing a third machine learning algorithm, wherein at least a second portion of the pressure feature vector PFV is provided at an output layer of the third machine learning algorithm.

According to some embodiments the third machine learning algorithm is implemented as a third artificial neural network <NUM>.

According to some embodiments the third artificial neural network <NUM> comprises at least one third convolution processor.

According to some embodiments the pressure signal feature processor <NUM> comprises at least one third pooling layer processor at an output side of the third artificial neural network <NUM>.

According to some embodiments the pressure signal feature processor <NUM> comprises at least one fourth rectified linear unit processor at an output side of the third artificial neural network <NUM>.

According to some embodiments the pressure signal feature processor <NUM> comprises at least one fifth convolution processor at an output side of the third artificial neural network.

According to some embodiments the first artificial neural network <NUM> comprises at least one first convolution processor <NUM>.

According to some embodiments the first artificial neural network <NUM> comprises at least one first convolution batch normalization processor <NUM>.

According to some embodiments the first artificial neural network <NUM> comprises at least one first rectified linear unit processor <NUM>.

According to some embodiments the classification processor <NUM> comprises at least one first pooling layer processor <NUM>, which is arranged at the output side of the first artificial neural network <NUM>.

According to some embodiments the classification processor <NUM> comprises at least one second rectified linear unit processor <NUM>, which is arranged at the output side of the first artificial neural network <NUM>.

In <FIG> the pressure signal PS is normalized in order to create the preprocessed pressure signal PPS. The pressure feature vector PVF is learned from the preprocessed pressure signal PPS by a third artificial neural network <NUM>. For the audio signal AS, the same approach as in the previous embodiment is implemented.

The outputs of the audio signal feature processor <NUM> and the pressure signal feature processor <NUM> are added and then fed to the first neural network <NUM>, which consists of a second rectified linear unit processor <NUM>, a first convolution processor <NUM> and a first convolution batch normalization processor <NUM>. The sum of the output of the first convolution processor <NUM> and the first convolution batch normalization processor <NUM> are input to a second rectified linear unit processor <NUM> followed by the first pooling layer processor <NUM> and by the softmax processor <NUM> for generating the classification output CO.

<FIG> shows a schematic view of a fifth embodiment of a classification system <NUM> for classifying an external impact on a window or on an access opening of an enclosed structure to the disclosure.

According to some embodiments the pressure signal pre-processor <NUM> comprises a pressure signal converter processor <NUM> configured for providing the pre-processed pressure signal PPS as two-dimensional pre-processed pressure signal time-frequency maps.

A further possible embodiment is depicted in <FIG>. The pressure signal PS and the audio signal AS are normalized and spectrograms are calculated. The spectrogram from the pressure signal PS is calculated with FFTs, which are concatenated over a shifted window. In this embodiment, no convolution batch normalization processor is used in the audio signal feature processor <NUM>, only after the feature combining processor <NUM>. in addition, a second pooling layer processor <NUM> is needed after the fourth convolution processor <NUM> to be able to combine the audio feature vector AFV with the learned pressure feature vector PFV from the pressure signal feature processor <NUM>. After combining the learned features vectors AFV and PFV, the same network architecture as in <FIG> is adopted for the classification processor <NUM> to obtain the classification output CO.

<FIG> shows a schematic view of a sixth embodiment of a classification system <NUM> for classifying an external impact on a window or on an access opening of an enclosed structure to the disclosure.

According to some embodiments the feature combining processor <NUM> is configured for producing the audio feature and pressure feature vector APV by producing a weighted concatenation or a weighted sum or a weighted elementwise multiplication of the audio feature vector AFV and the pressure feature vector PVF, wherein, by a weight computation processor <NUM>, a weight AFW for the audio feature vector AFV and a weight PFW for the pressure feature vector PFV is determined depending on the audio feature vector AFV and/or the pressure feature vector PFV.

According to some embodiments the feature combining processor <NUM> is configured for producing the audio feature and pressure feature vector APV by producing a weighted concatenation or a weighted sum or a weighted elementwise multiplication of the audio feature vector AFV and the pressure feature vector PFV, wherein, by a weight computation processor <NUM>, a weight AFW for the audio feature vector AFW and a weight PFW for the pressure feature vector PFV is determined depending on a hidden state HS of the first artificial neural network <NUM>.

According to some embodiments the feature combining processor <NUM> is configured for adding a first offset vector to the audio feature vector AFV and/or a second offset vector to the pressure feature vector PFV before producing the audio feature and pressure feature vector APV, wherein the first offset vector and/or the second offset vector are determined depending on the audio feature vector AFV and/or the pressure feature vector PFV.

According to some embodiments the feature combining processor <NUM> is configured for adding a first offset vector to the audio feature vector AFV and/or a second offset vector to the pressure feature vector PFV before producing the audio feature and pressure feature vector APV, wherein the first offset vector and/or the second offset vector are determined depending on a hidden state HS of the first artificial neural network <NUM>.

The weights and offsets can be learned by a standard back-propagation algorithm and gradient descent implemented as a multilayer perceptron network or by making use of more complex attention mechanism which takes the features (and hidden state of the following classification network if available) as inputs as illustrated in <FIG>.

As far as the weights (and the offsets) of the feature combining processor <NUM> are concerned, they could be learned directly or they could be obtained with an attention-like mechanism where we first calculate the score ei and then take the softmax for normalization <MAT> <MAT> with H being the hidden state of the global network if available, V and b being the parameters matrix and vector to be learned and αi being the final weight associated to the pre-processed feature.

Alternatively, one could directly optimize the following: <MAT>.

It should be noted that the individual networks in (<NUM>) and (<NUM>) and their hyper-parameters are jointly trained with their respective global network architecture, so that the overall loss function is minimized.

In addition, it should be noted that H in equation (<NUM>) and (<NUM>) is optional and will be part of the optimization when a previous hidden state is available. This happens for example when a recurrent neural network is deployed in place of a convolutional one as the first neural network <NUM>.

<FIG> shows a typical pressure signal PS over time and a typical audio signal AS over time, when a glass window is being opened.

<FIG> shows a typical pressure signal PS over time and a typical audio signal AS over time, when a glass window is being closed.

<FIG> shows a typical pressure signal PS over time and a typical audio signal AS over time, when a glass window is being hit.

<FIG> shows a typical pressure signal PS over time and a typical audio signal AS over time, when a glass window is being broken.

<FIG> shows a typical pressure signal PS over time and a typical audio signal AS over time, when a glass window is being slammed.

<FIG> are showing some typical signals measured at the pressure sensor and microphone when different types of events occur. It can be seen that the signals below have a very distinctive pattern depending on whether the glass window is being opened, closed, hit, slammed or broken. Also, it can be observed that the audio signals AS and the pressure signals PS are quite different in their properties with the audio signal AS having very specific patterns for slamming, hitting and glass breaking and the pressure signal PS having more distinctive behavior when trying to separate the open/close event. Similar considerations can be done by observing the frequency spectra of the different classes.

<FIG> shows a possible implementation for the convolution processors <NUM>, <NUM>, <NUM> and <NUM> described herein. Each of the convolution processors <NUM>, <NUM>, <NUM> and <NUM> explained in this disclosure may comprise (in this order) the following subunits: a first convolution layer <NUM> comprising eight two-dimensional Kernels, a first batch normalization layer <NUM>, a first rectified linear layer <NUM>, a second convolution layer <NUM> comprising five two-dimensional Kernels, a second batch normalization layer <NUM>, a second rectified linear layer <NUM>, a third convolution layer <NUM> comprising three two-dimensional Kernels, and a third batch normalization layer <NUM>.

In Table <NUM> a comparison of single stream and multi stream approaches is itemized. It could be seen, that the simple AND combination of a pressure algorithm and an audio algorithm already gives a better performance than each algorithm itself. Nevertheless, the smart multi stream approach described in this disclosure delivers an even better result than using the majority rule in addition to saving computation resources by sharing processing blocks across the different streams. The accuracy of <NUM>% has been achieved with an embodiment of a classification system as described with reference to <FIG>. However, other embodiments disclosed herein provide similar accuracy.

Claim 1:
Classification system for classifying an external impact on a window or on an access opening of an enclosed structure, the classification system comprising:
a microphone (<NUM>) configured for producing an audio signal (AS) corresponding to a sound within the enclosed structure;
a barometric pressure sensor (<NUM>) configured for producing a pressure signal (PS) corresponding to an air pressure within the enclosed structure;
an audio signal pre-processor (<NUM>) configured for deriving a pre-processed audio signal (PAS) from the audio signal (AS);
a pressure signal pre-processor (<NUM>) configured for deriving a pre-processed pressure signal (PPS) from the pressure signal (PS);
an audio signal feature processor (<NUM>) configured for extracting audio features from the pre-processed audio signal (PAS) in order to produce an audio feature vector (AFV), wherein the audio features are characteristic for the audio signal (AS) and the audio feature vector (AFV) comprises values of the audio features;
an pressure signal feature processor (<NUM>) configured for extracting pressure features from the pre-processed pressure signal (PPS) in order to produce a pressure feature vector (PFV), wherein the pressure features are characteristic for the pressure signal and the pressure feature vector (PFV) comprises values of the pressure features;
a feature combining processor (<NUM>) configured for combining the audio feature vector (AFV) and the pressure feature vector (PVF) in order to produce an audio feature and pressure feature vector (APV); and
a classification processor (<NUM>) configured for classifying the external impact on the window or the access opening by classifying the audio feature and pressure feature vector (APV) in order to produce a classification output (CO);
wherein the classification processor (<NUM>) is configured for executing a first machine learning algorithm, wherein the audio feature and pressure feature vector (APV) is fed to an input layer of the first machine learning algorithm, and wherein the classification output (CO) is based on an output of the first machine learning algorithm.