Patent Description:
In the field of sleep analysis one of the elements to study are respiratory related sounds, RRSs. An RRS is a short audio fragment of a sound originating from a patient during their sleep analysis, for example a snoring sound, a sighing sound, a heavy breathing sound, or a moaning sound. Further analysis of such sounds may then be used to diagnose sleep disorders, such as sleep apnoea. It may further be desirable to count the duration of each RRS, the frequency of the RRSs, the total number of RRSs, and analyse various aspects of the RRSs.

The RRSs and related metrics may be obtained from an audio recording of the sleeping patient.

One way to obtain such an audio recording is by attaching a recording microphone on the face of a patient, as close to a patient's nose and mouth as possible. This has the advantage that external sounds and noises are mitigated by design. However, the presence of such a microphone may negatively influence the patient's sleep, and as a result the detected RRSs may not accurately reflect the natural sleep of the patient.

Alternatively, an audio recording device, for example a digital audio recording device, such as a mobile phone, or a dedicated audio recording device, may be placed further in the vicinity of the target patient. This way, the patient is not hindered by a microphone or any other device on or close to their face, resulting in a more natural sleep. However, in this case the drawback is that the RRSs of another person may be recorded onto the audio recording if the patient is not sleeping alone in the room.

It is therefore an aim of the present invention to solve or at least alleviate one or more of the above-mentioned problems. In particular, the disclosure aims at providing a method for identifying RRSs of the target patient in a relatively comfortable way without hindering the patient's natural sleep.

To this aim, according to a first aspect, a computer-implemented method for obtaining respiratory related sounds, RRSs, originating from a target patient is provided, the method comprising the steps of:.

The input audio recording covers the sleeping environment of the target patient, i.e. apart from the target patient's RRSs, it may further comprise RRSs from other persons or animals and other environment sounds. The input audio recording thus comprises a plurality of the target patient's RRSs. Those are then all or partly selected during the selecting step. In order to distinguish the RRSs originating from the target patient from other sounds, the RRS sounds are selected based on a respiratory trace, i.e. a representation of the target patient's respiration as a function of time that covers the duration of the input audio recording. As the RRSs originating from the target patient are related to the target patient's respiration, there is a relation between these RRSs and the respiration. As a result, the RRSs originating from the target patient can be distinguished from other sounds in the input audio recording.

This results in a set of sounds that is free from other sounds that could negatively influence the analysis, allowing an accurate sleep analysis to be made. Further, as other sounds are filtered out, the audio recording does not need to be performed very close to the patient's mouth or chest. This means that the microphone does not suppress RRSs from the target patient, or does not cause unwanted RRSs itself.

The respiratory trace may further be obtained by techniques that are available in the art, for example by deriving the trace from a signal obtained by a polysomnograph, an electrocardiograph, an electromyograph, or a photoplethysmogram (PPG).

One step is the identification of RRSs. According to an embodiment, this step further comprises determining respiratory related sounds and non-respiratory related sounds, and discarding the non-respiratory related sounds.

In other words, the sounds that are not related to respiration are discarded from the audio recording first, resulting in a subset of sounds that are RRSs but which do not necessarily originate solely from the target patient. Based on the respiratory trace, the RRSs originating from the target patient are then selected from this subset.

According to an embodiment, the identifying comprises determining sets of sounds; wherein the sounds of a set originate from a same source; and wherein the selecting further comprises, based on the respiratory trace, selecting RRSs from a set of sounds originating from the target patient.

In other words, sounds are first divided into sets or clusters according to their origin. At that point it is not yet known which of the sets originate from the target patient. By reference to the respiratory trace, RRSs of a certain set can then be attributed to the target patient. Optionally, the identifying and discarding of non-RRSs may be performed before or after the determining of the sets.

The clustering of sounds into the sets according to their respective sources may for example be done by a trained classifier.

According to the invention, the selecting comprises determining a firs and/or second subset of the RRSs having a respective high and/or low probability of originating from the target patient.

In other words, only those RRSs with a probability of originating from the target patient above a certain threshold are selected, e.g. a probability higher than <NUM>%. This assures a low output error. Further, selecting RRSs with a high probability will typically be easy to determine, i.e., require low computing power and/or memory capacity.

Further, according to an embodiment, the low probability of originating from the target patient is lower than <NUM>%. This second subset may then be discarded from the result.

According to the invention, the selecting further comprises training a classifier based on the first and/or second subset to select RRSs originating from the target patient; and selecting the RRSs originating from the target patient by the trained classifier.

In other words, the results obtained according to the first and/or second subset may be further refined by adding other RRSs that were not assigned to the first and/or second subsets. To accomplish this, a classifier is first trained with one or both the subsets to classify the RRSs as either belonging to the target patient or not. In other words, the first and/or second subset is used as labelled data. Then, the trained classifier is used to further classify the other RRSs resulting in a larger selection of RRSs originating from the target patient.

Optionally, the training of the classifier may only be performed when a number of undetermined RRSs is too high, i.e. there are still many identified RRSs that neither have a high probability or a low probability of originating from the target patient. In such case it may be useful to perform a more computationally intensive classification operation.

According to an embodiment, the determination of the first subset comprises determining audio timestamps associated with the RRSs from the input audio recording and respiratory timestamps associated with the RRSs from the respiratory trace; and determining the first subset based on the audio and respiratory timestamps.

In other words, the audio timestamps indicate the occurrence of the respective RRSs in the input audio recording and the respiratory timestamps indicate the occurrence of the respective respiratory cycles of the target patient. As the RRSs of the target patient are related to the patient's respiration, the selection can be performed based on these determined timestamps. To this end, a timestamp may by characterized by any detectable time feature such as for example an onset, a local maximum or a local minimum. This way, the selection operation is reduced to first identifying the time features and then performing operations on these time features.

One operation may be to determine time differences between the audio timestamps and respective respiratory timestamps. As the RRS of the target patient is related to their respiration, the time differences that are associated with the patient will be rather constant, while the time differences associated with other sources will be more randomly spread.

By then determining a histogram of the time differences, the ones having a high probability of belonging to the target patient will be relatively more present in the peak of the histogram and the ones having a low probability will be relatively more present in the tails of histogram.

According to a second aspect, a controller is disclosed comprising at least one processor and at least one memory including computer program code, the at least one memory and computer program code configured to, with the at least one processor, cause the controller to perform a method according to the first aspect.

According to a third aspect a computer program product is disclosed comprising computer-executable instructions for performing a method according to the first aspect when the program is run on a computer.

According to a fourth aspect a computer readable storage medium is disclosed comprising a computer program product according to the third aspect.

<FIG> shows different steps of a computer-implemented method <NUM> for identifying respiratory related sounds <NUM>, RRSs, originating from a target, i.e. monitored, patient from an input audio recording <NUM>. An RRS corresponds to an audible event generated through breathing during sleep. Such RRS may for example correspond to a snoring sound, a sighing sound, a heavy breathing sound, a moaning sound or a sound made during an apnoeic event. An RRS occurs within a breathing cycle, e.g. during inhaling, during exhaling or both. A snoring patient therefore produces a sequence of RRSs during a certain time interval, e.g. for a few seconds, minutes or even hours. Having a trace of RRSs originating from a monitored patient is valuable for performing sleep analysis as it can reveal or explain different types of health conditions.

The method starts with obtaining an audio track <NUM> or audio recording <NUM> from which the RRSs <NUM> originating from the patient are to be identified or selected. The audio track is recorded within audible distance from the target patient, i.e. within the patient's sleeping environment. This may for example be done by placing an audio recording device next to the patient's bed or somewhere else in the patient's bedroom. An illustrative example of such audio recording is further shown in plot <NUM> where the amplitude <NUM> of the recorded audio signal is presented as a function of time.

From this audio recording <NUM>, the different RRSs <NUM>-<NUM> are identified in step <NUM> of method <NUM>. These identified RRSs may relate to one specific type of RRS, e.g. only snoring, or to several or even all possible RRSs. By the identification of the RRSs, other sounds or noises are excluded from the further steps, e.g. sounds from outside the room. An RRS may for example be identified by indicating its starting time, its ending time, and/or its time period allowing to uniquely identify it within the audio recording <NUM>.

The identification of RRSs may for example be performed by executing one or more of the following steps:.

The identified RRSs <NUM> do not necessarily all originate from the target patient. For example, some of them may originate from another person sleeping next to the patient or within the same room. Also, some RRSs may originate from animals, such as from a dog sleeping in the same room. Therefore, in a subsequent selection step <NUM>, a subset <NUM> of the RRSs <NUM> is selected as originating from the monitored patient. To do so, a respiratory trace <NUM> from the patient is used to select the subset <NUM>. Such a respiratory trace characterizes the breathing of the patient during the period of the audio recording <NUM>. Plot <NUM> illustrates such a trace of the patient as function of time. The rising edges may then correspond to an inhalation and the falling edges to an exhalation, or the other way around. A respiratory trace may also correspond to discrete timestamps characterizing different breathing cycles. There is an observable temporal relationship between the trace <NUM> and the RRSs originating from the patient, while the other RRSs will not show such temporal relationship. Based on this the RRSs <NUM> originating from the patient are selected as output of step <NUM>.

A respiratory trace is obtained directly or derived indirectly from a measurement on the patient. For example, the trace may be derived from a signal obtained by a polysomnograph, an electrocardiograph, an electromyograph, a photoplethysmogram (PPG), or an accelerometer.

According to an embodiment, the selection <NUM> of RRSs <NUM> may be performed by the steps <NUM> as illustrated in <FIG>. First, in steps <NUM> and <NUM> timestamps <NUM> and <NUM> are identified for respectively the RRSs <NUM> and the respiratory trace <NUM>. For the RRSs <NUM>, an RRS timestamp <NUM> may characterize the beginning of an RRS, an end of an RRS or any predetermined time reference within the occurrence of an RRS. For the respiratory trace <NUM>, a respiratory timestamp <NUM> identifies a respiration cycle, for example a beginning, end or any predetermined time reference during a respiration cycle, either inhaling or exhaling. Then, in step <NUM>, the differences <NUM> between the timestamps <NUM>, <NUM> are determined, i.e. for each RRS timestamp <NUM> the time difference is determined with a nearby respiratory timestamp <NUM>, e.g. with the next or previous respiratory timestamp. As a result, a sequence of time differences <NUM> is obtained wherein each time difference is associated with a respective RRS. From these time differences <NUM>, a histogram <NUM> is constructed in a next step <NUM>. Histogram <NUM> represents the occurrences of a certain time difference or time difference interval. In such a histogram <NUM>, the time differences with a high occurrence show a strong temporal correlation between the associated RRSs and respiratory trace and, therefore, have a high probability of originating from the patient. Similarly, the time differences with a low occurrence show little temporal correlation between the associated RRSs and respiratory trace and, therefore, have a low probability of originating from the patient. Accordingly, the RRSs <NUM> having an occurrence higher than a certain first threshold are then selected as having a high probability of originating from the patient and added to the selection <NUM> of patient RRSs. Further RRSs <NUM> having an occurrence lower than a certain second threshold may then be selected as having a low probability of originating from the patient. The remaining RRSs <NUM> are then left as unassigned. The unassigned RRSs <NUM> may still be used to further extend to the set of patient RRSs <NUM> as further described in the embodiment with reference to <FIG> and <FIG>.

Another way of selecting the patient RRSs <NUM> is by calculating the coherence of one or more RRSs <NUM> with the respiratory trace <NUM>, i.e. the degree of synchronization between the audio signal of the one or more RRSs and the respiratory signal during the same time interval. In this case, one or more RRSs with a high coherence are considered as having a high probability of originating from the patient and one or more RRSs with a low coherence are considered as having a low probability of originating from the patient, thereby again obtaining similar sets <NUM>, <NUM>, <NUM> of RRSs. Similar to the method of <FIG>, the RRSs <NUM> with a high probability are then selected as originating from the patient.

The selection of RRSs from the patient by probabilities, e.g. by the steps of <FIG>, may be further extended depending on the outcome. For example, a considerable amount of RRSs <NUM> may still be unassigned, i.e. having neither a low or high probability of originating from the patient. In such a case, steps <NUM> as illustrated in <FIG> may be performed. In the first step <NUM>, an initial selection <NUM> is made by selecting the RRSs with a high and/or low probability, e.g. by performing the steps <NUM> as described with reference to <FIG>. Then, in step <NUM>, further RRSs are identified as originating from the patient based on the sets of RRSs with high and/or low probabilities, e.g. sets <NUM> and <NUM>. Based on these sets, some of the unassigned RRSs are further assigned as either originating from the patient or not. This step <NUM> can be performed in different ways. According to a first example, step <NUM> comprises the training of a classifier to classify RRSs according to whether they originate from the patient. For the training the RRSs with a high probability and/or with a low probability are used as labelled training data. The trained classifier is then used to add yet unassigned RRSs, e.g. RRSs <NUM>, to the selection <NUM>. According to a second example, an unsupervised clustering method is used to select unassigned RRSs that have a similar feature content of similar temporal coherence with RRSs from the high or low probability set. The unassigned RRSs that are clustered with the high probability set are then added to the selection <NUM>.

<FIG> and <FIG> further illustrate the steps <NUM>. <FIG> shows a first plot with the audio recording <NUM> together with the identified RRSs <NUM> as they were, for example, obtained by step <NUM> of <FIG>. <FIG> further shows a second plot with the respiratory trace <NUM>. In the respiratory trace <NUM>, the onsets of the RRSs <NUM> are indicated with circles <NUM> and represent the RRS timestamps <NUM>. In the respiratory trace <NUM>, the periodic minima of the trace are indicated by crosses <NUM> and represent the respiratory related timestamps <NUM>. The time difference <NUM> is then represented by the space between the dashed line representing the RRS timestamp and the previous or next dotted line representing the RR timestamp. The RRSs <NUM> as shown in <FIG> are all originating from the patient. Therefore, there is a strong temporal relationship between the RR and RRS timestamps <NUM>, <NUM> which can be observed by the almost constant time differences <NUM>. <FIG> then shows a histogram <NUM> of time differences derived from RRSs that only originate from the patient as illustrated in <FIG>.

Similar to <FIG> shows a first plot with the audio recording <NUM> together with the identified RRSs <NUM> as for example obtained by step <NUM> of <FIG>. <FIG> further shows a second plot with the respiratory trace <NUM>. In the respiratory trace <NUM>, the onsets of the RRSs <NUM> are indicated with circles <NUM> and represent the RRS timestamps <NUM>. In the respiratory trace <NUM>, the periodic minima of the trace are indicated by crosses <NUM> and represent the respiratory related timestamps <NUM>. The time difference <NUM> is then represented by the space between the dashed line representing the RRS timestamp and the closest dotted line representing the RR timestamp. The RRSs <NUM> as shown in <FIG> are not originating from the patient. Therefore, there is a weak temporal relationship between the RR and RRS timestamps <NUM>, <NUM> which can be observed by the highly varying time differences <NUM>. <FIG> then shows a histogram <NUM> of time differences derived from RRSs that only originate from the patient as illustrated in <FIG>.

<FIG> then shows a histogram <NUM> based on time differences from both <FIG>, i.e. a combination of histograms <NUM> and <NUM>. As such, the data of histogram <NUM> may correspond to the histogram data <NUM> of method <NUM>. As explained with reference to step <NUM> of <FIG>, a first threshold <NUM> may then be defined in order to select RRSs with a high probability <NUM> and a second threshold <NUM> may then be defined in order to select RRSs with a low probability <NUM>, <NUM>. The remaining RRSs are then left unassigned as illustrated by regions <NUM> and <NUM>.

According to an embodiment, a further clustering step may be performed in the method <NUM> as illustrated in <FIG>. This is further illustrated with reference to the method of <FIG>. In a first step <NUM> which may correspond to step <NUM>, RRSs <NUM> are identified from an input audio recording <NUM>. Then, an additional clustering step <NUM> is performed. In this step <NUM>, the RRSs are grouped in a cluster when they have a high probability of belonging to the same source.

A way of clustering <NUM> is to first determine a set of features characterizing the RRSs, for example Mel-frequency cepstral coefficients, MFCCs, the signal power within a specific frequency range, the temporal features such as the signal mean and standard deviation, features characterizing the entropy of the RRS, features characterizing the formant and pitch. Additionally, or complementary, RRSs occurring in a temporally repetitive pattern may be identified thereby obtaining different chains of RRSs. Then the RRSs are clustered into different plausible sources based on the association with the temporal chain and/or based on the similarities between the different derived features. Clustering based on features may for example be performed by clustering algorithms such as K-means clustering and Gaussian Mixture Model, GMM, clustering. Clustering based on the obtained temporal chains may for example be performed by identifying repetitive RRS patterns that have a specific time interval between occurrences. By the clustering, RRSs may still be left unassigned, i.e. not belong to a certain source by a high probability. In such case, a further supervised clustering step can be performed. A classifier is then trained to classify RRSs into clusters by using the already clustered RRSs as labelled training data. For the classifier, a support vector machine, SVM, or neural network may be used.

The so-obtained clusters of RRSs <NUM> are then used as input for the further selection step <NUM> in which clusters with a high and/or low probability of originating from the patient are identified. The cluster with high probability are then selected as output <NUM>. Step <NUM> may be performed in the same way as step <NUM> or as step <NUM> but based on clusters of RRSs instead of individual RRSs. Further, an additional step <NUM> may be performed wherein yet unassigned clusters of RRSs are added to the output <NUM> in the same way as step <NUM> but based on clusters of RRSs instead of individual RRSs.

The steps according to the above described embodiments may be performed by any suitable computing circuitry, for example a mobile phone, a tablet, a desktop computer, a laptop and a local or remote server. The steps according to the above described embodiments may be performed on the same device as the audio recording device. To this end, the audio recording may also be performed by for example a mobile phone, a tablet, a desktop computer or a laptop. The steps according to the above described embodiments may also be performed by a suitable circuitry remote from the environment of the patient. In such case, the audio recording may be provided to the circuitry over a communication network such as the Internet or a private network.

<FIG> shows a suitable computing system <NUM> comprising circuitry enabling the performance of steps according to the described embodiments. Computing system <NUM> may in general be formed as a suitable general-purpose computer and comprise a bus <NUM>, a processor <NUM>, a local memory <NUM>, one or more optional input interfaces <NUM>, one or more optional output interfaces <NUM>, a communication interface <NUM>, a storage element interface <NUM>, and one or more storage elements <NUM>. Bus <NUM> may comprise one or more conductors that permit communication among the components of the computing system <NUM>. Processor <NUM> may include any type of conventional processor or microprocessor that interprets and executes programming instructions. Local memory <NUM> may include a random-access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor <NUM> and/or a read only memory (ROM) or another type of static storage device that stores static information and instructions for use by processor <NUM>. Input interface <NUM> may comprise one or more conventional mechanisms that permit an operator or user to input information to the computing device <NUM>, such as a keyboard <NUM>, a mouse <NUM>, a pen, voice recognition and/or biometric mechanisms, a camera, etc. Output interface <NUM> may comprise one or more conventional mechanisms that output information to the operator or user, such as a display <NUM>, etc. Communication interface <NUM> may comprise any transceiver-like mechanism such as for example one or more Ethernet interfaces that enables computing system <NUM> to communicate with other devices and/or systems, for example with other computing devices <NUM>, <NUM>, <NUM>. The communication interface <NUM> of computing system <NUM> may be connected to such another computing system by means of a local area network (LAN) or a wide area network (WAN) such as for example the internet. Storage element interface <NUM> may comprise a storage interface such as for example a Serial Advanced Technology Attachment (SATA) interface or a Small Computer System Interface (SCSI) for connecting bus <NUM> to one or more storage elements <NUM>, such as one or more local disks, for example SATA disk drives, and control the reading and writing of data to and/or from these storage elements <NUM>. Although the storage element(s) <NUM> above is/are described as a local disk, in general any other suitable computer-readable media such as a removable magnetic disk, optical storage media such as a CD or DVD, -ROM disk, solid state drives, flash memory cards,. could be used.

Claim 1:
A computer-implemented method (<NUM>, <NUM>) for obtaining respiratory related sounds (<NUM>, <NUM>), RRSs, originating from a target patient, the method comprising the steps of:
- obtaining an input audio recording (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) of a sleeping environment of the target patient;
- obtaining a respiratory trace (<NUM>, <NUM>, <NUM>, <NUM>) of the target patient's respiration, wherein the respiratory trace is obtained directly or indirectly from a measurement on the patient;
- identifying (<NUM>, <NUM>, <NUM>) RRSs (<NUM>, <NUM>, <NUM>, <NUM>) in the input audio recording; and
- selecting (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), based on the respiratory trace, from the RRSs, the RRSs (<NUM>) originating from the target patient;
characterized in that the selecting comprises:
determining (<NUM>) a first and/or second subset of the RRSs (<NUM>, <NUM>) having a respective high and/or low probability of originating from the target patient;
training (<NUM>, <NUM>) a classifier based on the first and/or a second subset to select RRSs originating from the target patient; and
selecting the RRSs originating from the target patient (<NUM>) by the trained classifier.