情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

Apnea and Sleeping-state Recognition by
Combination Use of Open-air/Contact Microphones
Abidah Alﬁ Maritsa1,a)Ayumi Ohnishi1,b)Tsutomu Terada1,c)Masahiko Tsukamoto1,d)Abstract: The increasing importance of sleeping quality and the awareness of sleep related disorders have
led to emerging research in the ﬁeld of wearable sensor with the purpose to make the sleep sensing become
more comfortable and accessible. Among various methods, the use of audio sensor is known to be one of
the most direct approach. However, further research using audio sensor to detect sleeping-state and apnea
should be done to explore various new contexts and possibilities. Thus, this study proposes a wearable
system consisting two microphones in the form of open-air microphone and contact microphone to improve
the number of recognized contexts and this system is able to detect breathing, heartbeat, swallowing, body
movement, and oral sound for the further use of sleeping-state and apnea severity detection. Audio data
combination methods of Aggregation Methods and Stacking Methods were evaluated to improve the accuracy of the context detection. The Stacking Method with Support Vector Machine Polynomial Kernel asboth ﬁrst and second level classiﬁcation resulted the best performance of 85.1% accuracy and 18% average
improvement.

1. Introduction

suming. Thus, the urge for a wearable detection system
in sleep sensing especially sleep disorder diagnosis has en-

As one-third of human life, sleep is having a major rolecouraged studies in this ﬁeld. Imtiaz et al. (2021) in

to maintain health and function as human[1]. Lacking

their review study of Sensing Technologies for Wearable

in quality of sleep correlated with several body dysfunc-

Sleep Staging presented the information that the number

tion and in long term it is also causing serious physicalof publications in wearable sleep sensing has been increas-

and mental disorders. Several complications associateding since 2014[3].

by this situation are cardiac and cerebrovascular prob-

Among overall sleep disorders and problems, breathing-lems[2], traﬃc accidents, mood swings, and depression.

related sleep disorder is one of the most concerning prob-

Data also showed that this lack of sleep quality relatedlems[4]. Sleep apnea is a major breathing-related sleep

with ﬁnancial also social impact due to deﬁcient attention

disorder indicated by pauses of breathing during sleep[1]and premature judgements in many vital areas[3]which aﬀects 22% of male population and 17% of female

Research in sleep sensing ﬁeld is emerging due to the

population[6]. However, this number might be larger as

increasing prevalence and inﬂuence of sleep disorders in

the data shown that 80% of apnea patients are remain un-

daily life[4]. These research are mainly focusing in ﬁnd-diagnosed[7] due to the low awareness of the disease anding alternatives of diagnosis modalities in sleep disorderthe complexity of the detection using PSG in hospitals[8].

and sleep quality detection, since currently polysomnog-

Apnea events during sleep are commonly detected byraphy (PSG) is still being the standardized procedure as

analyzing several bio-signals such as respiratory, heart-detection in hospital[5]. This procedure is known to bebeat, movement, snoring, oxygen saturation, pharyngeal

complex, less comfortable, and expensive also time con-movement, and more. Several approaches used by wear-

1

able devices are based on few-channel electrocardiogram

a)
b)
c)
d)

Kobe University
abidah-alﬁ@stu.kobe-u.ac.jp
ohnishi@eedept.kobe-u.ac.jp
tsutomu@eedept.kobe-u.ac.jp
tuka@kobe-u.ac.jp© 2022 Information Processing Society of Japan(ECG), accelerometer, electromyogram (EMG), electrooculogram (EOG), electroencephalogram (EEG), oronasal
airﬂow, audio sensors, and plethysmograph[3], [8]. How-87

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

ever, detecting breathing pattern using airﬂow is usuallyaudio sensors in the form of open-air and contact mi-considered as the most straightforward approach for de-crophones to detect the contexts.

tecting breathing disorders including sleep apnea[8].
Detecting breathing pattern using airﬂow sensor might
be the most straightforward approach but at the sametime it can be considered as not a comfortable and wearable solution as its placement can block or disturb airways.
Considering this reason, we saw the use of audio sensors
as a more wearable approach with similar purpose.

• This paper compares three diﬀerent methods to combine data of the two audio sensors to improve theaccuracy of the context recognition.

2. Related Research
2.1 Sleep Quality and Sleep-Related Disorders
Recently, concern in maintaining sleep quality and dis-

Wearable devices to detect apnea and sleeping state us-

eases associated with sleep disorders are growing. This is

ing popular modalities such as accelerometer, ECG, EEG,

due to the increasing awareness of the indication that theand plethysmograph can only detect limited type of con-lack of sleep quality also sleeps disorders can lead to grow-texts (2 or 3) simultaneously[3]. The popular examples

ing health burden in modern societies[12]. Lack of quality

are the use of accelerometer to only detect movement orof sleep causes serious physical conditions such as car-

ECG to detect heartbeat. While actually there are vari-diovascular disease, cognitive impairment, and metabolicous contexts that can be used to detect apnea and sleeping

dysfunction[1], [12]. Sleep quality also proven to be highly

state[3], [8]. Audio sensors are known for the ability to ob-

associated with serious neurological disorder and mental

tain a lot of signal information. In the ﬁeld of bio-signals

conditions. Disturbance in sleep quality can lead to se-

and bio-contexts recognition, the use of a common open-

vere mood swings and depressions. This condition also

air microphone is known for it’s capability to detect loudrecently associated with ﬁnancial and economic loss due

human sounds such as breathing[8], body movement, andto the high cost of medications, decreased productivity,

oral sounds such as snoring[3]. While in order to detectroad accidents, and premature decisions which caused byintrinsic body sounds such as pharyngeal movement or

the lack quality of sleep.

swallowing and heartbeat pattern, a modiﬁed approach of

Sleep disorders are conditions that caused sleep im-

contact microphone can be seen as an alternative. Thus,

pairments and decreasing quality of sleep. Several major

combining more than one type of audio sensors can ex-

sleep disorders are insomnia, circadian rhythm disorders,pand the number of recognized contexts.

breathing related sleep disorder, hypersomnia/narcolepsy,

Among 131 studies reviewed by Imtiaz et al. in 2021,

parasomnia, and restless legs syndrome/periodic limb

only three of them mentioned the use of audio sensor.

movement[1].

Two of the mentioned studies used a single microphone

the prevalence of sleep-related breathing disorders has

while the other one used a microphone and accelerometer

raised the most attention[4]. And among the sleep-related

to detect respiratory and body movement. However, none

breathing disorders, Apnea or Sleep Apnea Syndrome isof the study mentioned in the paper used the combination

introduced as the most common sleep-related breathinguse of multiple audio sensors[3].

disorder with an increasing prevalence over the last two

Thus, this research proposes a wearable device for de-Compared to the other sleep disorders,

decades[4].

tecting human contexts related with sleep-related disorders especially sleep apnea using the combination of two
audio sensors in the form of contact microphone and open

2.2 Apnea and the Detection Approaches
Apnea or Sleep Apnea Syndrome is a condition indi-

air microphone. The resulting system is aimed not to only

cated by disturbed breathing during sleep[13].

as alternative for sleep disorder detection but to monitor

events are usually followed by arousals and consciousness

Apnea

and improve human sleep quality in general.

also ﬂuctuating heartbeat pattern and often associated

The contributions of this study are as follows:with loud and frequent snoring during sleep[2]. One apnea• This study proposed novel and multiple contexts thenevent is described as a stop of breathing for more than 10s

outputted the number of each context, thus this value

period and the severity of apnea is assessed based on thecan be used not only to detect apnea or other sleep-

Apnea-Hypopnea Index (AHI) which categorized as mild

ing disorders but it can also be used in daily life to(5-15 apnea events/hour), moderate (15-30 events/hour),monitor sleep quality.

and severe (>30 events per/hour)[14].

• Device presented in this research uses two diﬀerent© 2022 Information Processing Society of Japan

Along with the other sleep disorders, sleep apnea is

88

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

also diagnosed in a certiﬁed sleep center or hospital withoxygen saturation[8], also swallowing or pharyngeal activ-

polysomnography (PSG) as the standardized procedure.

ity[16], [17] are known to be related and useful to detect

Polysomnography consists nine sensor leads; three elec-

apnea and potentially sleeping state or other sleep related

troencephalography (EEG) leads, two electrooculography

problems.

(EOG) leads, and three electromyography (EMG) leads,
and a single electrocardiography (ECG) lead along withnasal pressure sensor, thermistor, and two belts with respiratory inductive plethysmography to record the respiration activity[4].

2.3 Multiple Sensors and Audio Sensor for Sleep
Sensing and Apnea Detection
Among modalities and context, the use of EEG, accelerometer, pulse oximetry and ECG are mentioned toThis procedure is known to be very uncomfortable, ex-

be the most used sensors in wearable devices to detect

pensive, and non-accessible. Thus, various studies areapnea and sleeping state[3], [8]. However, the mentioned

conducted to ﬁnd alternatives for simpler recognition of

sensors can only detect limited number of 2-3 contexts,

bio-contexts related to the disease detection adapting thewhile as mentioned before that there are various type ofmentioned sensors in a standard polysomnography pro-signals can be used to detect the problems. Mentioned

cedure. Nasal pressure sensor, thermistor, and belts res-approaches also did not address the most direct context

piratory inductive plethysmography are used for detect-of breathing or respiratory pattern. In order to improve

ing respiratory events and patterns to estimate the apnea

the number of detected contexts, a multi-sensors approachseverity using the mentioned AHI score[15]. This method

was investigated. Some wearable devices using this ap-of detecting breathing is known to be the most direct

proach was mentioned by Imtiaz et al. and Mendonça et

approach since apnea is a breathing related sleep disor-

al. in their study such the use of audio sensor along withders[8], however the use of mentioned sensors might beaccelerometer[3], [8], [18]. Study by Kalkbrenner et. al in

the most direct, but it is often not the most comfortable2019 used audio sensor and accelerometer for automatedto use during sleeping.

sleep stage classiﬁcation resulting in 86.9% accuracy for

The next modality and also one of the most popular tosleep/wake classiﬁcation. However, the targeted contexts

be used in wearable devices to detect apnea is the use ofof mentioned system was limited to detect cardiorespira-pulse oximetry to measure O2 saturation[8] since it may

tory pattern and body movement[18].

fall dramatically during an apneic event due to the stop of

The use of audio sensor is known as a wearable alterna-breathing. The stop of breathing and apnea event causes

tive for in sleep sensing and apnea detection device. Au-arousal as mentioned previously, thus EEG, EOG, EMG,

dio data is also well-known to contain extensive amountalso accelerometer are used for detecting brain activityof data and information. The use of common or open-air

and sleep stages[3], [15].

Apnea events also indicatedmicrophone is also known to be able to detect respiratory

by the changes of heartbeat pattern due to respiratory

pattern[8], while the use of contact microphone can detectchanges. Another well know indication of sleep apnea is

intrinsic biological sounds. Conﬁgurations and combina-

snoring, thus audio sensors are also used to detect snoringtions of microphones can obtain various contexts which

and it can also used to detect breathing pattern[4], [8].

are important in sleep sensing such as breathing pattern,

Recent research by Yagi et al. in 2014 and Bhutada etsnoring and other voice such as speaking, movement, swal-

al. in 2020 mentioned that swallowing or pharyngeal im-

lowing, and others[4]. Thus, the use of multiple audio sen-pairments presence in apnea patients[16], [17]. Yagi etsors as mentioned before have the possibility to improveal. concluded that swallowing frequency during sleep in-

the number of detected contexts.

creases with the increasing of apnea severity [16] while

In the construction of audio based wearable system

Bhutada et al. mentioned in their review study that 65%

for sleep sensing and apnea detection, several parame-(n =11) of the paper they reviewed revealed the presenceters should be considered such as the most suitable place-of pharyngeal swallowing impairments in patients with ap-

ment and attachment of the device to the body[19], [20],nea[17]. However, we have not found any study on wear-[22], window segmenting sizes, feature values to be ex-able devices using this novel swallowing context to detect

tracted[21], [22], and also the most eﬀective machine learn-

apnea or sleeping state [3], [4], [8].

ing algorithm as detection modality[8].

Thus, contexts such as respiratory or breathing pattern,Placement and attachment of sensor to the body is es-heartbeat, body movements, snoring or speaking sounds,

sential especially in the use of audio sensors since they© 2022 Information Processing Society of Japan89

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

Positionsing, and lying[24] and the used of these approaches for thearound neck such as near carotid artery and suprasternal

combination of audio sensors in sleep sensing and apnea

notch[22] are considered the suitable position for wear-

detection hasn t been evaluated yet[3], [8]. On the otherable devices to detect bio-signals such as tracheal sounds,

hand, research by Valipour et al. in 2017 used two audio

breathing, snoring[23], swallowing sound, and even heart-

sensors to detect vital signs of heartbeat and respiratory

beat[19], [22].

with the purpose of general health monitoring, however

are directly aﬀect the quality of the data.

Once the signals of the body sounds are obtained, vari-this research did not do any combination approach nor

ous feature extraction methods can be used as preprocess-speciﬁed their purpose with more number of contexts[25].

ing before inputting the data to any artiﬁcial intelligence

Thus, this study would like to evaluate the use of mul-algorithm. Time domain, statistical features, and also fre-tiple audio sensors as wearable device in recognizing con-quency domain features often used as features[8]. Speechtexts related with apnea and sleeping monitor. This re-recognition based approach also considered in several au-

search also inspected the use of several methods to com-

dio context recognition in the ﬁeld of sleep sensing andbine data from the sensors and their contribution in im-apnea detection. Thus, Mel s frequency cepstral coeﬃ-

proving the detection accuracy.

cient (MFCC) is also preferred to detect oral sounds suchas snoring and speaking during sleep and also to detect
breathing events with the accuracy of 81%[8], [9].

3. Proposed Method
This research constructed and evaluated a wearable de-

Various research compares sleep sensing and apnea de-vice to detect human contexts related with apnea detec-

tection using multiple machine learning algorithms. How-tion and sleep monitoring using multiple audio sensors.

ever, the use of simple algorithms such as support vec-

This research compared several parameters to build thetor machines and random forest achieved the best out-

most suitable design of the device and models for the sys-

put[8] with the accuracy of 75.76%[10] and 86.3%[11] re-tem. And lastly, this study also adopted and compared

spectively. Both are also popular to be implemented inseveral fusion methods to combine data from the sensors

wearable devices[8].

and evaluates the eﬀect towards the accuracy of the system.

2.4 Combination of Multiple Sensors
The use of multiple audio sensors in wearable device todetect human context can capture a broad range of ac-

3.1 Preliminary Experiment and Prototype Construction

tivities and even small gestures and very sensitive signals.

In the initial stage of the research, this study carried out

The combination of the sensors using the correct method

a preliminary experiment to examine the most suitable

can increase the accuracy and improve the results of thepositions to attach the device in terms of wear-ability andsystem[24].

the quality of acquired data. Three positions were com-

One of the common approach of the combination of mul-pared: a) position A=neck under the jaw area (around thetiple sensors in the ﬁeld of wearable devices is Aggrega-

carotid artery), b) position B= at the lower part of neck,tion Method. Aggregation method combines extracted

between the clavicular bone (suprasternal notch area), andfeatures from sensors data to construct the training andc) position C = the left chest area.

testing dataset for the classiﬁcation model[24]. Though

Audio data from the mentioned positions were collected

this method is the most common approach, Garcia-Ceja

using two diﬀerent types of microphones: an open-air mi-(2019) in their work mentioned that this approach may

crophone and a contact microphone. Both of the micro-

also not optimal due to the diﬀerent statistical proper-

phones used in experiment were commercial KY-038 mi-ties of each type of sensor. The same research proposed

crophones, however one of the microphone is modiﬁed andwhat the author mentioned as multi-view stacking, thatconnected to a chest piece of stethoscope to be functioning

can be simply explained as a multi-level classiﬁcation.

as the contact microphone.

The method trained ﬁrst-level learner for each sensor andA ﬁve minutes

length recording data were obtained

then the outputs are trained using stacked generalization.

from one subject. The data was separated into 4:1 train-

However, the mentioned research used accelerometer anding and testing data. Then the data was preprocessedsound sensor in the ﬁeld of Human Activity Recognition

using 3s segmenting with 0.25 step length with the sam-(HAR) to detect activities such as standing, sitting, walk-pling frequency of 44.1 kHz. Features extracted in this© 2022 Information Processing Society of Japan90

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

図 2: Device Attachment

図 3: Experiment Setting

mentary audio and video data recorded using professional
microphone and video camera also collected to construct
図 1: Proposed Device

ground-truth label.
The experiment for each subject with the environment
as shown in Figure 3 was divided into three parts which

preliminary experiment was 10 FFT peaks, mean, vari-

were: a) Minutes 0-8: participants were asked to perform

ance, range, and energy value. Then a simple evalua-natural breathing and periodic swallowing, b) Minutes 8-tion in Weka and MATLAB using SVM was conducted.

9: participants were asked to perform body movements

During the preliminary experiment, three initial contexts

such as rolling to the left and right, c) Minutes 9-10: par-were detected such as heart-beat, breathing, and swallow-

ticipants were asked to perform oral sounds to mimic snor-

ing context. For the heart-beat detection, the positions

ing and sleeping sound.

A, B, and C obtained the accuracy of 95.8%, 96.5%, and95.9% respectively. While both breathing and swallowing

3.3 Data Analysis

context accuracy detection were 65.4%, 75.9%, and 36.9%

Data acquired from the experiment was then processed

respectively. Based on the mentioned result, Position B

and several aspects were compared to obtain the most(suprasternal notch) was chosen as the most suitable po-suitable parameters for the system such as window sizes,

sition to attach the device.

feature values, and machine learning algorithms.

A wearable device design was constructed as shown in

Initially, three window sizes were examined. The 10

Figure 1 following result of the preliminary experiment.

minutes

3D printed case was designed to maintain both of the mi-segmented using three diﬀerent length of window sizes ofcrophones compact. The case was printed using acryloni-

1-,2-,3-s lengths with a step size of 0.25. Then, two diﬀer-

trile butadiene styrene (ABS) material and the printed

ent feature values were compared which were statistical-

case was then polished and smoothed so that it will befrequency domain features and MFCC features. The ﬁrst

more comfortable for the user to wear. The device is con-type of feature consisted of 10 FFT peak values, 1 maxi-

nected to PC through two diﬀerent audio channels for each

mum amplitude, 1 range, 1 variance, and 1 energy value of

microphone.

each segment. While the MFCC features were 13 MFCC

data with frequency sampling of 44.1 kHz werefeatures and 1 log energy value. Both of types of the
3.2 Experiment
Following the construction of the wearable device, ex-features were extracted in MATLAB programming environment.

periments were conducted using the device. Data fromThe dataset was then separated into 4:1 training data

seven healthy people in their 20s and 30s were collected

and testing data to detect contexts labelled as breathing,during the experiment. The purpose of this experiment

others, swallowing, movement, and oral sound. In order

was to collect audio data for the construction of the pre-to detect mentioned contexts, the prepared training data

diction/classiﬁcation model for the system to predict bio-and testing data were evaluated in the WEKA program tocontexts related to sleep sensing and apnea.

compare two diﬀerent machine learning algorithms which

A 10 minutes

experiment was conducted for each sub-are the Support Vector Machine (SVM) and Random For-

ject. The participants were asked to lay down in order to

est algorithm. While the raw heartbeat signal was pro-mimic a sleeping position while the device was attachedcessed using peak detection to determine the number ofas shown in Figure 2 and recording their data. Supple-

heartbeat per minute also the status of heartbeat such as© 2022 Information Processing Society of Japan91

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

3.4.1 Aggregation Method
Aggregration Method combined feature values of both
Sensor A and Sensor B, open air microphone as Sensor
A and contact microphone as Sensor B. The ﬂow of thismethod can be seen in Figure 4. Following the combination of features from both sensors, the data is then preprocessed. In the preprocessing stage, this research used
K-fold Cross Validation with k =2 followed by SMOTE
ﬁltering to prevent bias during the classiﬁcation process.
The preprocessed data was then classiﬁed using SVMpolynomial kernel. The preprocessing and classiﬁcation
processed were carried out in WEKA programming environment.
3.4.2 Stacking Method
図 4: Aggregate Method Structure

Stacking Method was done by evaluating the data using
two level classiﬁcation process as seen in Figure 5. Data
from each sensor was processed separately to output theﬁrst-level classiﬁcation output. This classiﬁcation resultof each microphone was then combined and used as the
input of the second-level classiﬁcation. Before inputing
the data into the ﬁrst-level classiﬁcation, the data waspreprocessed using the same approach as the aggregation
method which are K-fold Cross Validation with k =2 and
SMOTE ﬁltering.
This data was then inputted into the ﬁrst-level of classiﬁcation. The ﬁrst-level classiﬁcation algorithm was selected based on the optimum classiﬁer obtained from theprevious uncombined data analysis. The result of the ﬁrstlevel classiﬁcation or Output A and Output B was then inputted into the second-level classiﬁcation as Input A’ andInput B’. Both preprocessing and classiﬁcation method

図 5: Stacking Method Structure

were performed in WEKA programming environment.
The proposed stacking method was then comparing
models and results obtained from two the types of algorithm as the second-level classiﬁer. Firstly, decision tree or

low, normal, or high.

rule based classiﬁcation was selected as classiﬁcation algo3.4 Combination of Multiple Microphones Datarithm. J48 Decision Tree was used in WEKA to classify

This study selected and modiﬁed two signal fusion

each training data. The second type selected algorithm

methods based on the references to construct prediction

was SVM-polynomial kernel.

based on the combination of data obtained from the two

Accuracy of the resulting data from both classiﬁers weremicrophones used in the device, contact microphone andthen compared along with the comparison of all proposed

open air microphone.

combination methods to the non-combined results.

The methods evaluated in thisstudy was classiﬁed into Aggregation Method and Stacking Methods. The second method is divided into Decision Tree-based Stacking Method and SVM-based Stack-4. Results and Discussion
4.1 Optimum Parametersing Method. This process will be limited on evaluatingDiﬀerent parameters were compared and combined tothe optimum parameters obtained in the previous dataobtain the optimum characteristics to be applied in theanalysis.

future system. Data from seven subjects which were ac-© 2022 Information Processing Society of Japan92

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

表 1: Context Detection Accuracy

4.2 Combination Methods Result
This research evaluated two methods to combine audio

Open-airContact

Mic [%]Mic [%]

data from two microphones with objective to improve theBreath71.9

59.2

accuracy of the system. Methods used in this evaluation

Others21.1

59.5

was based on the existing sensor combination approaches

3

Swallowing

39.7

71.5

as mentioned in the previous related research explana-

4

Movement

92.1

95.9

tions[24]. Data and parameter combinations used in this5

Oral Sound

92.3

86.9

Index

Context

12

evaluation was based on the previous optimum parameters analysis. The previous evaluation conﬁrmed following
best parameters for open-air microphone data (Sensor A):

quired in the experiment were analyzed. This analysis

3s windowing size, 28 MFCC Features, and SVM Polyno-

compared window sizes, feature values, classiﬁcation al-

mial Kernel. While for contact microphone data (Sensor

gorithms, and also the performance of each type of mi-B): 3s windowing size, 28 MFCC Features, and SVM Poly-

crophones to detect the desired contexts. Processes held

nomial Kernel.

in MATLAB and WEKA programming environment re-4.2.1 Aggregation Method

sulted in the higher overall performance of contact mi-

In the evaluation using aggregation method, features

crophone to detect the contexts when compared with thefrom both sensors were combined resulting in the total of

open-air microphone with the highest accuracy of 76.9%

56 features from both of the sensors data. In the dataset

for the contact microphone while the open-air microphone

preparation, even though 1s windowing performed better

resulted best accuracy of 65.4%.

for Sensor B data, the MFCC features extracted from the

The combination of data obtained with contact micro-

3s segmented Sensor B data was still combined with thephone, 1s window segmenting, extracted MFCC features,features obtained from the 3s segmented Sensor A data

and SVM Polynomial Kernel resulted in better classiﬁca-

and vice versa.

tion of overall data from all the seven subjects. While theThe 2 fold cross validation process resulted in two dif-combination of 3s window segmenting, extracted MFCC

ferent sets of training and test set. The training set offeatures, and SVM Polynomial Kernel performed better

each data then processed with SMOTE ﬁltering resulting

result to classify the open-air microphone data. Thus,in the same number of data for each classes. Training andbased on the mentioned result it was assumed that men-testing data then classiﬁed using SVM Polynomial Kerneltioned combinations performed better to classify dataclassiﬁer due to the better performance of this classiﬁer infrom each microphone.

the previous evaluation. Average of result obtained two

Each context accuracy evaluation were performed asshown in Table 1. Table 1 shows that despite the higher

diﬀerent sets of training and test set for each trial were
calculated.

overall detection accuracy of the contact microphone data,

Table 2 presented the result of the aggregation method.

the open-air microphone obtained better performance toModel A and Model B were obtained from the 2-fold crossdetect breathing and oral sound contexts with both 71.9%

validation process and the average of these two models

and 92.3% accuracy. This result indicating that the per-was calculated. This average score was then compared toformance and accuracy of the system might be improved

the previous accuracy (Prev. Acc. in Table 2) which wasby combining both microphones. Another phenomenon

obtained in the previous evaluation of uncombined datais that body movement sound noises aﬀect the input ofusing the related parameters. The diﬀerence between thethe open-air microphone, however it does not aﬀect the

average accuracy using aggregation method with the pre-contact microphone so much. Since some types of noises

vious accuracy was formulated in the delta score.

aﬀect one of the microphones, combining the microphones

From the Table 2, it is shown that the method improved

can be be an eﬀective alternative. Thus, in the next stage

10 out of the 14 trials performed in this evaluation withof the study this research evaluated diﬀerent combination

the average delta of 11.4%. Using the proposed method,

methods for the data from two microphones with aim to

4 out of 14 accuracy of the data was decreased by the

improve the context detection accuracy. The result of thisaverage of 15.3%. Average accuracy for each window size

optimum parameters analysis was used as the basis for theof 1s and 3s also experiencing improvement of 59.4% andcombination methods.

58.8% respectively, compared to the previous accuracy of© 2022 Information Processing Society of Japan93

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

表 4: Staking Method: SVM-based Classiﬁer Accuracy

表 2: Aggregation Method Accuracy
Sub.
A
B
C
D
E
F
G

Win.

Model

Model

Avg.

Prev.

Size

A

B

Models

Acc.

1

63.5

74.4

69.0

22.9

46.1

3

58.8

60.6

59.7

26.7

33.0

1

61.9

75.0

68.5

66.2

2.3

3

58.4

71.4

64.9

59.3

5.6

1

68.6

70.8

69.7

62.7

7.0

3

61.4

73.0

67.2

60.9

6.3

1

22.8

20.3

21.6

52.2

-30.7

3

32.8

63.5

48.2

58.2

-10.1

1

63.3

63.2

63.3

72.0

-8.8

3

44.7

55.7

50.2

61.9

-11.7

1

66.5

71.9

69.2

67.9

1.3

3

62.7

63.6

63.2

59.5

3.7

1

52.6

57.1

54.9

52.2

2.7

3

57.1

59.5

58.3

52.7

5.6

Delta

表 3: Staking Method: Rule-based Classiﬁer Accuracy
Sub.
A
B
C
D
E
F
G

Sub.
A
B
C
D
E
F
G

Win.

Model

Model

Avg.

Prev.

Size

A

B

Models

Acc.

1

83.7

85.1

84.4

22.3

62.1

3

73.5

77.2

75.4

26.7

48.7

1

71.7

75.0

73.4

66.2

7.15

3

76.5

77.7

77.1

59.3

17.8

1

70.0

77.9

74.0

62.7

11.3

3

66.7

76.9

71.8

60.9

10.9

1

65.6

59.6

62.6

52.2

10.4

3

66.7

69.5

68.1

58.2

9.9

1

74.3

68.6

71.5

72.0

-0.6

3

54.8

66.7

60.8

61.9

-1.2

1

72.7

74.9

73.8

67.9

5.9

3

69.5

71.8

70.7

59.5

11.2

1

69.5

74.4

72.0

52.2

19.8

3

70.7

37.5

54.1

52.7

1.4

Delta

level classiﬁer resulting in two models for each trials. The

Win.

Model

Model

Avg.

Prev

Size

A

B

Models

Acc.

1

83.8

83.3

83.6

22.9

60.7

3

75.1

74.7

74.9

26.7

48.2

1

70.2

60.8

65.5

66.2

-0.7

the second classiﬁer and result as presented in Table 3 was

3

79.1

73.8

76.5

59.3

17.2

obtained. The proposed rule-based stacking method im-

1

71.1

79.1

75.1

62.7

12.4

proved 12 out of 14 trials with the average improvement

3

66.9

78.5

72.7

60.9

11.8

of 18.3% while the rest of the data experienced average of

1

66.1

46.6

56.4

52.2

4.1

4.8% loss. Both improvement and loss of this method were

3

66.1

69.9

68.0

58.2

9.8

1

54.2

72.1

63.2

72.0

-8.9

3

74.7

66.2

70.5

61.9

8.6

1

73.1

68.8

71.0

67.9

3.0

experiencing improvement of 69.7% and 71% respectively

3

56.3

72.1

64.2

59.5

4.7

which were better when compared with previous proposed

1

70.6

75.5

73.1

52.2

20.9

method and the uncombined result.

3

70.7

70.1

70.4

52.7

17.7

4.2.3 Stacking Method:Delta

same data analysis process as the previous aggregation
method was performed for the result of this method.
Rule-based Classiﬁer of J48 Decision Tree was used assuperior compared with the previous aggregation method.
Average accuracy for each window size of 1s and 3s also

SVM-based Classiﬁca-

tion
56.5% and 54.2%.
4.2.2 Stacking Method:

The same process as the previous rule-based stacking
Rule-based Classiﬁca-tion
Stacking method with rule-based classiﬁer of J48 Deci-method were performed while in this proposed method,
SVM polynomial kernel was used as the second-level classiﬁer. This proposed method resulted the best accuracy

sion Tree was performed as the second level classiﬁcationcompared with the previous evaluated methods.

process. Each sensor datasets with MFCC features werehighest average accuracy of 84.4% as shown in Table 4

preprocressed with k-fold cross validation and SMOTE ﬁl-and the average improvement of 18% were obtained. Bothtering and then inputted into the SVM algorithm as thestacking methods have relatively similar average improve-ﬁrst level classiﬁcation. This process resulted in output inment of accuracy while the third method result in mini-the format of predictions (1: breathing, 2: others, 3: swal-

mum loss of -0.85% which was the lowest out of all pro-lowing, 4: movement, 5: oral sound) and the new dataset

posed methods. Average accuracy for each window size

was prepared using ﬁve previous predictions from eachof 1s and 3s also experiencing improvement of 73.1% andsensors as features. Thus, the new dataset consisted of 10

68.3% respectively. Table 2, Table 3, and Table 4 showedfeatures values: 5 features from Sensor A and 5 featuresthat data obtained from Subject A achieved the best accu-

from Sensor B. The data then inputted into the secondracy and highest improvement using the proposed meth-© 2022 Information Processing Society of Japan

94

The

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

図 6: Ground Truth and Aggregation Method Time Series

図 8: Ground Truth and SVM-based Method Time Series

Result with 1:breathing, 2:others, 3:swallowing, 4:move-

Result with 1:breathing, 2:others, 3:swallowing, 4:move-

ment, and 5:oral sound

ment, and 5:oral soundcontexts were detected. The initial data analysis resulted
in the optimum parameters of window sizes, feature values, and classiﬁcation algorithms for each microphones.
The contact microphone data resulted in better classiﬁcation of 76.9% accuracy. However, there was a possibilityof improving the system accuracy by combining both microphones data. Thus, this research proposed combination methods of aggregation method and stacking methods. The stacking methods have two types of approach
図 7: Ground Truth and Rule-based Method Time Series

which was rule-based classiﬁer of J48 Decision Tree andResult with 1:breathing, 2:others, 3:swallowing, 4:move-

SVM-based classiﬁer as the second-level classiﬁcation al-ment, and 5:oral soundgorithm. The proposed methods resulted in the improvement of the overall accuracy of 11.4-18.3% with the lastods. Data from Subject A resulted in 74.4%, 83.8%, andmethod of SVM-based stacking method marked the best

85.1% accuracy for all methods respectively with 46.6%-

accuracy of 85.1% and 62.1% best improvement.

62.1% improvement while the previous accuracy show that

Comparing this research result to the previous stud-

the speciﬁc data have the worst value. This result showsies, this research was able to detect more number of con-that the proposed method can improve low accuracy data.

texts (six contexts) compared to the mentioned studies inIn the opposite, data from the Subject E marked theboth apnea and sleeping-state detection ﬁeld [3], [8]. This

best accuracy in the previous uncombined approach, how-

study evaluated several aspects such as windowing sizes,

ever the speciﬁc data experienced loss using the proposedfeature values, and classiﬁcation algorithms for this spe-methods. Thus, it can be concluded that the combination

ciﬁc proposed system. Thus, even though previous related

approach does not support the detection improvement forstudies have evaluated such parameters with accuracy as

this speciﬁc data.

mentioned in Section 2.3, it is hard to compare the re-Figure 6, 7, and 8 compared the time series result ofsult obtained in this study with previous research sinceeach methods . From the ﬁgure, it can be seen that thethe evaluated parameters were speciﬁc to the system con-SVM-based stacking method resulted the best prediction

structed in this study. Then, this study also evaluatedfor most of the contexts.

several combination methods of both microphone to in-

5. Conclusion and Future Workvestigate whether the mentioned methods can improve theaccuracy of the contexts detection. This combination use

This research constructed a wearable device and evalu-

of multiple audio sensors are novel [3], [8] with the result

ated methods to detect contexts for sleeping-state sensingthat the approach obtained more contexts and improved

and apnea detection. Breathing, others (non-breathing

detection accuracy. A second prototype with a more com-state), heartbeat, swallowing, movement, and oral soundfortable and smaller design will be developed to improve© 2022 Information Processing Society of Japan95

情報処理学会 インタラクション 2022
IPSJ Interaction 2022

INT22010
2022/2/21

the wear-ability of the device. Sleeping environment experiments will also be held to analyze the performance ofthe system.

[13]

謝辞 This work was supported by JST CREST Grant

Number JPMJCR18A3, Japan.
[14]

参考文献
[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

M. K. Pavlova and V. Latreille: Sleep Disorders, American Journal of Medicine, Vol. 132, No. 3, pp. 292–299(Mar. 2019).
K. Zhu, M. Li, S. Akbarian, M. Hafezi, A. Yadollahi,
and B. Taati: Vision-Based Heart and Respiratory Rate
Monitoring during Sleep – a Validation Study for the
Population at Risk of Sleep Apnea, IEEE Journal ofTranslational Engineering in Health and Medicine, Vol.
7, pp. 1–8 (Oct. 2019).
S. Imtiaz: A Systematic Review of Sensing Technologies
for Wearable Sleep Staging, Sensors, Vol. 21, No.5, pp.
1–21 (Feb. 2021).
T. Penzel, C. Schöbel, and I. Fietze: New Technology toAssess Sleep Apnea: Wearables, Smartphones, and Accessories, F1000Research, Vol. 7, pp. 1–12 (Mar. 2018).
H. Nakano, M. Hayashi, E. Ohshima, N. Nishikata,and T. Shinohara: Validation of a New System of Tracheal Sound Analysis for the Diagnosis of Sleep Apneahypopnea Syndrome, Sleep, Vol. 27, No. 5, pp. 951–957(Aug. 2004).
K. A. Franklin and E. Lindberg: Obstructive Sleep Apnea is A Common Disorder in The Population―A Review on the Epidemiology of Sleep Apnea, Journal ofThoracic Disease, Vol. 7, No. 8, pp. 1311–1322 (Aug.
2015).
I. Fietze, N. Laharnar, A. Obst, R. Ewert, S. B. Felix, C.
Garcia, S. Gläser, M. Glos, C. O. Schmidt, B. Stubbe,
H. Völzke, S. Zimmermann, and T. Penzel: Prevalenceand Association Analysis of Obstructive Sleep Apnea
with Gender and Age Diﬀerences―Results of Ship-trend,
Journal of Sleep Research, Vol. 28, No. 5, pp. 1–9 (Oct.
2018).
F. Mendonça, S. S. Mostafa, A. G. Ravelo-Garcı́a, F.
Morgado-Dias, and T. Penzel: A Review of Obstructive Sleep Apnea Detection Approaches, IEEE Journalof Biomedical and Health Informatics, Vol. 23, No. 2,
pp. 825–837 (Apr. 2018).
R. F. Pozo, J. L. B. Murillo, L. H. Gomez, E. L. Gonzalo,
J. A. Ramirez, and D. T. Toledano: Assessment of Severe Apnoea through Voice Analysis, Automatic Speech,
and Speaker Recognition Techniques, EURASIP Journal on Advances in Signal Processing, Vol. 2009, No. 1,
pp. 1–11 (June 2009).
T. Praydas, B. Wongkittisuksa, and S. Tanthanuch:
Obstructive Sleep Apnea Severity Multiclass Classiﬁcation Using Analysis of Snoring Sounds, The 2nd World
Congress on Electrical Engineering and Computer Systems and Science (EECSS’16), pp. 16–20 (Aug. 2016).
T. Rosenwein, E. Dafna, A. Tarasiuk, and Y. Zigel:
Breath-by-breath Detection of Apneic Events for OSA
Severity Estimation using Non-contact Audio Recordings, 2015 37th Annual International Conference of
the IEEE Engineering in Medicine and Biology Society (EMBC), Vol. 2015, pp. 7688–7691 (Aug. 2015).
N. Darchia, N. Oniani, and I. Sakhelashvili: Relationshipbetween Sleep Disorders and Health Related Quality of© 2022 Information Processing Society of Japan

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

Life―Results from the Georgia SOMNUS Study, International Journal of Environmental Research and Public
Health, Vol. 15, No. 8, pp. 1–15 (July 2018).
A. Ramachandran and A. Karuppiah: A Survey on Recent Advances in Machine Learning based Sleep Apnea
Detection System, Healthcare, Vol. 9, No. 7, pp. 1–19
(July 2021).
A. M. Osman, S. G. Carter, J. C. Carberry, and D. J.
Eckert: Obstructive Sleep Apnea: Current Perspectives,
Nature and Science of Sleep, Vol. 10, pp. 21–34 (Jan.
2018).
M. Shokoueinejad, C. Fernandez, E. Carroll, F. Wang, J.
Levin, S. Rusk, N. Glattard, A. Mulchrone, X. Zhang, A.
Xie, M. Teodorescu, J. Dempsey, and J. Webster: Sleep
Apnea: A Review of Diagnostic Sensors, Algorithms, andTherapies, Physiological Measurement, Vol. 38, No. 9,
pp. R204–R252 (Aug. 2017).
K. Yagi, A. A. Lowe, N. T. Ayas, J. A. Fleetham, and F.
R. Almeida: Swallowing and Breathing Patterns during
Sleep in Patients with Obstructive Sleep Apnea, Sleep
and Breathing, Vol. 19, No. 1, pp. 377–384 (July 2014).
A. M. Bhutada, W. A. Broughton, and K. L. Garand:
Obstructive Sleep Apnea Syndrome (OSAS) and Swallowing Function―A Systematic Review, Sleep andBreathing, Vol. 24, No. 3, pp. 791–799 (Feb. 2020).
C. Kalkbrenner, R. Brucher, T. Kesztyüs, M. Eichenlaub, W. Rottbauer, and D. Scharnbeck: AutomatedSleep Stage Classiﬁcation Based on Tracheal Body Sound
and Actigraphy, GMS German Medical Science, Vol. 17,
No. 2, pp. 1–12 (Feb. 2019).
A. K. Sabil, M. Glos, A. Günther, C. Schöbel, C. Veauthier, I. Fietze, and T. Penzel: Comparison of Apnea Detection Using Oronasal Thermal Airﬂow Sensor, Nasal
Pressure Transducer, Respiratory Inductance Plethysmography, and Tracheal Sound Sensor, Journal of Clinical Sleep Medicine, Vol. 15, No. 2, pp. 285–292 (Feb.
2019).
S. Li, B. Lin, C. Tsai, C. Yang, and B. Lin: Design ofWearable Breathing Sound Monitoring System for Realtime Wheeze Detection, Sensors, Vol. 17, No. 1, pp. 1–15
(Jan. 2017).
C. Avci and A. Akbaş: Sleep Apnea Classiﬁcation basedon Respiration Signals by Using Ensemble Method, BioMedical Materials and Engineering, Vol. 26, No. s1, pp.
S1703–S1710 (Sep. 2015).
C. Kalkbrenner, M. Eichenlaub, and R. Brucher: Development of a New Homecare Sleep Monitor using Body
Sounds and Motion Tracking, Current Directions in
Biomedical Engineering, Vol. 1, No. 1, pp. 30–33 (Sep.
2015).
W. Shi, B. Xue, S. Guo, D. Y. T. Goh, and W. Ser: Obstructive Sleep Apnea Detection using Diﬀerence in Feature and Modiﬁed Minimum Distance Classiﬁer, 2018
40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp.
1–4 (Oct. 2018).
E. Garcia-Ceja, C. Galván-Tejada, and R. Brena: Multiview Stacking for Activity Recognition with Sound andAccelerometer Data, Information Fusion, Vol. 40, pp.
45–56 (Mar. 2018).
A. Valipour and R. Abbasi-Kesbi: A Heartbeat andRespiration Rate Sensor Based on Phonocardiogram for
Healthcare Application, 25th Iranian Conference on
Electrical Engineering (ICEE), pp. 45–48 (May 2017).

96