Journal of Robotics, Networking and Artificial Life, Vol. 5, No. 2 (September 2018) 135-138___________________________________________________________________________________________________________A Training Method for the Speech Controlled Environmental Control System
Based on Candidate Word Discriminations
Taro Shibanoki*, Masaki Watanabe*, Go Nakamura†, Takaaki Chin†, Toshio Tsuji‡*

Ibaraki University, 4-12-1, Nakanarusawacho, Hitachi, 316-8511, Japan
Hyogo Rehabilitation Center, 1070 Akebonocho, Nishi-Ku, Kobe,651-2181, Japan‡Hiroshima University, 1-4-1, Kagamiyama, Higashi-Hiroshima,739-8527, Japane-mail: taro.shibanoki.ts@vc.ibaraki.ac.jp, 14t4070a@vc.ibaraki.ac.jp, g_nakamura@assistech.hwc.or.jp,
chin@assistech.hwc.or.jp, tsuji@bsys.hiroshima-u.ac.jp
http://bs.cis.ibaraki.ac.jp†Abstract
This paper proposes a concept of a training system for the speech controlled environmental control system: BioRemote based on candidate word discriminations. The proposed system can provide three-types of voice signal
training: (1) volume, (2) tempo/timing and (3) candidate word which are important for accurate speech recognition
based on false recognition results. During the training, such three kinds of features are extracted from measured voice
signals and visually and auditory fed back to the user in real time. This allows the user to train speech abilities evenif false recognition results are extracted because of slurred speech. The efficacy of the proposed system wasdemonstrated through training experiments for slurred speech conducted with healthy participants. The results
showed that the proposed system was capable for the training of speech abilities.
Keywords: speech training, environment control system (ECS), speech recognition, candidate word, learning-type
look-up table.

to fully recover verbal functioning because of individualdifferences and degrees of disability.
This paper proposed a training system for a speechcontrolled environmental control system based on
candidate word discrimination that can acquire the skillof fixed speech. After the training, the user’s intention
can be accurately discriminated, even if the user cannot
fully recover his/her verbal functioning.

1. Introduction
The number of disabled people in Japan continues toincrease annually and stands at 1.76 million. The
population of severely disabled people in particular wasaround 760,000, and such patients associated with aspeech disability reached 81,000 [1].
Against such a background, many speech-controlledenvironmental control systems (ECSs) have beendeveloped [2], [3]. However, it is difficult for patientswith dysarthria to use such systems, since the models
used in these systems considers standard adults’ speech.
Although some studies have investigated the use ofspeaker-dependent models to support the learning ofindividual users’ voices [4], and the authors’ researchgroup also proposed the voice signal-based manipulation
method for ECS based on candidate word discrimination[5], fluctuating speech makes discrimination difficult.
The speech training system can support the recovery ofas much of a user’s speech as possible, and it has been
widely discussed as motivation and long-term
experiences for users [6], [7]. However, it can be difficult

2. Speech Training System Based on Candidate
Word Discrimination
Figure 1 shows the structure of the proposed speechtraining system based on candidate word discrimination.
The proposed system provides training for patients withdysarthria to speak the same way every time, even withslurred speech. This training can be applied to control thetraining of a voice-controlled environmental control
system [5] with slurred speech.
The proposed speech training system consists of a PC
with a feedback display, audio processor, andmicrophone. During the speech training, the display

Copyright © 2018, the Authors. Published by Atlantis Press.
This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
135

Journal of Robotics, Networking and Artificial Life, Vol. 5, No. 2 (September 2018) 135-138___________________________________________________________________________________________________________

Commandgeneration…

Featureextraction

2.1. Voice signal processing

Intention
estimation

Voice SpeechControl
Infraredsignal recognition command signals
results

provides the extracted features of voice signals andcurrent status of the users’ abilities to improve theirspeech skills. The details of the system are described in
the following subsections.

Domestic
appliances

Operator

The structure of the voice signal processing is shown inFig. 1. First, the amplitude and timing information ofvoice signals are extracted, and discrimination results arethen obtained using the candidate words/phenomes
Wh/Mh and the log-likelihoods T (Wh) with the candidate
word discrimination method [5].

VolumeTiming

Candidate word(L)(L)

Microphone

M11M21Fig. 1. Structure of the proposed training system

2.1.1. Extraction of voice signal features
PC

Voice signals are recorded using a microphone andsampled at 16 [kHz]. The amplitude information v(t) ofthe measured voice signals with full-wave rectificationand low-pass filtering (cut-off frequency: 1 [Hz]) is
obtained based on the gains of the amplifier andmicrophone input levels.
Feature vector X used for speech recognition is thendefined as the low-frequency components of Melfrequency cepstrum coefficients (MFCCs) for each frame,and the output probabilities P(X|W) of a feature vector X
from word W = {w1, w2, …, wK} (wk: word, K: number ofwords) are calculated using an N-gram model andphoneme-hidden Markov model (phoneme HMM)
dividing the words W into phonemes m = {m1, m2, …,mJ} (mj: phoneme, J: number of phonemes) and matchingphoneme HMM to X. Subsequently, the top H words Wh(h = 1, 2, …, H) with the maximum log-likelihood, theirphonemes Mh, and log-likelihoods T(Wh) are extractedusing Julius [8].

Microphone(a) Volume training(b) Tempo/timingtraining(c) Candidate word
speech training

Fig. 2. Scenes from the training

to phoneme (L)Mci (i = 1, 2, …, Ic; Ic: number of learning
data for class c) of each discrimination class memorized
in the learning-type LUT. The coincidence 𝑠 , between(D)
Mu and (L)Mci is then calculated, and a class with amaximum value of 𝑟 representing the average of all𝑠 , values (the number of coincident phonemes) is thentaken as the discrimination result. To disambiguate
discrimination, the difference between log-likelihoods
T((D)Wu) and T((L)Wci) is used to determine the result when
the same values for some classes are obtained.
2.2. Speech training for candidate word
discrimination method
For speech therapy, the treatment of articulation,
prosody and pitch range, speech rate, vocal volume, orresonance is important [6]. The proposed speech trainingis therefore composed of three stages (see Fig. 2), as
explained in the following subsections.
Before the training, to make the learning data sets usedin each training, the trainee is instructed to utter C words
T times, and the maximum/minimum and averagevolumes of each word are determined. The durations of
C× T words [startDct, endDct] used in the timing/tempo
control training are also determined using Julius [8].

2.1.2. Intention estimation using candidate word
discrimination
The user’s intention is discriminated using the learningtype look-up table (LUT) [5]. The user is instructed toutter C words (corresponding to the control commands ofdomestic appliances) multiple times, and top V words Wcv
with their maximum log-likelihood and their phonemes
Mcv and log-likelihood T(Wcv) in H extracted words arecorresponded to each discrimination class (c = 1, 2, …,C; v = 1, 2, …, V; V < H) in the learning stage. In thediscrimination stage, a new set of H words are extracted,and the phenomes (D)Mu (u = 1, 2, …, U; U < H) of top U
words with their maximum log-likelihood are compared136

Journal of Robotics, Networking and Artificial Life, Vol. 5, No. 2 (September 2018) 135-138___________________________________________________________________________________________________________

2.2.1. Volume control training

3.

In this training, the trainee practices adjusting the vocal
volume level. During the training, the amplitude
information extracted from a measured voice signal ispresented on the display with the desired values(predetermined maximum/minimum and average). The
trainee controls his/her voice signal so the extracted
amplitude information follows the average value andfalls within the min./max. range. The training result isevaluated using the following equation.
𝑆100

𝑉0𝑣 t𝑣 t𝑉𝑣 t ,𝑣 t

𝑉𝑉

3.1. Method
To verify the efficacy of the proposed training system,training experiments were performed with three healthymales (subjects A–C, 22.3
1.15 [year]). In theexperiments, participants were instructed to speak with
their tongue touching the maxillary central to simulateslurred speech. The parameters used in the experiments
were set as C = 7, T = 10, H = 10, V = 10, U = 5, D = 10,
𝐿1, 𝐷= 10, 𝐷= 500, and 𝑆 =50. The otherparameters, K, J, and Ic, were adjusted based on the input
voice signal durations and learning procedure results.
Ten sessions of each training stage were performed in thetraining experiments, and discrimination experiments
were also performed before and after training to verify
the effectiveness of the proposed training method. In thediscrimination experiments, participants were asked toutter each word three times without feedback.

. 1

𝑉The closer the amplitude information is to the average,
the higher the score, and if it exceeds the min./max.
values, the score becomes zero. The average of the 𝑆
value of speech duration is output at the end of each trial.
2.2.2. Tempo/timing control training
For accurate discrimination using the candidate word
discrimination method, it is also important to control thetiming and time from the start to the end of speech. In thistraining, the trainee controls the tempo/timing of his/herspeech. During the training, the stored voice signals ofeach word are randomly shown in the display. The trainee
is instructed to regulate his/her speech duration andtiming according to the pre-specified timing shown inwaveforms. The system evaluates the ratio of extracted
speech duration to pre-specified duration in this training.

3.2. Results and discussion
Figure 3 shows examples of experimental results. From
this figure, the training scores are stable as the number ofsessions increased, and the average discrimination rates
before and after training have relatively high accuracy.
Although participants simulated slurred speech, they
could utter each word the same way from the beginningof the training.
Therefore, other training experiments were performed sothe participants could mimic other participants’ speech.
In the experiments performed, Sub. A trained using Subs.
B and C’s learning data sets. These experimental results
are shown in Fig. 4, and it is confirmed that Sub. A cannot
follow other participants’ speech during 10 training
sessions. An additional 10 training sessions were
therefore performed with words, the score of which wasbelow 40. In the latter 10 sessions, each score wasgradually increased as the number of training sessions
increased. The average discrimination rates before andafter 20 training sessions were 61.90 48.56 [%] and80.95
39.27 [%], respectively, and the significant
difference before and after training was confirmed at alevel of 1 [%]. These outcomes indicated that theproposed training system was capable of speech training.

2.2.3. Candidate word speech training
Candidate word speech training is conducted so thetrainee speaks approximately the same way every time.
The candidate phonemes for each discrimination word
and extracted trainee’s phonemes are shown in thedisplay during this training. The trainee practices tocontrol his/her speech so similar candidate phonemes areextracted. The score of this training is defined as a ratioof the number of complete/ambiguous coincidence incandidate and extracted phonemes:𝑆⎧ 100100
⎨
⎩𝑆/𝐷𝐿

𝑀 ,𝑀

𝑆/𝐷0𝐿0𝐿0𝑀 ,𝑀𝐿𝐿

Training Experiments

, (2)

𝑀 ,𝑀

where 𝐿 ∙ represents Levenshtein distance.

137

Journal of Robotics, Networking and Artificial Life, Vol. 5, No. 2 (September 2018) 135-138___________________________________________________________________________________________________________

4. Conclusion(a) Volume training(b) Tempo/Timing training(c) Candidate word speechtraining
Fig. 3. Experimental results(d) Discrimination rates

This paper proposes a speech training system for thevoice signal controlled ECS based on candidate word
discriminations. The proposed training system provides
three types of speech training that are important to speakin the same way every time. In the training experiments,
it could be confirmed that the trainees’ speech skills were
gradually improved through training using the proposedsystem.
In future work, the authors plan to perform training
experiments for patients with dysarthria and establish an
online tuning method of training levels for each stage.
Acknowledgements
This work was partially supported by JSPS/MEXT
KAKENHI Grant Numbers 17K12723 and 26330226.
References
1. Ministry of Health, Labour and Welfare, “Ministry ofHealth, Labour and Welfare Fact-Finding Investigation of
Fisically Disabled,” http://www8.cao.go.jp/shougai/
whitepaper/h25hakusho/zenbun/furoku_08.html (accessedDecember 2017).
2. Glamo Inc., iRemocon Wi-Fi, http://i-remocon.com/aboutiremoconwifi/ (accessed December 2017).
Nature Inc., Nature Remo, http://nature.global/ (accessedDecember 2017).
3. M. S. Hawley, P. Enderby, P. Green, S. Cunningham, S.
Brownsell, J. Carmichael, M. Parker, A. Hatzis, P. O. Neilland R. Palmer, A Speech-controlled Environmental
Control System for People with Severe Dysarthria,
Medical Engineering & Physics, 29 (5), 2007, pp. 586-593.
4. T. Shibanoki, G. Nakamura, K. Shima, T. Chin and T.
Tsuji, Operation Assistance for the Bio-Remote
Environmental Control System Using a Bayesian
Network-based Prediction Model, Proceedings of 37th
Annual International Conference of the IEEE Engineeringin Medicine and Biology Society, Milan, Italy, 2015, pp.
1160-1163.
5. J. Mühlhaus, H. Frieg, K. Bilda,and U. Ritterfeld, GameBased Speech Rehabilitation for People with Parkinson’s
Disease, UAHCI 2017, Part III, LNCS10279 (M. Antonaand C. Stephanidis Eds), 2017, pp. 76-85.
6. J. Tamplin, A Pilot Study into the Effect of Vocal
Exercises and Singing on Dysarthric Speech,
Neurorehabilitation, 23, 2008, pp. 207-216.
7. Large vocabulary Continuous Speech Recognition Engine,Julius, http://julius.sourceforge.jp/index.php (accessedDecember 2017).

(a) Volume training(b) Tempo/Timing training(c) Candidate word speech training(d) Discrimination rates
Fig. 4. Experimental results using the other subjects’learning data sets (Sub. A).

138