Ego Noise Reduction and Sound Localization
Adapted to Human Ears Using Hose-shaped Rescue Robot
Narumi Mae1 , Koei Yamaoka1 , Yoshiki Mitui2 , Mitsuo Matsumoto1 , Shoji Makino1
Daichi Kitamura2 , Nobutaka Ono3 , Takeshi Yamda1 and Hiroshi Saruwatari2
1
University of Tsukuba, Japan
E-mail: mae@mmlab.cs.tsukuba.ac.jp,
yamaoka@mmlab.cs.tsukuba.ac.jp,
maki@tara.tsukuba.ac.jp,
takeshi@cs.tsukuba.ac.jp

2
The University of Tokyo, Japan
E-mail: {yoshiki mitsui, daichi kitamura,
hiroshi saruwatari}@ipc.i.u-tokyo.ac.jp,

3Tokyo Metropolitan University, Japan
E-mail: onono@tmu.ac.jp

Abstract
Rescue robots have been developed for search and rescue operations in times of large-scale disasters. Such a robot is used tosearch for survivors in disaster sites by capturing their voiceswith its microphone array. However, since the robot has many
vibration motors, ego noise is mixed with voices, and it is difficult to differentiate the ego noise from a call for help from adisaster survivor. In our previous works, an ego noise reduction
technique that combines a method of blind source separation
called independent low-rank matrix analysis, noise cancellationand postfilter called MOSIE was proposed. Moreover, we experimentally confirm that the operator can perceive the direction of a survivor’s location from the processed stereo sound.
However, the stereo sound was not suitable for people to listenbecause it was recorded by robot’s microphones. To solve thisproblem, in this study, we applied an extension of microphonespacing by virtual microphone technique. By performing ina simulated disaster site, we confirm that the operator can perceive the direction of a survivor’s location by applying a speechenhancement technique combining independent low-rank matrix analysis, noise cancellation to the observed multichannel
noisy signals and an extension of microphone spacing by virtual microphone technique.

Stereomicrophone

Vibration motor

Figure 1: Hose-shaped rescue robot.

①

③

⑦
⑤

②

⑥

④

⑧

camera30cm

Front: vibration motors: microphones①～⑧ : microphone number

1. Introduction
It is important to develop robots for search and rescue operations during large-scale disasters such as earthquakes. The
Tough Robotics Challenge is one of the research and development programs in the Impulsing Paradigm Change throughDisruptive Technologies Program (ImPACT) [1]. One of therobots developed in this program is a hose-shaped rescue robot.
This robot is long and slim and it can investigate narrow spaces
into which conventional remotely operable robots cannot enter. This robot searches for disaster survivors by capturing their
voice with its microphones, which are attached around itself.
However, there is a serious problem when recording speech using the robot. Because of the mechanism used to operate therobot, very loud ego noise is mixed in the microphones. In ourprevious works [2], an effective noise reduction technique forstereo signal was proposed. we also confirmed that an operator
can perceive the direction of a survivor’s location by applying
our previous speech enhancement technique to observed multichannel noisy signals recorded by a hose-shaped rescue robot
with a pairwise (stereo) microphone array. However, in ourprevious work, difference in spacing between microphones andthe two ears of the operator was not taken into account. In thispaper, we apply virtual microphone technique to extend the microphone spacing of the robot to match human ear spacing.

Figure 2: Structure of hose-shaped rescue robot.

2. Hose-shaped Rescue Robot and Ego Noise
2.1 Hose-Shaped Rescue Robot
Figure 1 shows an image of the hose-shaped rescue robot.
The robot basically consists of a hose as its axis with cilia tapewrapped around it and has eight microphones, seven vibration
motors, a camera, and lamps. Figure 2 shows the positions of
its microphones and vibration motors. In the robot, two microphones are attached between each vibration motor, and themicrophones are sequentially rotated by 45◦ with each edge.
In other words, the robot has four stereo microphones that are
rotated at 45◦ intervals. Furthermore, the robot moves forward
slowly as a result of the reaction between the cilia and floor
through the vibration of the cilia tape induced by the vibration
motors. Figure 3 schematically shows the principle of movement of the hose-shaped rescue robot. When the motors vibrate, state (1) changes to state (2) through the friction between- 371 -

impulse response between a source and microphone, we canapproximately represent the observed signal asx(f, τ ) = A(f )s(f, τ ).

Figure 3: Principle of movement of hose-shaped rescue robot.

(4)Here, A(f ) = (a(f, 1) · · · a(f, M )) is an M × M mixing matrix of the observed signals. Denoting W (f ) =h(w(f, 1) · · · w(f, M )) as the demixing matrix, the separated
signal y(f, τ ) is represented asy(f, τ ) = W (f )x(f, τ ),(5)

where h is the Hermitian transpose. We use ILRMA, which is amethod unifying IVA and ISNMF. ILRMA allows us to model
the statistical independence between sources and the sourcewise time-frequency structure at the same time. We explain
the formulation and algorithm derived by Kitamura et al. [3,4].
The observed signals are represented as
X(f, τ ) = x(f, τ )x(f, τ )h ,(6)

where X(f, τ ) is the correlation matrix between channels of
size M × M . The diagonal elements of X(f, τ ) represent realvalued powers detected by the microphones, and the nondiagonal elements represent the complex-valued correlations between the microphones. The separation model of MNMF that isthe cilia and floor, then state (2) changes to state (3) as a result an extension of simple NMF for multichannel signals, X̂(f, τ ),of the cilia slipping. The hose-shaped rescue robot moves by used to approximate X(f, τ ) is represented as
repeating such changes in its state.
∑∑X(f, τ ) ≈ X̂(f, τ ) =H(f, m)t(f, l, m)v(l, τ, m), (7)ml
2.2 Problem in Recording Speech
Figure 4: Flow of the proposed method.

Recording speech using the hose-shaped rescue robot has aserious problem. During the operation of the robot, very loudego noise is mixed in the input to the microphones. The main
sources of the ego noise are the driving sound of the vibration motors, the fricative sound generated between the cilia andfloor, and the noise generated by microphone vibration. In an
actual disaster site, the voice of a person seeking help may be
not sufficiently loud to capture and it may be smaller than theego noise.
3. Proposed method
The method consists of three steps (Fig. 4). In the first step,
we apply a BSS method called independent low-rank matrix
analysis (ILRMA) that is used to estimate both the speech andego noise for multichannel signal. In the second step, a noise
cancellation process is applied to the resulting speech signal
estimated by the BSS method. In the third step, we apply virtual
microphone technique to extend the microphone spacing of therobot to match human ear spacing.

where m = 1 · · · M is the index of the sound sources, H(f, m)
is an M × M spatial covariance matrix for each frequency iand source m, and H(f, m) = a(f, m)a(f, m)h is limited to arank-1 matrix. This assumption corresponds to t(f, l, m) ∈ R+
and v(l.τ, m) ∈ R+ being the elements of the basis matrix
T (m) and activation matrix V (m), respectively. This rank-1
spatial constraint leads to the following cost function:[∑ ∑|y(f, τ, m)|2∑
− 2 log | det W (i)|Q=l t(f, l, m)v(l, τ, m)
m
f,τ
] (8)
∑
∑
+
log
t(f, l, m)v(l, τ, m) ,m

namely, the estimation of H(f, m) can be transformed to theestimation of the demixing matrix W (i). This cost function isequivalent to the Itakura–Saito divergence between X(f, τ ) andX̂(f, τ ), and we can derivet(f, l, m) ←√3.1 Independent Low-Rank Matrix Analysist(f, l, m)

We assume that M sources are observed using M microphones (determined case). The sources and the observed andseparated signals in each time-frequency slot are as follows:
(1)
(2)
(3)

−2

Σj |y(f, τ, m)|2 v(l, τ, m) (Σl′ t(f, l′ , m)v(l′ , τ, m))−1
Σj v(l, τ, m) (Σl′ t(f, l′ , m)v(l′ , τ, m)), (9)v(l, τ, m) ←√v(f, l, m)s(f, τ ) = (s(f, τ, 1) · · · s(f, τ, M ))t ,x(f, τ ) = (x(f, τ, 1) · · · x(f, τ, M ))t ,y(f, τ ) = (y(f, τ, 1) · · · y(f, τ, M ))t ,lr(f, τ, m) =where f and τ are indexes of frequency and time, respectively,
and t denotes the vector transpose. All the entries of these
vectors are complex values. When the window size in shorttime Fourier transform (STFT) is sufficiently longer than the- 372 -

−2

Σi |y(f, τ, m)|2 t(f, l, m) (Σl′ t(f, l′ , m)v(l′ , τ, m))−1
Σi t(f, l, m) (Σl′ t(f, l′ , m)v(l′ , τ, m))

∑

,
(10)

t(f, l, m)v(l, τ, m),

(11)

l

Z(f, m) =1∑
1
x(f, τ )x(f, τ )h ,J j r(f, τ, m)w(f, m) ← (W (f )Z(f, m))

−1

e(m),

(12)
(13)

Primary
microphone

Speaker𝑠1 (𝑡)＋−

𝑦𝑠 𝑡 = 𝑠1 𝑡 + 𝑛𝑟 (𝑡)

●

𝒉1
𝒉2

𝑧 𝑡 = 𝑦𝑠 𝑡 − 𝑛ෞ𝑟 (𝑡)⋮⋮

𝑛1 (𝑡)

Noisesources⋮𝒉𝐾𝑦𝑛 (𝑡)𝑛2 (𝑡)𝑛𝐾 (𝑡)

Referencemicrophone⋮𝑛ෞ𝑟 (𝑡)

Figure 6: Block diagram of signal processing with virtual microphone array technique

Filter෡𝒉⋮Figure 5: Single noise canceller.
Figure 7: Arrangement of actual and virtual microphones
where em is a unit vector whose mth element is one. We cansimultaneously estimate both the sourcewise time-frequency
model r(f, τ, m) and the demixing matrix W (f ) by iterating(9)–(13) alternately. After the cost function converges, the separated signal y(f, τ ) can be obtained as (5). Note that since
the signal scale of y(f, τ ) cannot be determined, we apply a
projection-back method [6] to y(f, τ ) to determine the scale.
The demixing filter in ILRMA is time-invariant over several
seconds. To achieve time-variant noise reduction, we apply anoise canceller for the postprocessing of ILRMA to reduce theremaining time-variant ego noise components. An noise canceller usually requires a reference microphone to observe only
the noise signal. Thus, we utilize the noise estimates obtainedby ILRMA as the noise reference signals.

3.3 Virtual Microphone Technique
F In this work, we used virtual microphone technique [9] toextend the microphone spacing of the robot to match humanear spacing. In this work, we generate a virtual microphonesignal zv (f ′ ) by nonlinear interpolation or extrapolation of two
real microphone signals in frequency domain as an estimateof a signal at a virtual microphone position where there is noreal microphone (Fig. 6). In this work, we consider the virtual
microphone at the point with a distance ratio of 1 : (α − 1)
(α > 1) from the two real microphone positions (Fig. 7).
Here, we formulate the extrapolation of the phase as follows.
The phase and amplitude of the signal at microphone i are denoted by Ai (f ′ ) and ϕi (f ′ ), and are respectively given as
Ai (f ′ ) = |zi (f ′ )| ,3.2 Noise Canceller(18)′

Im (zi (f )).
(19)
ϕi (f ′ ) = ∠zi (f ′ ) = arctan
The noise canceller [7] requires a reference microphone loRe (zi (f ′ ))cated near a noise source. The recorded noise reference signal
nr (t) is utilized to reduce the noise in the observed speech sig- The phase of the virtual microphone signal is estimated as folnal s1 (t) as shown in Fig. 5. We here assume that both s1 (t) lows.
and nr (t) are simultaneously recorded. The observed signal
ϕv (f ′ ) = αϕ1 (f ′ ) + (α − 1) ϕ2 (f ′ ).
(20)
contaminated with the noise source can be represented as
Where, α is microphone interpolation parameter. Note that theobservedphase has an aliasing ambiguity given by ϕi (f ′ ) ±ys (t) = s1 (t) + nr (t).
(14)
2ni π with integer ni , therefore, we derived an unwrap phase
We consider that the noise signal nr (t) is strongly correlated from a phase angle for extrapolating phase difference between
with the reference noise signal yn (t) and that nr (t) can be rep- phases at Mic. 1 and Mic. 2. The virtual microphone signal
is obtained as follows in terms of the extrapolated phase andresented by a linear convolution model as
amplitude:nr (t) ≃ n̂r (t) = ĥ(t)t yn (t),(15)
zv (f ′ ) = Av (f ′ ) exp (jϕv (f ′ )) .
(21)twhere yn (t) = [yn (t) yn (t − 1) · · · yn (t − N + 1)] is the ref′erence microphone input from the current time t to the past N In this paper, Av (f ) is output signal of Noise canceller, and′samples and ĥ(t) = [ĥ1 (t) ĥ2 (t) · · · ĥN (t)]t is the estimated we obtained zi (f ) by an FFT of the whole observed signal.
impulse response. From (15), the speech signal s1 (t) is ex- 4. Experimental Evaluation
tracted as follows by subtracting the estimated noise ĥ(t)t yn (t)from the observation:
4.1 Conditionz(t) = x(t) − ĥ(t)t yn (t),(16)

where z(t) is the estimated speech signal. The filter ĥ(t) canbe obtained by minimization of the mean square error. In thispaper, we use the normalized least mean square (NLMS) algorithm [8] to estimate ĥ(t). From the NLMS algorithm, theupdate rule of the filter ĥ(t) is given asĥ(t + 1) = ĥ(t) + µz(t)yn (t).
||yn (t)||2(17)

In the evaluation, we used signals recorded by the hoseshaped rescue robot. We recorded signals arriving from threedifferent directions and evaluated processed sound without virtual microphone and proposed method. Figure 8 shows positionof microphones and a speech source.
The processed stereo signal that the operator hears wasrecorded by microphones 1 and 2, which are attached to thefront of the robot, to which the projection-back method is applied to adjust the scale to the observed signal at microphones
1 and 2.

- 373 -

Figure 9: Direction of arrival which the subjects answered.

under the condition of with virtual microphones might not
be localized in the azimuths of the actual sound sources.
Figure 8: Position of microphones and a speech source.

5. Conclusion
In this paper, we have proposed noise reduction method
adapted to human ears using hose-shaped rescue robot. The
experimental results showed that the proposed method has better direction recognition accuracy than conventional method.

Table 1: Experimental conditions
Sampling frequency
Window length of ILRMA
Window shift of ILRMA
Number of bases
Number of iterations
Filter length of noise canceller
Step size of NLMS (µ)

16 kHz2048 samples
STFT length/415501600 taps
0.1

References[1] “Impulsive Paradigm Change through Disruptive Technologies Program(ImPACT),” http://www.jst.go.jp/impact/program07.html.
[2] N. Mae, Y. Mitsui, S. Makino, D. Kitamura, N. Ono, T. Yamada, and H.
Saruwatari, “Sound source localization using binaural different for hoseshaped rescue robot,” Proc. APSIPA 2017.
[3] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari, “Efficient multichannel nonnegative matrix factorization exploiting rank-1 spatial model,” Proc. ICASSP, pp. 276–280, 2015.

We recorded signals arriving from left and right and evaluated each signal. First, we applied ILRMA to multichannel
noisy signals that consist of a survivor’s voice and ego noise.
We found an estimated signal that includes most of the speechcomponents, which was used as yL (t) and yR (t), by employing
the spectrograms and microphones chosen in advance as reference microphones. To adjust the scale to the observed signal at
microphones 1 and 2, we applied the projection-back method
to an estimated signal.
Next, we applied the noise canceller for postprocessing of
ILRMA to reduce the remaining time-variant ego noise. Then,
we used the other microphone as a reference microphone, forexample, we used microphone 2 as a reference microphonewhen applying the noise canceller to the estimated signal observed by microphone 1.
Next, we applied an extension of microphone spacing byvirtual microphone technique. Finally, the operator hears thestereo sound. In this experiment, we use 4 signals: 2 signalsthat applied ILRMA and noise canceller and 2 signals that processed by proposed method.
To confirm that the location of the survivor can be determined by hearing the processed sound, the operator hears thesound with a headphone and chooses either left or right. Other
experimental conditions are shown in Table 1.

[4] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari, “Determined blind source separation unifying independent vector analysis andnonnegative matrix factorization,” IEEE Trans. ASLP, vol. 24, no. 9, pp.
1626–1641, 2016.
[5] M. Ishimura, S. Makino, T. Yamada, N. Ono, and H. Saruwatari, “Noise
reduction using independent vector analysis and noise cancellation for ahose-shaped rescue robot,” Proc. IWAENC, 2016.
[6] N. Murata, S. Ikeda, and A. Ziehe, “An approach to blind source separation
based on temporal structure of speech signals,” Neurocomputing, vol. 41,
no. 1-4, pp. 1–24, 2001.
[7] B.Widrow, J. R. Glover, J. M. McCool, J. Kaunitz, C. S. Williams, R.
H. Hearn, J. R. Zeidler, Jr, E. Dong, and R. C. Goodlin, “Adaptive noise
cancelling: Principles and applications,” Proc. IEEE, vol. 63, pp. 1692–
1716, 1975.
[8] E. Hansler and G. Schmidt, “Acoustic Echo and Noise Control: A Practical
Approach”, John Wiley & Sons, New York, 2004.
[9] H. Katahira, N. Ono, S. Miyabe, T. Yamada, and S. Makino, ”Nonlinear
speech enhancement by virtual increase of channels and maximum SNR
beamformer, ” EURASIP Journal on Advances in Signal Processing, vol.
2016, no. 1, pp. 1–8, Jan. 2016.
[10] M. Nakano, H. Kameoka, J. Le Roux, Y. Kitano, N. Ono, and S.
Sagayama, Convergence-guaranteed multiplicative algorithms for nonnegative matrix factorization with β-divergence, Proc. IEEE MLSP, pp.
283-288, 2010.

4.2 Result
For each signal, directions of arrival which subjects answered are shown in Fig. 9. The subjects answered that soundimages under the condition of without virtual microphone
were localized in the azimuths of around 70-degree and 110degrees (white circles in Fig. 9), close to the center (front) due
to short inter-aural time difference. Oppositely, with virtual
microphones , the sound images are perceived to be closer tothe direction of the sound sources, in the azimuths of around
10-degree and 170-degree (black circles in Fig. 9). In this paper, since the amplitude is not extrapolated, the sound images- 374 -