654

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 23, NO. 4, APRIL 2015

Multichannel Signal Separation Combining
Directional Clustering and Nonnegative Matrix
Factorization with Spectrogram Restoration
Daichi Kitamura, Member, IEEE, Hiroshi Saruwatari, Member, IEEE, Hirokazu Kameoka, Member, IEEE,
Yu Takahashi, Member, IEEE, Kazunobu Kondo, Member, IEEE, and Satoshi Nakamura, Senior Member, IEEE

Abstract—In this paper, to address problems in multichannel
music signal separation, we propose a new hybrid method thatcombines directional clustering and advanced nonnegative matrix
factorization (NMF). The aims of multichannel music signal separation technology is to extract a speciﬁc target signal from observed
multichannel signals that contain multiple instrumental sounds.
In previous studies, various methods using NMF have been proposed, but many problems remain including poor separation accuracy and lack of robustness. To solve these problems, we propose
a new supervised NMF (SNMF) with spectrogram restoration anda hybrid method that concatenates the proposed SNMF after directional clustering. Via the extrapolation of supervised spectral
bases, the proposed SNMF attempts both target signal separation
and reconstruction of the lost target components, which are generated by preceding directional clustering. In addition, we experimentally reveal the trade-off between separation and extrapolation
abilities and propose a new scheme for adaptive divergence, where
the optimal divergence can be automatically changed in each timeframe according to the local spatial conditions. The results of anevaluation experiment show that our proposed hybrid method outperforms the conventional music signal separation methods.
Index Terms—Multichannel signal separation, music signal
processing, nonnegative matrix factorization (NMF), spectrogram
restoration.

I. INTRODUCTIONM

USIC signal separation technologies have attracted considerable interest and been intensively studied [1], [2]in recent years. These techniques are underdetermined separation problems because almost all musical tunes are provided ina stereo format and the number of sources is at least two. As a

Manuscript received May 22, 2014; revised October 06, 2014; accepted January 20, 2015. Date of publication February 06, 2015; date of current version
March 06, 2015. The associate editor coordinating the review of this manuscript
and approving it for publication was Prof. DeLiang Wang.
D. Kitamura is with the Department of Informatics, School of Multidisciplinary Sciences, The Graduate University for Advanced Studies, Tokyo
101–8430, Japan (e-mail: d-kitamura@nii.ac.jp).
H. Saruwatari and H. Kameoka are with the Graduate School of Information
Science and Technology, The University of Tokyo, Tokyo 113–8656, Japan(e-mail: hiroshi_saruwatari@ipc.i.u-tokyo.ac.jp; kameoka@hil.t.u-tokyo.ac.
jp).
Y. Takahashi and K. Kondo are with the Research and Development 1,
Yamaha Corporation, Shizuoka 438–0192, Japan (e-mail: yu.takahashi@music.
yamaha.com; kazunobu.kondo@music.yamaha.com).
S. Nakamura is with the Graduate School of Information Science, Nara Institute of Science and Technology, Nara 630–0192, Japan (e-mail: s-nakamura@is.
naist.jp).
Digital Object Identiﬁer 10.1109/TASLP.2015.2401425

means of addressing underdetermined signal separation, in recent years, nonnegative matrix factorization (NMF) [3], which
is a type of sparse representation algorithm, has received muchattention. NMF for acoustical signals decomposes an input spectrogram into the product of a spectral basis matrix and its activation matrix. The methods of signal separation based on NMF
are roughly classiﬁed into unsupervised and supervised algorithms. The former method attempts separation without using
any training sequences, instead being subjected to various constraints, as proposed in [4]–[6]. However, these techniques have
difﬁculty in clustering the decomposed spectral bases into a
speciﬁc target sound because the entire procedure should be
carried out in a blind fashion. To solve this problem, supervised NMF (SNMF) has been proposed [7]–[9]. This method
includes a priori training, which requires some sound samples ofa target instrument, and separates the target signal using supervised bases. SNMF can extract the target signal to some extent,
particularly in the case of a small number of sources. However,for a mixture consisting of many sources, the extraction performance is markedly degraded because of the existence of instruments with similar timbre.
To apply NMF-based separation methods to multichannel
signals, multichannel NMF has been proposed as an unsupervised separation method [10], [11]. This method is a naturalextension of NMF for a stereo or multichannel signal andis a uniﬁed method that addresses the spatial and spectral
separation problems simultaneously. However, such unsupervised separation is a difﬁcult problem, even if the signal
has multichannel components, because the decompositionis underspeciﬁed. Hence, these algorithms suffer from poor
separation accuracy and lack robustness. For multichannel
signal separation, directional clustering has also been proposed
as an unsupervised method [12], [13]. This method quantizes
directional information via time-frequency binary masking.
However, there is an inherent problem that sources located in
the same direction cannot be separated using only the directional information. Furthermore, the extracted signal is likely
to be distorted because some target components may be lost by
the effect of binary masking in the directional clustering.
To cope with these problems, in this paper, we propose a new
SNMF with spectrogram restoration and a hybrid method thatconcatenates the proposed SNMF after directional clustering.
This approach can reconstruct lost target components, which
are dispersedly generated by directional clustering, from only
the observable valid components using supervised bases. Such2329-9290 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

KITAMURA et al.: MULTICHANNEL SIGNAL SEPARATION COMBINING DIRECTIONAL CLUSTERING AND NMF

reconstruction with supervised bases can be considered as basis
extrapolation. In [14], bandwidth expansion with supervised
basis extrapolation was proposed. However, this method only
attempts to predict unseen high-frequency components. In contrast, our proposed method reconstructs dispersedly lost components in parallel with source separation. Via the extrapolation ofsupervised spectral bases, SNMF with spectrogram restoration
attempts both target signal separation and reconstruction of thelost target components, which are generated by the preceding
binary masking performed in directional clustering.
Next, we provide an experimental analysis of basis extrapolation ability and reveal the mechanism of the marked shift ofthe optimal divergence in SNMF with spectrogram restorationand the trade-off between separation and extrapolation abilities.
An evaluation experiment of the separation using artiﬁcial andreal-recorded music signals shows the effectiveness of the proposed hybrid method.
Finally, on the basis of the above-mentioned ﬁndings, we
propose a new scheme for framewise divergence selection in
the proposed hybrid method to separate the target signal using
the optimal divergence. The results of an evaluation experimentshow that the proposed hybrid method with adaptive divergencecan achieve high performance under all spatial conditions, indicating the improved robustness of the proposed method.
The rest of this paper is organized as follows. In Section II,
conventional methods for single-channel and multichannel
signal separation are described. In Section III, we propose a
new SNMF with spectrogram restoration and a hybrid method
and experimentally reveal the trade-off between separationand extrapolation abilities. In Section IV, an improved method
based on adaptive divergence is presented. Following a discussion on the results of the experiments, we present ourconclusions in Section V.
II. CONVENTIONAL SIGNAL SEPARATION METHODS

655

Fig. 1. Decomposition model of simple NMF.

and. The following equation represents the cost function of NMF:
(2)
where
is an arbitrary distance function, e.g., ItakuraSaito divergence (IS-divergence), generalized Kullback-Leibler
divergence (KL-divergence), and Euclidean distance (EUC-distance). In this study, we use the following generalized divergence called -divergence [15] in the cost function:

(3)
where
and
are matrices whose entries
areand, respectively. This divergence is a family of
cost functions parameterized by a single shape parameter thattakes IS-divergence, KL-divergence, and EUC-distance as special cases (, and 2, respectively).
The multiplicative update rules for andthat minimize
the cost function based on -divergence are given by [16]A. Conventional Single-Channel Signal Separation Methods(4)1) Overview of NMF: NMF is a type of sparse representation
algorithm that decomposes a nonnegative matrix into two nonnegative matrices as

(5)

(1)
where
is an observed nonnegative matrix, which
is an amplitude spectrogram for applying NMF to the acoustic
signal;is often called the basis matrix, which includes bases (frequently-appearing spectral patterns in ) ascolumn vectors; andis often called the activation
matrix, which involves activation information of each basis of. In addition,
and are the numbers of rows and columnsof , respectively, and is the number of bases of . Fig. 1 depicts the decomposition model of NMF, where the number ofbases equals two. In this Fig., the basis matrix includes two
types of spectral patterns as the bases to represent the observed
matrix using time-varying gains in the activation matrix. In thedecomposition of NMF, a cost function is deﬁned to optimize
the variables andusing an arbitrary divergence between

wheretrices ,,, andare the nonnegative entries of ma, and , respectively. In addition,is given by(6)

We can optimize andby some iterations of these update
rules. The convergence of these update rules has been theoretically proven for all real values of [16].
2) SNMF: The signal separation using NMF is achievedby extracting only the target spectral bases. However, such
unsupervised approaches have difﬁcultly in clustering thedecomposed spectral patterns into speciﬁc target instruments.
Furthermore, each basis may be forced to include a multi-instrumental spectral pattern. To solve this problem, SNMF has
been proposed [7]–[9]. This supervised scheme consists of656

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 23, NO. 4, APRIL 2015

two processes, namely, a priori training and observed signal
separation.
In SNMF, as the supervision, a priori spectral patterns (bases)
should be trained in advance to achieve signal separation. Hereafter, we assume that we can obtain speciﬁc solo-played instrumental sounds, which is the target of the separation task. The
trained bases are constructed by NMF as
(7)
where
is the amplitude spectrogram of a speis a
ciﬁc instrumental signal used for training,
nonnegative matrix that involves bases of the target signal ascolumn vectors, andis a nonnegative matrix that
corresponds to the activation of each basis of . In addition,is the number of frequency bins,is the number of frames of
the training signal, and is the number of bases. Therefore, thebasis matrix constructed by (7) is used for the supervision of
the target instrumental spectrum.
The following equation represents the decomposition modelin the separation process with trained supervision :(8)
is the observed spectrogram,
whereis the activation matrix that corresponds to ,isthe residual spectral patterns that cannot be expressed by,andis the activation matrix that corresponds to. Moreover, is the number of frames of the observed signal
and is the number of bases of . Strictly speaking, some
papers call this method semi-supervised NMF to discriminate
between the words “semi-supervised” (only the target soundis trained) and “fully supervised” (the target and interference
sounds are trained). However, we simply describe this method
as “supervised” in this paper because we do not intend to compare semi-supervised and fully supervised cases, as reported inother papers. In SNMF, the matrices , , and are optimized
under the condition that is known in advance. Hence,
ideally represents the target instrumental component andrepresents other interfering components after the decomposition.
The cost function for (8) is deﬁned as(9)
Also, the update rules for (9) are given by
(10)

(11)

(12)
,
,
where
of the matrices ,,, and, , , andare the nonnegative entries, respectively, and(13)

Fig. 2. Conﬁguration of directional clustering.

This supervised method can separate the target signal to someextent, particularly in the case of a small number of sources.
However, for the case of a mixture consisting of many sources,
such as more realistic musical tunes, the source extraction performance is markedly degraded because of the existence of instruments with similar timbre.
B. Conventional Multichannel Signal Separation Methods
1) Directional Clustering: Decomposition methods employing directional information of the multichannel signal havealso been proposed as unsupervised separation techniques [12],[13]. In this paper, we only focus on the gain-based directional
clustering method, which is a simple version of the techniquein [12]. Fig. 2 shows the conﬁguration of directional clustering.
First, the time-frequency components of a stereo mixed signal
are represented in a two-dimensional spacehaving the amplitude of each channel as the coordinate axes,
where
and
are the amplitudes of the left and rightchannels, respectively. Next, these components are normalized
over the unit circle to make apparent clusters, which correspond to each directional component. Finally, these clusters are
separated by the -means clustering method. Therefore, thismethod is equivalent to the quantization of directional information via time-frequency binary masking under the assumptionthat the sources are completely sparse (double disjoint) in thetime-frequency domain.
Such directional clustering works well, even in an underdetermined situation. However, there is an inherent problem
that sources located in the same direction cannot be separatedusing the directional information. Furthermore, the extracted
signal is likely to be distorted because of the effect of binary
masking in directional clustering. The signal in the target direction, which is obtained by directional clustering, has many
spectral chasms because the assumption of sparseness in thetime-frequency domain does not always hold completely. In
other words, the resolution of the spectrogram clustered as thetarget-direction components is degraded by time-frequency binary masking. Fig. 3 shows an example of the spectrum of asignal separated by directional clustering. The obtained spectrum has many chasms owing to the binary masking.
2) Multichannel NMF: Multichannel NMF, which is anatural extension of NMF for a stereo or multichannel musicsignal, has been proposed as an unsupervised signal separation
method [10], [11]. The algorithms used in this method employ
a Hermitian positive deﬁnite matrix that models the spatial
property of each NMF basis and each frequency bin. Therefore,

KITAMURA et al.: MULTICHANNEL SIGNAL SEPARATION COMBINING DIRECTIONAL CLUSTERING AND NMF

657

Fig. 3. Example of spectrum of signal separated by directional clustering.

multichannel NMF utilizes a frequency-wise transfer function
between the signal source and microphone as a cue for basis
clustering. However, such unsupervised separation is a difﬁcult
problem, even if the signal has multichannel components,
because the decomposition is underspeciﬁed. Hence, thesealgorithms involve strong dependence on initial values and lackrobustness.
III. SNMF WITH SPECTROGRAM RESTORATION
AND HYBRID METHOD
A. SNMF with Spectrogram Restoration
1) Motivation and Strategy: To separate the target source utilizing directional information, we can guess a hybrid methodthat concatenates SNMF after directional clustering (hereafterreferred to as naive hybrid method). This hybrid method can effectively extract the target instrument because the directionallyclustered signal contains only a few instruments. Moreover, theresidual interfering signal in the same direction can be removed
by SNMF.
However, such naive hybrid method has a problem that theextracted signal may suffer from the generation of considerable
distortion. This is because the spectrogram obtained from directional clustering has many spectral chasms owing to the binary
masking procedure. These spectral losses may deteriorate theseparation performance because SNMF is forced to incorrectly
ﬁt these spectral chasms using supervised bases. To solve thisproblem, in this section, we propose a new SNMF with spectrogram restoration as an alternative to the conventional SNMF
for the hybrid method [17].
Fig. 4 shows the signal ﬂow in the proposed hybrid methodthat includes SNMF with spectrogram restoration. The algorithm of SNMF with spectrogram restoration utilizes index
information determined in directional clustering. For example,
if the target instrument is localized in the center cluster alongwith the interference, SNMF is only applied to the existing
center components using index information (active binarymask). Therefore, the spectrogram of the target instrument
is reconstructed using more matched bases because spectral
chasms are treated as unseen, and these chasms have no impact
on the cost function in SNMF with spectrogram restoration.
In addition, the components of the target instrument lost after
directional clustering can be extrapolated using the supervisedbases. In other words, the deteriorated target spectrogram isrecovered with the spectrogram restoration via supervised basis
extrapolation. Furthermore, a soft directional mask, which
employs probabilities instead of binary indexes, can also be
applied to the proposed hybrid method (see Appendix A).

Fig. 4. Signal ﬂow of proposed hybrid method; SNMF with spectrogram
restoration is performed after directional clustering.

The proposed method requires directional information of thetarget signal, namely, we have to know which directional cluster
includes the target signal. In this paper, the target signal is always located in the center direction, and such a priori information for the direction is given. However, when the target signal
is located in an other direction, we can apply the proposed hybrid method in the same manner. In addition, even if the target
direction is unknown, we can obtain the separated signal by applying SNMF with spectrogram restoration to all the directions(clusters) and choosing the result with the highest quality.
To illustrate the separation mechanism step by step,
Fig. 5(a) shows the direction of arrival (D.O.A.) histogramof each source (shaded with various patterns to distinguishthe sources) in the stereo signal, (b) shows the separated
components that are clustered around the center direction
after directional clustering, and (c) shows the separated target
component obtained by SNMF with spectrogram restoration.
In Fig. 5(a), the source components are distributed in all
directions with some overlapping. This is because the soundsources are received with the room reverberation. After directional clustering (Fig. 5(b)), the center sources lose some of
their components (i.e., the tails on both sides), and the othersource components leak in the center cluster. The lost tail ofthe center sources corresponds to the binary-masked points inthe time-frequency domain, and the leaked tails in the center
cluster are the components of left- and right-side interference
sources, which are not masked in directional clustering. After
SNMF with spectrogram restoration, the proposed algorithm
restores the lost components by supervised basis extrapolation(Fig. 5(c)).
However, this basis extrapolation includes an underlying
problem. If the time-frequency spectra are almost unseen in
the spectrogram, which means that the indexes are almost all
zero, a large extrapolation error may occur. Then, incorrectbases are chosen and ﬁtted to a small number of time-frequency
points by incorrectly modifying the activation matrix . In theworst case, the activation matrix contains very large values
at a speciﬁc time. For example, when only one grid point isobserved and the other points are masked in a frame, this frame
is able to be extrapolated with any type of supervised bases.

658

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 23, NO. 4, APRIL 2015

The ﬁrst term represents the main cost of separation in SNMF.
Since the divergenceis only deﬁned in spectral grid
point whose index is one, the chasms in the spectrogram are
ignored in this SNMF decomposition. The second term forces
the minimization of the value of. Hence, the supervised bases are chosen so as to minimize the scale ofinproportion to the number of zeros in the index matrix in eachframe to avoid the extrapolation error [18]. In other words, thispenalty term regulates the extrapolation. As another means of
avoiding the extrapolation error, some people may guess that asimple sparse regularization for the activation can also be introduced instead of the proposed regularization. This issue will
be discussed in Appendix B. The third penalty term forces theother bases to become as different as possible from the supervised bases and can improve the separation performance [19].
3) Auxiliary Function Technique: The update rules of NMF
are usually derived by the auxiliary function technique, which
is an extension of the expectation-maximization algorithm. To
explain this technique, let us consider a general optimization
problem of ﬁnding an optimum parameter vector
thatsatisﬁes(15)
where is a cost function. In the auxiliary function technique,we have to ﬁnd an auxiliary function
satisfying(16)
Fig. 5. Directional source distribution of (a) observed stereo signal, (b) separated components in center cluster, and (c) component separated and extrapolated by spectrogram restoration.

If such an observed grid point has large value but the chosen
basis has a spectral valley at this grid point, a large gain of isgenerated for the chosen basis; this leads to unexpected spectral
peaks outside the observed grid point. Such an extrapolation
error generates very loud and unnatural sounds in the waveformdomain. To avoid this, we should add a new penalty term [18]in the cost function, as described in the next section.
2) Cost Function: We deﬁne the cost function of SNMF
with spectrogram restoration using -divergence. Here, theindex matrix
is obtained from the binary masking
preceding the directional clustering. This index matrix has
speciﬁc entries of unity or zero, which indicates whether or noteach grid point of the spectrogram belongs to the target directional cluster. The cost function in SNMF with spectrogram
restoration is deﬁned using the index matrix as

are called auxiliary variables. Then, instead of diwhererectly minimizing the cost function, the auxiliary function
is minimized in terms of and , alternately.
The iterative update rules are obtained as
(17)
(18)
In these updates, a monotonic decrease in
is guaranteed.
In addition, the update rules of auxiliary variables in an NMFbased method can usually be written in a closed form, and we
can obtain efﬁcient update rules for NMF variables.
4) Derivation of Update Rules: Similarly to in [16], we derive the update rules based on -divergence using an auxiliary
function technique. Here, we rewrite the cost function (14) using-divergence as
(19)
(20)
(21)

(14)
where
is the set of objective variables,isan entry of the index matrix , and are the weighting parameters for each term,
is a Frobenius norm, andrepresents the binary complement of each entry in the index matrix.

(22)
where constant terms are omitted.

KITAMURA et al.: MULTICHANNEL SIGNAL SEPARATION COMBINING DIRECTIONAL CLUSTERING AND NMF

First, we deﬁne the upper-bound function for. The ﬁrst
term ofis convex forand concave for, andthe second term is convex forand concave for. Applying Jensen’s inequality to the convex function and thetangent line inequality to the concave function, we can deﬁne theupper-bound function
using auxiliary variables
,
,
,
, and
that satisfy
,
, and
as
(23)

659

Similarly to (27)–(31), (34) is the update rule for the auxiliary
variable.
Third, we deﬁne the upper-bound function forusing theauxiliary variable
that satisﬁes
as(35)
The equality in (35) holds if and only if the auxiliary variable isset as follows:
(36)

where
(24)

Equation (36) is the update rule for the auxiliary variable.
Finally, using (23), (32), and (35), we can deﬁne the upperbound function
as
(37)

(25)
(26)

where is the set of auxiliary variables. The update rules withrespect to each variable are determined by setting the gradientto zero.
From, we obtain(38)

The equality in (23) holds if and only if the auxiliary variables
are set as follows:
where
(27)

(39)
(28)
(40)

(29)

(41)

(30)
(31)

By solving (38) forassuming nonnegativity, we obtain

Therefore, (27)–(31) are the update rules for auxiliary variables,, , , and, which correspond to (17).
Second, we deﬁne the upper-bound function for . This term
is convex forand concave for. Similarly to(23)–(26), we can deﬁne the upper bound functionusing
auxiliary variables
and
as
(32)
where

(42)
This equation is one of the updates of the primary variables
and corresponds to (18). Then we can obtain more efﬁcient update rules ofby substituting the update rules of the auxiliary
variables (27), (29), (31), and (34) into (42) as follows:
(43)

(33)
The equality in (32) holds if and only if the auxiliary variable
is set as (27) andis set as follows:
(34)

where

is given by
(44)

660

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 23, NO. 4, APRIL 2015

The update rules of the other variables are similarly obtained asfollows:

Thus, we set the upper bound of
to a constantand deﬁne the corresponding p.d.f. with normalization coefﬁcient ,which is given by

(45)

(50)

(46)
where

is given by(47)

The convergence of these update rules has been theoretically[16].
proven for all real values of and
B. Experimental Analysis of Basis Extrapolation Based on
Generation Model
1) Optimal Divergence for Basis Extrapolation and Generation Model: The proposed method attempts both signal separation and basis extrapolation using the supervised bases .
In previous studies, the analysis of optimal divergence has been
discussed for signal separation [19], [20]. However, there has
been no discussion on the optimal divergence for the extrapolation techniques using NMF. In this section, we experimentally
analyze the extrapolation ability based on a statistical generation
model of the observed data , and determine the optimal divergence for basis extrapolation for various andvalues [21].
In NMF decomposition, the minimization of -divergence
between andcorresponds to a log-likelihood maximization under the assumption of the generation model offoreach [22]. The minimization ofis equivalent to
the maximization of. Here, we can rewriteasUsing (49), we can generate the most probable spectrogram foreach .
2) Simulation Conditions: To analyze the net extrapolation
ability, we simulated the spectrogram restoration task. In thissimulation, we generated random i.i.d. values, which obey thecorresponding generation model (49) for each , as the observed
data matrix . We compared, 1, 2, 3, 4 and, 1,
2, 3, and we used the same divergence in the training andseparation processes. The size of this data matrix was set toand. We set the parameters of each p.d.f.
to
,
,
,
,
, and
.
These parameters were determined so as to generate nonnegative random i.i.d. values that obey each corresponding generation model. Note that the parameters – simply determine
the scales of the input random variables and basically can be
set to arbitrary values without loss of generality. In addition, weused two types of data-missing pattern , in which 75% or 98%
of the spectral grid points were missing in a uniform manner,and the missing data
imitated the binary-masking procedure. The supervised bases were obtained by training using
the same data matrix , namely,in (7) and (8). The
number of supervised bases, , was 100, which is the half thevalue of , and the number of other bases, , was 30. Therefore,
the task was to reconstruct the original from the observations
with missing data,, using the trained bases.
3) Simulation Results and Discussion: We used the sourcesto-artifacts ratio (SAR) deﬁned in [23] as the accuracy of theextrapolation. In this task, the observed signal does not have
any interference sources. Therefore, SAR, which measures theabsence of artiﬁcial distortion, is a good evaluation score for the
restoration of the target signal. Here, the estimated signal
is deﬁned as
(51)

(48)
where
represents a parameter of the maximum likelihood estimation. A probability density function(p.d.f.) that corresponds to (48) is given by

where
is the allowable deformation of the target
source,is the allowable deformation of the sourcesthat account for the interference of the undesired sources, andis an artifact term that may correspond to the artifactsof the separation algorithm, such as musical noise, or simply
undesirable deformation induced by the nonlinear property ofthe separation algorithm. The formula for SAR is deﬁned as
(52)

(49)
is a gamma function. These generation models of, 1, and 2 are equivalent to exponential, Poisson, andGaussian distributions, respectively. The generation models forcorrespond to a distribution in which the probability increases exponentially with increasing. Strictly, such a distribution is not a p.d.f. because it diverges whenincreases.
whereFig. 6 shows the SAR result for each divergence and regularization. From this result, it is conﬁrmed that a higher provides better basis extrapolation regardless of the type of regularization ( ). In NMF decomposition, if we set to a large
value, the trained bases tend to become anti-sparse (smooth). In
contrast, if is close to zero, the trained bases become moresparsity-aware, and this property is suitable for normal NMF-

KITAMURA et al.: MULTICHANNEL SIGNAL SEPARATION COMBINING DIRECTIONAL CLUSTERING AND NMF

Fig. 6. Extrapolation abilities for (a) 75%-binary-masked data and (b) 98%binary-masked data.

Fig. 7. Conceptual illustration of trade-off between separation and extrapola.
tion abilities. The overall performance is highest whenbased music source separation because of the inherent sparsityof music spectrograms (e.g.,
is recommended in [19],[20]). However, for basis extrapolation, sparse bases are notsuitable because it is difﬁcult to extrapolate them only from theobservable data. Therefore, we speculate that the optimal divergence in SNMF with spectrogram restoration, which attempts
to ﬁt the trained bases using spectral components except forchasms, is shifted torather than KL-divergence ()
because of the trade-off between separation and extrapolation
abilities, as illustrated in Fig. 7. This issue will be conﬁrmed
experimentally in the next section.
C. Comparison Between Proposed Hybrid Method andConventional Methods
1) Experimental Conditions: We conducted an objective
evaluation to conﬁrm the effectiveness of the proposed hybrid
method described in the previous section. In this experiment, we
compared the separation performance of six methods, namely,
simple directional clustering [12], multichannel NMF [11] andits supervised version (supervised multichannel NMF), simple
SNMF [19], naive hybrid method described in Section III-A1,
and the proposed hybrid method including SNMF with spectrogram restoration after directional clustering, in terms of theirability to separate artiﬁcial and real-recorded music signals.
The supervised multichannel NMF employs a priori trainingof the target spectral bases as well as SNMF and the hybrid
methods, and we initialized the spatial covariance matricesof the supervised bases as the center direction for directional
supervision of the target source. Also, we compared evaluation
scores obtained with various andfor SNMF, naive hybrid
method, and the proposed hybrid method by setting ﬁve divergences and three regularizations, namely,, 1, 2, 3, 4 and, 1, 2. We used the same divergence ( ) in the training
and separation processes for the supervised methods.
In this evaluation, we conducted two experiments to consider
artiﬁcial signal and real-recorded signal cases. We used stereo
signals containing four melody parts (depicted in Fig. 8) withthree compositions (C1–C3) of instruments as shown in Table I.

661

Fig. 8. Scores of each part.

Fig. 9. Scores of each training sound that contain notes over two octaves. Notethat only target instrumental sound is used in training stage.
TABLE ICOMPOSITIONS OF MUSICAL INSTRUMENTS

The training signal
consisted of notes over two octaves
that covered all the notes of the target instrument in the observed signal (see Fig. 9). This was artiﬁcially generated by a
YAMAHA MU-1000 PCM-based MIDI synthesizer (hereafterreferred to as Tone Generator A). Note that only the target instrument was trained in the training stage. We prepared three
types of observed test signals , namely, test signals generatedby Tone Generator A, another type of PCM-based MIDI synthesizer Microsoft GS Wavetable Synth (hereafter referred toas Tone Generator B), and Garritan Personal Orchestra 4 (hereafter referred to as Tone Generator C). The test signal generated by Tone Generator A has the same timbre as the training
sound, meaning that the best supervised bases were given forthe separation task. The test signal generated by Tone Generator
B provides different synthesized instrumental sounds, and thatgenerated by Tone Generator C imitates more realistic sounds
based on professionally recorded sample sounds. In addition,
when using Tone Generator B and Tone Generator C, we added
independent white Gaussian noises to the left and right channels of the observed signal withdB to simulate
background noise. In particular, these stereo signals were mixed
down to a monaural format only when we evaluated the separation accuracy of SNMF because SNMF is a separation method
for a monaural input signal. .
In the artiﬁcial signal case, the observed signals were produced by mixing four sources with the same power. The observed signal contained one source each in the left and right directions and two sources in the center direction based on the sinelaw (see Fig. 10(a)). The target instrument was always locatedin the center direction along with another interfering instrument,

662

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 23, NO. 4, APRIL 2015

Fig. 10. Location of four sources with sine law used in (a) artiﬁcial signal and(b) real-recorded signal cases. Numbered black circles represent locations of
instruments in stereo format. The angle of left- and right-side sources are
in artiﬁcial signal case and within real-recorded signal case.

and we prepared two patterns in which the left and right sourceswere located atand 45 , respectively. The sampling
frequency of all the signals was 44.1 kHz. The spectrograms
were computed using a 92-ms-long rectangular window with a
46 ms overlap shift. These STFT settings were determined so as
to obtain sufﬁcient frequency resolution. The number of iterations used for the training and separation were 500. The numberof a priori bases, , was set to 100 to prepare four bases foreach of the training notes (24 notes). In addition, the numberof clusters used in directional clustering was 3, the number of apriori bases, , was 100, and the number of bases for matrix ,, was 30. The weighting parameter for the orthogonal penalty,, was set to 10000 because suitably chosen high value gives
a good separation result [19]. The weighting parameter for the
regularization term, , affects the extrapolation and quality ofseparated sound. In this experiment, was set to the optimalvalue based on the development dataset, which comprised theobserved signals whose target was an oboe. The rest of the observed signals were used as a test dataset.
In the real-recorded signal case, we recorded each instrumental solo signal and the supervision sound, which were generated by Tone Generator A, using a NEUMANN KU 100 binaural microphone in an experimental room whose reverberation time was 200 ms. The levels of background noise and thesound source measured at the microphone were 37 dB(A) and60 dB(A), respectively. The geometry of the loudspeaker andbinaural microphone is shown in Fig. 10(b), where.
The target source and the supervision sound were always locatedat position No.1 in Fig. 10(b). The observed signal was produced by mixing these recorded signals at the same power. The
other conditions were the same as those of the artiﬁcial signal
case.
2) Experimental Results: We used the signal-to-distortion
ratio (SDR) deﬁned in [23] as the evaluation score. The formulafor SDR is deﬁned as(53)
SDR indicates the total evaluation score, which involves
the quality of the separated target sound and the degree ofseparation.
Figs. 11–13 show the average SDR scores of the proposed
hybrid method and the other methods for each divergence ( )and each regularization ( ) in the artiﬁcial signal case with

Fig. 11. Average SDR scores in artiﬁcial signal case using Tone Generator A
and (b).
when (a)and, where the four instruments are shufﬂed with 12 combinations in each of the compositions C1–C3,
and three input signals whose target is the oboe are used as adevelopment dataset. Therefore, these results are the averagesof 33 input signal patterns (test dataset). Also, Fig. 14 shows
the average SDR scores in the real-recorded signal case. From
the SDR scores in Figs. 11––14, we can conﬁrm that directional clustering and multichannel NMF do not have satisfactory
performance because they cannot discriminate the sources inthe same direction. Supervised multichannel NMF also cannot
achieve satisfactory separation performance. For this reason, it
is expected that (a) this method should be used to classify four
source clusters with two-channel inputs, compared with two
clusters (target and the rest;and) in SNMF, and (b) as
the number of clusters increases, this method should optimize
more parameters such as spatial covariance matrices and latentvariables, even if the target bases are given. In particular, thescores of multichannel NMF in Figs. 12–13 are markedly worse.
In multichannel NMF, we must cluster the decomposed bases
using their spatial covariance matrices to achieve the separation.
However, if the diffuse noise exists, this method cannot separate
the target signal well because such spatially uniform noise interferes with the clustering of decomposed bases. In contrast,
SNMF-based methods can reduce such background noise bypushing them into the non-target component
as interfering
sources. This result shows an advantage of SNMF methods interms of the robustness against the background noise. Also,

KITAMURA et al.: MULTICHANNEL SIGNAL SEPARATION COMBINING DIRECTIONAL CLUSTERING AND NMF

663

Fig. 13. Average SDR scores in artiﬁcial signal case using Tone Generator C
and (b).
with white noise when (a)Fig. 12. Average SDR scores in artiﬁcial signal case using Tone Generator B
and (b).
with white noise when (a)in artiﬁcial signal cases, the sources are spatially arranged according to only the difference of amplitudes (sine law) between
channels. Therefore, two sources at the center (Nos. 1 and 4 inFig. 10(a)) have identical spatial properties. Thus, multichannel
NMF cannot distinguish these sources and never achieves good
separation in artiﬁcial signal cases. In contrast, for the realrecorded signals, supervised multichannel NMF achieves a certain level of separation because the two sources at the center
direction have different room transfer functions (i.e., different
spatial covariance matrices) as shown in Fig. 10(b).
The methods using SNMF give better results and the proposed hybrid method using SNMF with spectrogram restoration outperforms all other methods in both artiﬁcial and realrecorded signal cases. The naive hybrid method is inferior to
SNMF when, where this hybrid method utilizes both
directional clustering and SNMF. This is because the naive hybrid method is affected by spectral chasms and cannot reconstruct such lost components. Furthermore, we can conﬁrm that
the EUC-distance-based cost function () is the optimal
divergence for the proposed hybrid method, whereas KL-divergence () is the best divergence even for conventional
SNMF [19], [20]. This marked shift of the optimal divergencefor SNMF with spectrogram restoration is due to the trade-off
between the separation and extrapolation abilities, as predicted

Fig. 14. Average SDR scores in real-recorded signal case when.

in Section III-B. In addition, the regularization with KL-divergence () is slightly better than that with the other divergences but the difference is not signiﬁcant, except when.
IV. SNMF WITH SPECTROGRAM RESTORATION BASEDON ADAPTIVE DIVERGENCE
A. Divergence Dependence on Local Chasm Condition
In Section III, we revealed the mechanism of the shift in the
optimal divergence in the SNMF methods. This shift is due to664

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 23, NO. 4, APRIL 2015

the trade-off between separation and extrapolation abilities. The
optimal divergence for SNMF with spectrogram restoration depends on the density of spectral chasms in each time frame of thespectrogram obtained by the preceding directional clustering.
Therefore, the optimal divergence temporally ﬂuctuates because
the spatial condition is not consistent in a general music signal,and the divergence of SNMF should be automatically changedin each time frame. To solve this problem, in this section, we
propose a new scheme for framewise divergence selection to
separate the target signal using the optimal divergence.
If there are many chasms in a frame of the binary-masked
spectrogram, it is preferable for SNMF to have high extrapolation ability. In contrast, if the density of chasms is low, separation ability is required rather than extrapolation ability. Therefore, it is expected that EUC-distance should be used in theframes with many chasms and KL-divergence should be used in
the other frames. To improve the total separation performance of
SNMF with spectrogram restoration for all types of input signal,
we introduce an adaptive-divergence-based cost function as described in the next section.
B. Cost Function and Update Rules
Considering the above-mentioned dependence of divergenceon the local chasm condition, we propose to adapt the divergence in each frame of the spectrogram so that it is the optimalvalue according to the density of chasms in each frame and athreshold value (), where the density of chasms
can be calculated from the index matrix . A straightforward but naive extension to this purpose is to apply independent
SNMF with spectrogram restoration to short-time-period data
while switching the divergence in an online manner (hereafterreferred to as online hybrid method). In this method, however,the size of each input matrix becomes small and the dimensionality is reduced. This degrades the separation performance because the trained bases
can represent any small-dimension
matrix and no component is pushed into the interference.
To cope with this problem and maintain sufﬁcient dimensionality of the matrix, we propose a new batch SNMF with spectrogram restoration that includes an adaptive-divergence-based
cost function covering the whole input matrix (see Fig. 15). The
proposed cost function
is deﬁned as

Fig. 15. Adaptive divergence algorithm in proposed method.

NMF and EUC-distance-based NMF, respectively. Also,and
are the entries ofand, respectively,andare the weighting parameters for each term, and. The divergence is determined fromand in eachframe. Therefore, this method can be considered as framewisely
adaptive SNMF (hereafter referred to as adaptive-divergencebased hybrid method) to achieve both optimal separation andextrapolation.
The update rules based on (54) are obtained by the auxiliary
function approach. Similarly to in Section III-A2, we can design
the upper-bound function
using auxiliary variables
,
,
,
,
, and
that
,
,
, and
satisfy
as
(58)

(59)

(54)
where

(55)

(56)

(60)

(61)

(57)
and
are the supervised
wherebasis matrices trained in advance using KL-divergence-based(62)

KITAMURA et al.: MULTICHANNEL SIGNAL SEPARATION COMBINING DIRECTIONAL CLUSTERING AND NMF

665

(63)

(64)
(65)
(66)
The equality in (59) holds if and only if the auxiliary variables
are set as in (28) and as follows:

Fig. 16. Scores of each part. The observed signal consists of four measures.
TABLE IISPATIAL CONDITIONS OF EACH DATASET

(67)
(68)
(69)
(70)
(71)
The update rules are obtained as follows by differentiating theupper-bound function (58) w.r.t. each objective variable andsubstituting of the equality conditions (67)–(71);
(72)

(73)

(74)

where

,

,

, and

are given by
(75)
(76)
(77)
(78)

The update rules of SNMF with spectrogram restoration basedon adaptive divergence are deﬁned as (72)–(74).
C. Evaluation Experiment
1) Experimental Conditions: To conﬁrm the effectiveness ofthe proposed algorithm, we compared seven methods, namely,
SNMF based on KL-divergence and EUC-distance [19], simple
directional clustering [12], multichannel NMF [11] and its

supervised version, the conventional hybrid method based on
KL-divergence and EUC-distance, the online hybrid method
described in Section IV-B, and the proposed hybrid methodthat uses adaptive divergence.
In this experiment, similarly to in Section III-C1, we produced artiﬁcial and real-recorded stereo signals containing four
melody parts (depicted in Fig. 16) with the three compositions(C1–C3) of instruments shown in Table I. The artiﬁcial training
and observed signals were generated with the same conditionsin Section III-C1. These stereo signals were mixed down to amonaural format only when we evaluated the separation accuracy of SNMF. In addition, we prepared four spatially different
dataset patterns of the observed signals, SP1–SP4, as shown in
Table II. Note that the target signal was always located in thecenter direction along with another interference signal as shownin Fig. 10, and the left- and right-side interference signals were
instantaneously moved to the center direction in the middle ofthe song for SP1–SP3. In the hybrid method, a large number ofchasms were produced by directional clustering in the measureswithcompared with those with. Therefore,
we expected that EUC-distance-based hybrid method would besuitable for the dataset of SP4 rather than the dataset of SP1.
The threshold value was set to 20%, which appears to be relatively small. This is because the separated sound quality is particularly important in music signal separation and the spectral
chasms should be actively extrapolated. The type of regularization was. The other experimental conditions were thesame as those in Section III-C1.
2) Experimental Results: Fig. 17 shows the average SDR
scores for each method and each dataset pattern. These results
are the averages of 33 input signal patterns, similarly to in
Section III-C1. The SDR scores of SNMF are the same for all
datasets because the input signals for SNMF were mixed down
to a monaural format.
From these results, the KL-divergence-based hybrid method
achieves high separation accuracy for the datasets of spatial
patterns SP1 and SP2 because these signals do not have many
spectral chasms. On the other hand, the EUC-divergence-based
hybrid method achieves high separation accuracy for SP4.

666

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 23, NO. 4, APRIL 2015

addition, the proposed hybrid method with adaptive divergenceachieves better separation for all datasets regardless of whether
or not many chasms exist. This is because the proposed method
selects the appropriate divergence and can automatically apply
the optimal divergence to each time frame.
D. Experimental Comparison Between
Adaptive-Divergence-Based Hybrid Method andAnother Strategy

Fig. 17. Average SDR scores of each method and each spatial condition in(a) artiﬁcial signal case using Tone Generator A, (b) artiﬁcial signal case using
Tone Generator B with white noise, (c) artiﬁcial signal case using Tone Generator C with white noise, and (d) real-recorded signal case.

This dataset has many spectral chasms because the signals are
mixed with a wide panning angle (), which yields
many chasms, and high extrapolation ability is required. In

1) Parallel Divergence Method: As another means of applying multiple divergence to a whole spectrogram (batch
method), the following method can also be considered forthe adaptation of divergence. First, we divide the whole
spectrogram
into two parts,and, based on thedensity of chasms
and threshold , whereconsistsof the frames with greater than andconsists of theother frames. Then, we apply the EUC-distance-based andKL-divergence-based proposed methods toand, respectively, in parallel. Finally, the separated whole spectrogramis reconstructed by concatenating the frames of the separated
spectrograms in the original order. Hereafter we refer to thismethod as parallel-divergence-based hybrid method. In thismethod, the update rules ofandare equivalent to (72)and (74), respectively. The difference between the parallel- andadaptive-divergence-based hybrid methods is how to deal with
the interference matrix. In the adaptive-divergence-based
hybrid method, a single
is optimized over all the framesin(thus, the dimensionality ofis identical to that of). On the other hand, the parallel-divergence-based hybrid
method prepares two interference matrices,
and
for
and
, respectively, whose frames are
disjoint and whose dimensionality is reduced compared withthat of . Generally speaking, in the SNMF-based methods, thedimensionality of the input spectrogram affects the separation
accuracy. This is because the interference matrix
becomes
an effective low-rank representation that ensures the successof separation when the number of frames increases. Therefore,
we expect the parallel-divergence-based hybrid method tounderperform compared with the adaptive-divergence-based
hybrid method because the numbers of frames inandare small. This phenomenon is also expected to become
more apparent as the number of frames indecreases. This
will be experimentally conﬁrmed in the next subsection.
2) Conditions and Results: We compared the two proposed
hybrid methods based on adaptive and parallel divergence. We
used three different lengths of the observed signals, which consist of two, three, and four measures with four melody parts(depicted in Fig. 16). As the spatial conditions, SP1 and SP2
were generated for the two-measure signal, SP1–SP3 were generated for the three-measure signal, and SP1–SP4 were generated for the four-measure signal, which were used in addition to the spatial conditions in Section IV-C. Similarly to in
Section IV-C1, we produced artiﬁcial and real-recorded stereo
signals containing the three compositions (C1–C3) of instruments shown in Table I. For the parallel divergence method, we
set the number of bases inandto 30. The other experimental conditions were the same as those in Section IV-C1.

KITAMURA et al.: MULTICHANNEL SIGNAL SEPARATION COMBINING DIRECTIONAL CLUSTERING AND NMF

Fig. 18. Average SDR scores of each method and each spatial condition in
artiﬁcial signal case using Tone Generator A: (a) two-measure-signal, (b) threemeasure-signal, and (c) four-measure-signal.

Fig. 19. Average SDR scores of each method and each spatial condition in
artiﬁcial signal case using Tone Generator B with white noise: (a) two-measuresignal, (b) three-measure-signal, and (c) four-measure-signal.

667

Fig. 21. Average SDR scores of each method and each spatial condition in
real-recorded signal case: (a) two-measure-signal, (b) three-measure-signal, and(c) four-measure-signal.

spectrogram restoration after directional clustering. Via extrapolation of supervised spectral bases, the proposed SNMF with
spectrogram restoration attempts both target signal separationand the reconstruction of the lost target components, which are
generated by the preceding binary masking performed in directional clustering.
Secondly, from experimental analysis based on the generation
model of the signal, it was revealed that the optimal divergencein SNMF with spectrogram restoration, which attempts to ﬁt thetrained bases using spectral components except for the chasms,
is shifted to an anti-sparse divergence rather than KL-divergence. This was due to the fact that a trade-off exists between
the separation and extrapolation abilities in SNMF. An experiment evaluating the separation using artiﬁcial and real-recorded
music signals showed the effectiveness of the proposed hybrid
method.
Finally, on the basis of this ﬁnding, we also proposed an improved hybrid method based on adaptive divergence. The proposed method adapts the divergence in each frame to the optimal
one using a threshold value for the density of chasms to separate and extrapolate the target signal with high accuracy. Experimental results showed that our proposed method can achieve
high separation accuracy under all spatial conditions.
APPENDIX AHYBRID METHOD WITH SOFT DIRECTIONAL MASK

Fig. 20. Average SDR scores of each method and each spatial condition in
artiﬁcial signal case using Tone Generator C with white noise: (a) two-measuresignal, (b) three-measure-signal, and (c) four-measure-signal.

Figs. 18–21 show the average SDR scores for each method
and each case. From these results, we can conﬁrm that the adaptive-divergence-based hybrid method outperforms the paralleldivergence-based hybrid method for all tasks. The degree of superiority is marked when the input signal is short (e.g., compare Fig. 18(a) with Fig. 18(c)), as predicted in the previous
subsection.
V. CONCLUSION
In this paper, we ﬁrst proposed a new multichannel signal separation method, i.e., a hybrid method that combines SNMF with

Instead of employing a hard clustering method for directional
separation, we can apply soft directional clustering, which
provides a soft time-frequency mask. Various types of soft
mask have been proposed [24]–[27]. Here, we generated a soft
mask using MESSL as proposed in [27]. In MESSL, we chosethe speciﬁc set of parameters,, with frequency-independent
Gaussian model for interaural level and phase differences [27],which gave the best SDR score in our task.
To evaluate the efﬁcacy of the soft-mask-based hybrid method, we conducted the experiment described in
Section III-C with a real-recorded signal case. Simple soft
directional clustering [27] gives an SDR of less than 5 dB.
Fig. 22 shows a comparison of the SDR between the proposed
hybrid methods using hard and soft directional masks with
SNMF. From these results, the hybrid method with a soft maskslightly improves the SDR, although a clear improvement is668

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 23, NO. 4, APRIL 2015

APPENDIX BSPARSE REGULARIZATION FOR ACTIVATION MATRIX
To avoid the extrapolation error, the sparse regularization for
can also be used. This can be achieved by substituting thefollowing equation forterm in (19):(79)
Fig. 22. Average SDR scores of hybrid methods with hard and soft masks inreal-recorded signal case.

or 2, which corresponds toornorm of .
whereThis penalty term increases sparseness of .
To compare the separation performance in the cases of (79)and (21), we conducted the same experiment in Section III-C.
The results are shown in Fig. 23. From these results, sparse regularization for can also avoid the extrapolation error but thescores do not outperform that of the proposed penalty. This isbecause the proposed penalty only affects the frames that have
many chasms whereas the penalty in (79) imposes unnecessary
sparseness to all the frames.
REFERENCES

Fig. 23. Average SDR scores with sparse and proposed regularizations in (a) artiﬁcial signal case using Tone Generator A, (b) artiﬁcial signal case using Tone
Generator B with white noise, (c) artiﬁcial signal case using Tone Generator Cwith white noise, and (d) real-recorded signal case.

not obtained. This is because even if all the target components
are lost in some STFT grid points as a result of binary masking,
SNMF with spectrogram restoration can reconstruct the lost
target components by basis extrapolation. Thus, the restoration
ability does not depend on the type of directional mask.

[1] N. Ono, K. Miyamoto, J. Le Roux, H. Kameoka, and S. Sagayama,“Separation of a monaural audio signal into harmonic/percussive components by complementary diffusion on spectrogram,” in Proc. 16th
Eur. Signal Process. Conf., 2008.
[2] A. Mesaros, T. Virtanen, and A. Klapuri, “Singer identiﬁcation in polyphonic music using vocal separation and pattern recognition methods,”
in Proc. 8th Int. Conf. Music Inf. Retrieval, 2007, pp. 375–378.
[3] D. D. Lee and H. S. Seung, “Algorithms for non-negative matrix factorization,” in Proc. Adv. Neural Inf. Process. Syst., 2001, vol. 13, pp.
556–562.
[4] T. Virtanen, “Monaural sound source separation by nonnegative matrix
factorization with temporal continuity and sparseness criteria,” IEEE
Trans. Audio, Speech, Lang. Process., vol. 15, no. 3, pp. 1066–1074,
Mar. 2007.
[5] A. Ozerov, C. Fevotte, and M. Charbit, “Factorial scaled hidden
markov model for polyphonic audio representation and source separation,” in Proc. IEEE Workshop Applicat. Signal Process. Audio
Acoust., 2009, pp. 121–124.
[6] H. Kameoka, M. Nakano, K. Ochiai, Y. Imoto, K. Kashino, and S.
Sagayama, “Constrained and regularized variants of non-negative matrix factorization incorporating music-speciﬁc constraints,” in Proc.
IEEE Int. Conf. Acoust., Speech, Signal Process., 2012, pp. 5365–5368.
[7] P. Smaragdis, B. Raj, and M. Shashanka, “Supervised and semi-supervised separation of sounds from single-channel mixtures,” in Proc. 7th
Int. Conf. Ind. Compon. Anal. Signal Separat., 2007, pp. 414–421.
[8] P. Smaragdis, “Convolutive speech bases and their application tosupervised speech separation,” IEEE Trans. Audio, Speech, Lang.
Process., vol. 15, no. 1, pp. 1–12, Jan. 2007.
[9] N. Mohammadiha, P. Smaragdis, and A. Leijon, “Supervised andunsupervised speech enhancement using NMF,” IEEE Trans. Audio,
Speech, Lang. Process., vol. 21, no. 10, pp. 2140–2151, Oct. 2013.
[10] A. Ozerov and C. Fevotte, “Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation,” IEEE Trans.
Audio, Speech, Lang. Process., vol. 18, no. 3, pp. 550–563, Mar. 2010.
[11] H. Sawada, H. Kameoka, S. Araki, and N. Ueda, “Efﬁcient algorithms
for multichannel extensions of Itakura-Saito nonnegative matrix factorization,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.,
2012, pp. 261–264.
[12] S. Araki, H. Sawada, R. Mukai, and S. Makino, “Underdeterminedblind sparse source separation for arbitrarily arranged multiple sensors,” Signal Process., vol. 87, no. 8, pp. 1833–1847, 2007.
[13] Y. Li and D. Wang, “On the optimality of ideal binary time-frequency
masks,” Speech Commun., vol. 51, no. 3, pp. 230–239, 2009.
[14] P. Smaragdis and B. Raj, “Example-driven bandwidth expansion,” inProc. IEEE Workshop Applicat. Signal Process. Audio Acoust., 2007,
pp. 135–138.
[15] S. Eguchi and Y. Kano, Robustifying maximum likelihood estimation
Inst. of Statist. Math., Tech. Rep., 2001.

KITAMURA et al.: MULTICHANNEL SIGNAL SEPARATION COMBINING DIRECTIONAL CLUSTERING AND NMF

669[16] M. Nakano, H. Kameoka, J. Le Roux, Y. Kitano, N. Ono, and S.
Sagayama, “Convergence-guaranteed multiplicative algorithms fornonnegative matrix factorization with beta-divergence,” in Proc. IEEE
Int. Workshop Mach. Learn. Signal Process., 2010, pp. 283–288.
[17] D. Kitamura, H. Saruwatari, Y. Iwao, K. Shikano, K. Kondo, and
Y. Takahashi, “Superresolution-based stereo signal separation via
supervised nonnegative matrix factorization,” in Proc. Digital Signal
Process., 2013, pp. T3C–2.
[18] D. Kitamura, H. Saruwatari, K. Shikano, K. Kondo, and Y. Takahashi,“Superresolution-based stereo signal separation with regularization ofsupervised basis extrapolation,” in Proc. 3D Systems Applicat., 2013,
pp. S10–4.
[19] D. Kitamura, H. Saruwatari, K. Yagi, K. Shikano, Y. Takahashi, and K.
Kondo, “Music signal separation based on supervised nonnegative matrix factorization with orthogonality and maximum-divergence penalties,” IEICE Trans. Fundam. Electron., Commun. Comput. Sci., vol.
E97-A, no. 5, pp. 1113–1118, 2014.
[20] D. FitzGerald, M. Cranitch, and E. Coyle, “On the use of the beta divergence for musical source separation,” in Proc. Irish Signals Syst.
Conf., 2009.
[21] D. Kitamura, H. Saruwatari, S. Nakamura, Y. Takahashi, K. Kondo,
and K. Kameoka, “Divergence optimization in nonnegative matrix factorization with spectrogram restoration for multichannel signal separation,” in Proc. Joint Workshop Hands-Free Speech Commun. Microphone Arrays, 2014, pp. 92–96.
[22] A. T. Cemgil, “Bayesian inference for nonnegative matrix factorisation
models,” Comput. Intell. Neurosci., vol. 2009, pp. 1–17, 2009.
[23] E. Vincent, R. Gribonval, and C. Fevotte, “Performance measurementin blind audio source separation,” IEEE Trans. Audio, Speech, Lang.
Process., vol. 14, no. 4, pp. 1462–1469, Jul. 2006.
[24] S. Srinivasan, N. Roman, and D. L. Wang, “Binary and ratio time-frequency masks for robust speech recognition,” Trans. Speech Commun.,vol. 48, pp. 1486–1501, 2006.
[25] S. Harding, J. Barker, and G. J. Brown, “Mask estimation for missing
data speech recognition based on statistics of binaural interaction,”
IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 1, pp. 58–67,
Jan. 2006.
[26] N. Madhu and J. Wouters, “Localisation-based, situation-adaptivemask generation for source separation,” in Proc. 4th Int. Symp.
Commun., Control, Signal Process., 2010.
[27] M. I. Mandel, R. J. Weiss, and D. P. W. Ellis, “Model-based expectation-maximization source separation and localization,” IEEE Trans.
Audio, Speech, Lang. Process., vol. 18, no. 2, pp. 382–394, Feb. 2010.

Hiroshi Saruwatari (M’00) received the B.E.,
M.E., and Ph.D. degrees from Nagoya University,
Japan, in 1991, 1993, and 2000, respectively. After
joining Nara Institute of Science and Technology,
he is currently a Professor of The University ofTokyo, Japan. His research interests include speech
and acoustic signal processing. He is a member of
the IEICE, Japan VR Society, and the AcousticalSociety of Japan.

Daichi Kitamura (M’15) graduated from Nara institute of Science and Technology, Japan, in 2014 andreceived the M.E. degree. He is currently pursuing
the Ph.D. degree at The Graduate University for
Advanced Studies (SOKENDAI), Kanagawa, Japan.
His research interests include source separation andmusic signal processing. Mr. Kitamura is a memberof the IEICE and the Acoustical Society of Japan.

Satoshi Nakamura received the B.S. from Kyoto
Institute of Technology in 1981 and the Ph.D. from
Kyoto University in 1992. He is currently a Professorof Nara Institute of Science and Technology, Japan.
His research interests include speech and language
processing. He has been Elected Board Memberof ISCA and IEEE Signal Processing Magazine
Editorial Board Member. He is a member of the
Acoustical Society of Japan.

Hirokazu Kameoka received B.E., M.S., and Ph.D.
degrees all from The University of Tokyo, Japan, in
2002, 2004, and 2007, respectively. He is currently a
Research Scientist at NTT CS Laboratories and a Visiting Associate Professor at The University of Tokyo.
His research interests include speech and music processing, and machine learning. He is the Information
Processing Society of Japan, and the Acoustical Society of Japan.

Yu Takahashi (S’07) received the B.E. degree in information engineering from the Himeji Institute ofTechnology, Japan, in 2005 and the M.E. and Ph.D.
degrees in information science from the Nara Institute of Science and Technology, Japan, in 2007 and2010, respectively. He joined Yamaha Co., Ltd. in2010. His research interests include array signal processing and blind source separation. He is a memberof the Acoustical Society Japan and a member of theJapanese Society for Artiﬁcial Intelligence.

Kazunobu Kondo received the B.E., M.E. and
Ph.D. degrees from Nagoya University, Japan, in
1991, 1993, and 2014, respectively. He joined theElectronics Development Center, Yamaha Co., Ltd.
in 1993. He is currently an Assistant Manager of
Yamaha Research and Development Division. His
research interests include blind source separationand noise reduction. He is a member of the IEICE
and the Acoustical Society of Japan.