Effective Optimization Algorithms
for Blind and SupervisedMusic Source Separation with
Nonnegative Matrix Factorization
byDaichi Kitamura

Dissertation
submitted to the Department of Informatics
in partial fulfillment of the requirements for the degree ofDoctor of Philosophy

SOKENDAI (The Graduate University for Advanced Studies)
March 2017

iii

Committee

AdvisorDr. Nobutaka Ono
Associate Professor of National Institute of Informatics/SOKENDAI

Subadvisor

Dr. Junichi Yamagishi
Associate Professor of National Institute of Informatics/SOKENDAI

Subadvisor

Dr. Hiroshi Saruwatari
Professor of The University of Tokyo

ExaminerDr. Ken Hayami
Professor of National Institute of Informatics/SOKENDAI

ExaminerDr. Imari Sato
Professor of National Institute of Informatics/SOKENDAI

v

Acknowledgments

This dissertation is a summary of three years of study carried out at School ofMultidisciplinary Sciences, SOKENDAI (The Graduate University for Advanced
Studies), Japan.
I would like to express the deepest appreciation to Dr. Nobutaka Ono,
Associate Professor of National Institute of Informatics and SOKENDAI, mydissertation adviser, for his continuous teaching and essential advice on bothtechnical and non technical issues. This work could not have been accomplishedwithout his well-directed advice, helpful suggestions, and fruitful discussions
with him. I have learned many valuable aspects of being a researcher from his
attitude toward study and have always enjoyed conducting research with him.
I would also like to express my deep gratitude to Dr. Hiroshi Saruwatari,
Professor of The University of Tokyo, my dissertation subadviser, for his valuableguidance and constant encouragement. Without his encouragement help andguidance, this dissertation would not have materialized.
I would like to express my sincere thanks to Dr. Junichi Yamagishi, Associate
Professor of National Institute of Informatics and SOKENDAI, my dissertationsubadviser, for his instructive comments for my research and on the dissertation.
I would like to offer my special thanks to Dr. Ken Hayami and Dr. Imari Sato,
Professors of National Institute of Informatics and SOKENDAI, members of thedissertation committee, for their valuable comments on the dissertation.
This work could not have been achieved without the collaboration of many
researchers. I especially thank Dr. Hiroshi Sawada and Dr. Hirokazu Kameoka,
researchers at Nippon Telegraph and Telephone Corporation, and Dr. Yu
Takahashi and Dr. Kazunobu Kondo, researchers at Yamaha Corporation, forvi
their beneficial and valuable comments and advice.
Many staff and members of my research group have supported me in carryingout experiments and writing this dissertation at National Institute of Informatics,
SOKENDAI, and The University of Tokyo; I would especially like to express myappreciation of the clerical supports for me, valuable discussions with them ontechnical issues, and their provision of a comfortable research environment. I
also wish to express my deep gratitude to Ms. Shigeko Morimoto, a secretary of
our laboratory, for her kind help and support in all aspects of my research.
I received satisfactory financial supports from Japan Society for the Promotionof Science as a Research Fellow DC1 and Research Organization of Informationand Systems as a Research Assistant with Superordinate Wages during my Ph.D.
pursuit. I deeply appreciate their supports of a research fund and salaries.
I appreciate the opportunity to have studied with all the members in our
research group at National Institute of Informatics, SOKENDAI, and The
University of Tokyo. I thank Dr. Trung-Kien Le, Dr. Keiko Ochi, and Dr. Shinji
Takaki who are Postdoctoral Fellows at National Institute of Informatics, foruseful discussions on this work. I would also like to thank Mr. Keisuke Imoto
who is a Ph.D. student and a colleague of mine from the same laboratory forworking hard together. I also thank many foreign internship students in ourlaboratory.
Finally, I wish to show my great appreciation to my family, Masao Kitamura,
Chizu Kitamura, and Izumi Masutani, for their support over many years, and I
would also like to offer my special thanks to Ms. Manami Kitamura who has
always supported my academic life since I enrolled in Master Course of Nara
Institute of Science and Technology.
March 13th, 2017.

vii

Abstract

In this dissertation, to address a music source separation problem, several
optimization algorithms are proposed. Music source separation is a technique toextract or separate specific music sources from an observed mixture signal that
contains multiple music instrumental and vocal sounds. There are many feasible
applications for this technique, for example, audio remixing by users, automaticmusic transcription, and musical instrument education. A general audio source
separation problem has been investigated for a long time, particularly in the
speech signal processing field to reduce background noise and enhance only the
speech signal in the observation. Many techniques have been proposed forvarious recording conditions in the past, and they can roughly be divided into
two situations: determined (or overdetermined) and underdetermined cases. In
the determined situation, sufficient number of observations (microphones usedin the recording) can be utilized for solving the separation problem, whereas theunderdetermined situation, which includes monaural observation, basicallylacks such multi-dimensional information. Also, presence of external priorinformation (supervision) such as music scores, source locations, or soundexamples of each source in the mixture is another important issue. The source
separation techniques without any prior information is often called blind source
separation, which is the most difficult but a practical technique.
The objective of this dissertation is to develop an effective optimization
algorithm for the music source separation and to achieve satisfactory separation
performance. Two main topics are here addressed: determined (and overdetermined) blind source separation and single-channel (underdetermined)semi-supervised source separation. The semi-supervised source separation

viii
exploits sound examples of only the target source for the separation, namely,
only the target source is extracted from the mixture. In both the topics, an
important property of music signals is focused to effectively capture theirstructures. Since typical music signals consist of limited number of components
such as discrete pitches and musical notes and include many reiteration ofsimilar or the same spectral patterns (timbers), the power spectrogram of musicsignals tends to have a low-rank structure. On the basis of this nature in musicsignals, for both the topics discussed in this dissertation, a popular algorithmof matrix decomposition called nonnegative matrix factorization (NMF) is
exploited for modeling the structure of music signals. By applying NMF to thespectrogram of audio signal, the frequently appearing spectral patterns and theirtime-varying gains can be extracted as bases and activations. These components
are useful for modeling the audio signals and achieving the source separation.
For the problem of determined blind source separation, independent component
analysis (ICA) and its multivariate extension, independent vector analysis(IVA), are traditional and reliable approaches and can provide good separation
results particularly for a mixture signal of speech. These approaches estimate
spatial demixing filters by assuming that the sources are mutually independent.
This assumption is valid in a practical mixture signal and make the separation
problem solvable in a fully blind fashion. However, the separation accuracy of
ICA and IVA for music signals is not satisfactory. This is because the generalmusic signals frequently contain spectral overlaps and co-occurrences between
sources, which result in a harmony of music, and these properties weaken theinherent independence between the sources. Also, the both methods assume
only the non-Gaussian source distribution as an unspecific source model and do
not utilize any information about the structure in the spectrogram of each source.
To solve this problem, in this dissertation, the unified method of NMF andIVA called independent low-rank matrix analysis (ILRMA) is proposed, which
performs simultaneous estimation of the spectrogram structure of each source
and their spatial demixing filters. The optimization algorithm in ILRMA ensures
faster convergence, more stable performance, and better computational efficiency
compared with conventional methods including multichannel extension of
NMF (MNMF), which is a state-of-the-art method for source separation. Also,

ix
theoretical relationships between IVA, MNMF, and ILRMA are revealed, namely,
ILRMA is essentially equivalent to MNMF with a constraint for the mixing
system, and IVA is also a special case of ILRMA.
For the single-channel semi-supervised source separation task, semisupervised NMF, which aims to extract only the target source from the
mixture, is the most popular approach. In this method, sound examples of thetarget source are utilized for preparing the supervised bases (spectral dictionary)of the target source. However, when the target source and the other sourcesin the mixture signal share similar or the same spectral patterns (bases), theseparation performance of semi-supervised NMF is degraded because suchshared components cannot be separated. This fact means that the supervisedbases must be discriminative from the other bases of non-target sources. On thebasis of this fact, in this dissertation, a new training algorithm that provides
discriminative supervised bases is proposed for semi-supervised NMF. In
this method, other sound examples, which are candidates of the non-target
signals in the observed mixture, are utilized only for learning which spectral
components will be frequently shared between the target and non-target sources.
Furthermore, a new efficient initialization scheme for NMF is proposed. Since
an optimization in NMF requires initial values for bases and activations, all the
results of applications based on NMF always depend on the initialization. The
proposed initialization is based on a maximization of mutual independence
between the activations using nonnegative ICA algorithm. The efficacy of the
proposed method for several source separation tasks including ILRMA andsemi-supervised NMF with discriminative basis training is experimentallyconfirmed.

xi

List of Abbreviations

BF
BSILRMA
BSS
DAE
DC
DNN
DOA
EM
EU distance
EUNMF
FDICA
FSNMF
ICA
i.i.d.
ILRMA
IP
IS divergence
ISNMF
IVA
KAM
KL divergence

Beamforming
Basis-shared independent low-rank matrix analysis
Blind source separation
Denoising autoencoder
Directional clustering
Deep neural network
Direction of arrival
Expectation-maximization
Squared Euclidean distance
Nonnegative matrix factorization based on squared
Euclidean distanceFrequency-domain independent component analysis
Full-supervised nonnegative matrix factorization
Independent component analysis
Independent and identically distributed
Independent low-rank matrix analysis
Iterative projection
Itakura–Saito divergence
Nonnegative matrix factorization based on Itakura–Saito
divergence
Independent vector analysis
Kernel additive model
Generalized Kullback–Leibler divergence

xii
KLNMF
MFCC
MIDI
ML
MNMF
MSM
MU
MWF
NICA
NMF
NNDSVD
PCA
PLCA
REPET
SDR
SSNMF
STFT
SVD
TDOA
TFM

Nonnegative matrix factorization based on generalized
Kullback–Leibler divergence
Mel-frequency cepstrum coefficients
Musical instrument digital interface
Maximum likelihood
Multichannel nonnegative matrix factorization
Multichannel sparse modelingMultiplicative update
Multichannel Wiener filtering
Nonnegative independent component analysis
Nonnegative matrix factorization
Nonnegative double singular value decomposition
Principal component analysis
Probabilistic latent component analysis
Repeating pattern extraction technique
Signal-to-distortion ratio
Semi-supervised nonnegative matrix factorization
Short-time Fourier transform
Singular value decomposition
Time difference of arrival
Time-frequency masking

xiiiContents

List of Figures

xvii

List of Tables

xxv

1

2

3

Introduction

1

1.1

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1

1.2

Prior Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2

1.3

Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

1.4

Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4Preliminaries

7

2.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7

2.2

Mathematical Formulation . . . . . . . . . . . . . . . . . . . . . . .

8

2.2.1

Multichannel Mixing and Demixing Systems . . . . . . . .

8

2.2.2

Single-Channel Mixing System . . . . . . . . . . . . . . . .

12

2.3

Existing Conventional Techniques and Their Categorization . . .

12

2.4

Motivations for Developing New Algorithms . . . . . . . . . . . .

17

2.5

Basic Principle of NMF . . . . . . . . . . . . . . . . . . . . . . . . .

22

2.6

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

Determined and Overdetermined Blind Source Separation Based on
Independent Low-Rank Matrix Analysis
27
3.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

3.2

Basic Principles of ICA, FDICA, IVA, and ISNMF . . . . . . . . . .

28

xiv

Contents

3.3

3.4

3.5

3.6

3.7

3.8

3.2.1

ICA and FDICA . . . . . . . . . . . . . . . . . . . . . . . . .

28

3.2.2

IVA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

3.2.3

ISNMF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

3.2.4

Time-Varying Gaussian IVA . . . . . . . . . . . . . . . . . .

40

Independent Low-Rank Matrix Analysis . . . . . . . . . . . . . . .

41

3.3.1

Motivation and Strategy . . . . . . . . . . . . . . . . . . . .

41

3.3.2

Derivation of Cost Function . . . . . . . . . . . . . . . . . .

42

3.3.3

Update Rules . . . . . . . . . . . . . . . . . . . . . . . . . .

44

3.3.4

Summary of Algorithm . . . . . . . . . . . . . . . . . . . .

48

Relationship between IVA, MNMF, and ILRMA . . . . . . . . . . .

51

3.4.1

Generative Model in MNMF and Spatial Covariance . . .

51

3.4.2

Existing MNMF Models . . . . . . . . . . . . . . . . . . . .

52

3.4.3

Equivalence between ILRMA and MNMF with Rank-1
Spatial Model . . . . . . . . . . . . . . . . . . . . . . . . . .

53

Experimental Analysis of ILRMA using Artificial Observation . .

56

3.5.1

Difference between Assumption in Source Model . . . . .

56

3.5.2

Difference between Assumption in Spatial Model . . . . .

57

3.5.3

Experimental Validation . . . . . . . . . . . . . . . . . . . .

58

Comparison of Speech and Music Separation Performance . . . .

64

3.6.1

Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

64

3.6.2

Experimental Analysis of Optimal Number of Bases . . . .

65

3.6.3

Comparison of Separation Performance . . . . . . . . . . .

68

3.6.4

Experiments on Three-Source Case with Music Signals . .

78

3.6.5

Experimental Analysis of Optimal Window Length . . . .

80

Extension of ILRMA for Overdetermined and Reverberant Recording 89
3.7.1

PCA for Overdetermined BSS . . . . . . . . . . . . . . . . .

89

3.7.2

Relaxation of Rank-1 Spatial Model in ILRMA . . . . . . .

90

3.7.3

Clustering with Spectral Correlations . . . . . . . . . . . .

91

3.7.4

Auto-Clustering with Basis-Shared ILRMA . . . . . . . . .

92

3.7.5

Experiments and Results . . . . . . . . . . . . . . . . . . . .

94

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

99

Contents
4

Single-Channel Semi-Supervised Source Separation Based on Discriminative Nonnegative Matrix Factorization
101
4.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

4.2

Existing NMF-Based Single-Channel Source Separation . . . . . . 102

4.3

Conventional Supervised NMF and Discriminative Training ofSupervised Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

4.4

4.5

4.6
5

xv

4.3.1

Conventional Supervised NMF . . . . . . . . . . . . . . . . 103

4.3.2

Drawback in Supervised NMF and Motivation for Discriminative Basis Training . . . . . . . . . . . . . . . . . . . 107

4.3.3

Algorithm of Discriminative Basis Training for FSNMF . . 108

New Algorithm for Discriminative Basis Training . . . . . . . . . 109
4.4.1

Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

4.4.2

Discriminative and Reconstructive Basis Training in SSNMF111

Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.5.1

Simple Experiment Using Piano and Flute Tones . . . . . . 112

4.5.2

Music Source Separation . . . . . . . . . . . . . . . . . . . . 112

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

Effective Initialization for Nonnegative Matrix Factorization Based on
Statistical Independence
119
5.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

5.2

Conventional NMF Initializations . . . . . . . . . . . . . . . . . . . 120

5.3

5.4

5.2.1

Initialization with Random Values . . . . . . . . . . . . . . 121

5.2.2

Initialization without Random Values . . . . . . . . . . . . 121

Efficient NMF Initialization Based on ICA . . . . . . . . . . . . . . 122
5.3.1

Motivation and Strategy . . . . . . . . . . . . . . . . . . . . 122

5.3.2

Combination of PCA and ICA . . . . . . . . . . . . . . . . 124

5.3.3

Proposed Initialization using NICA . . . . . . . . . . . . . 125

5.3.4

Proposed Initialization using ICA and Differential of Data
Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

5.3.5

Nonnegativization . . . . . . . . . . . . . . . . . . . . . . . 127

Experimental Comparisons . . . . . . . . . . . . . . . . . . . . . . 128
5.4.1

Performance as Initial Value for NMF . . . . . . . . . . . . 128

xvi

Contents

5.5

5.6
6

5.4.2 Full-Supervised Audio Source Separation . . . . . . . . . . 133
Application to ILRMA and Discriminative SSNMF . . . . . . . . . 136
5.5.1 BSS Based on ILRMA with Various Initialization for NMF 136
5.5.2 Discriminative SSNMF with Various Initialization for NMF 137
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Conclusion
143
6.1 Summary of Dissertation . . . . . . . . . . . . . . . . . . . . . . . . 143
6.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Bibliography

147

A Derivation of Shape Parameter for Artificial Random Spectrogram with
Constant Kurtosis
177

xvii

List of Figures

2.1
2.2

Mixing system when N = 3 and M = 2, where only direct paths
are depicted as arrows. . . . . . . . . . . . . . . . . . . . . . . . . .

9Discreteness in music signals, where score is beginning of Prelude(op. 28, no. 7) by Frédéric Chopin. Typical music consists oflimited number of discrete parts. . . . . . . . . . . . . . . . . . . .

18

2.3

Example of power spectrogram of drums sound obtained from“another dreamer-the ones we love” in SiSEC2011 dataset, wheregrayscale indicates spectral power and white is stronger than black. 19

2.4

Example of power spectrogram of guitar sound obtained from“another dreamer-the ones we love” in SiSEC2011 dataset, wheregrayscale indicates spectral power and white is stronger than black. 19

2.5

Example of power spectrogram of vocals sound obtained from“another dreamer-the ones we love” in SiSEC2011 dataset, wheregrayscale indicates spectral power and white is stronger than black. 20

2.6

Example of power spectrogram of male speech sound obtained
from “dev1_male3_src” in SiSEC2011 dataset, where grayscale
indicates spectral power and white is stronger than black. . . . . .

20

Example of cumulative singular values of music and speech
spectrograms, where all signals are truncated to be the same
signals length. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

Decomposition model of simple NMF, where K = 2. Basis matrix
involves representative spectral patterns, and activation matrix
represents time-varying gains for each basis. . . . . . . . . . . . .

24

2.7

2.8

xviii

List of Figures

3.1

Permutation problem in FDICA and its solver, where N = M = 2.

32

3.2

Mixing and demixing model in IVA, where N = M = 2. . . . . . .

33

3.3

Spherically symmetric multivariate Laplace distribution, wheres̄i j,n can be considered as either real or imaginary part of si j,n andI = 2 (bivariate case). Two frequency components s1 j,n and s2 j,n
are uncorrelated but have mutual dependences, which is calledhigher-order correlation. . . . . . . . . . . . . . . . . . . . . . . . .

34

3.4

Principles of source estimation in (a) FDICA and (b) Laplace
IVA. Separation filter (demixing matrix) is optimized so that
estimated signals obey non-Gaussian source model. Whereas
FDICA assumes non-Gaussian source distribution p(si j,n ) for eachfrequency component, IVA assumes non-Gaussian multivariate
source distribution p(s j,n ) that has spherically symmetric property. 36

3.5

Circularly symmetric complex Gaussian distribution. Probability
does not depend on phase arg(qi jl ) but only depends on amplitude|qi jl | or power |qi jl | 2 because of circularly symmetric property. . .

37

3.6

Comparison of source models (variance structures) in (a) IVA and(b) ISNMF, where grayscale in each time-frequency slot indicatesscale of variance. IVA has uniform variance over frequency bins,
and all the frequency bins have the same activations (time-varying
gains), whereas ISNMF employs limited number of bases tocapture low-rank structure resulting in more flexible source model. 41

3.7

Relationship between IVA, ILRMA, and MNMF from viewpointof flexibility of spatial and source models. . . . . . . . . . . . . . .

56

3.8

Artificial source that consists of R bases. . . . . . . . . . . . . . . .

58

3.9

Artificial DOA with Gaussian distributions. . . . . . . . . . . . . .

60

3.10 Separation results of (a) first source and (b) second source forvarious numbers of bases. . . . . . . . . . . . . . . . . . . . . . . .

61

3.11 Separation results of (a) first source and (b) second source forvarious angles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63

3.12 Separation results of (a) first source and (b) second source forvarious variances. . . . . . . . . . . . . . . . . . . . . . . . . . . .

63

List of Figuresxix

3.13 Recording conditions of impulse responses (a) E2A and (b) JR2 for
two-source case. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

65

3.14 Average SDR improvements for female speech (dev1) with 1 m
microphone spacing and 130 ms reverberation time: (a) first
speaker and (b) second speaker. . . . . . . . . . . . . . . . . . . . .

67

3.15 Average SDR improvements for song ID4 with impulse response
E2A: (a) guitar and (b) synth. . . . . . . . . . . . . . . . . . . . . .

67

3.16 Cumulative singular values of each source spectrogram in dev1
female speech and song ID4 music, where all sources are truncatedto be the same signal length. . . . . . . . . . . . . . . . . . . . . . .

68

3.17 Average SDR improvements for female speech (dev1) with 1 m
microphone spacing, where reverberation time is (a) 130 ms and(b) 250 ms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71

3.18 Average SDR improvements for male speech (dev1) with 1 m
microphone spacing, where reverberation time is (a) 130 ms and(b) 250 ms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

72

3.19 Average SDR improvements for music signal song ID3 with
impulse response (a) E2A and (b) JR2. . . . . . . . . . . . . . . . .

73

3.20 Average SDR improvements for music signal song ID4 with
impulse response (a) E2A and (b) JR2. . . . . . . . . . . . . . . . .

74

3.21 Convergence of z1k from k = 1 to k = K in music signal case. . . .

75

3.22 SDR convergence for music signal song ID4 with impulse response
E2A: (a) guitar and (b) synth. . . . . . . . . . . . . . . . . . . . . .

77

3.23 Results of subjective scores obtained by Thurstone pairwise
comparison method, where 48 pairs of separated speech and 48
pairs of separated music signals are presented in random order to
14 examinees, who selected which signal they preferred fromthe viewpoint of total quality of separated sound. Scores show
relative tendency of selection. . . . . . . . . . . . . . . . . . . . . .

78

3.24 Probability of selection regarding difference between two subjective scores. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

79

3.25 Recording conditions of impulse responses (a) E2A and (b) JR2 forthree-source case. . . . . . . . . . . . . . . . . . . . . . . . . . . .

80

xx

List of Figures
3.26 Average SDR improvements for music signal song ID4 in threesource case with impulse response (a) E2A and (b) JR2. . . . . . .

81

3.27 Averaged SDR improvements averaged for music signal song ID1
with impulse response JR2 (reverberation time is 470 ms): (a)L = 5 and K = 10, (b) L = 10 and K = 20, (c) L = 20 and K = 40,(d) L = 30 and K = 60, (e) L = 40 and K = 80, and (f) L = 50 and
K = 100. Scores are averaged for two sources and 10 trials withdifferent pseudorandom seeds. . . . . . . . . . . . . . . . . . . . .

84

3.28 Averaged SDR improvements averaged for music signal song ID2
with impulse response JR2 (reverberation time is 470 ms): (a)L = 5 and K = 10, (b) L = 10 and K = 20, (c) L = 20 and K = 40,(d) L = 30 and K = 60, (e) L = 40 and K = 80, and (f) L = 50 and
K = 100. Scores are averaged for two sources and 10 trials withdifferent pseudorandom seeds. . . . . . . . . . . . . . . . . . . . .

85

3.29 Averaged SDR improvements averaged for music signal song ID3
with impulse response JR2 (reverberation time is 470 ms): (a)L = 5 and K = 10, (b) L = 10 and K = 20, (c) L = 20 and K = 40,(d) L = 30 and K = 60, (e) L = 40 and K = 80, and (f) L = 50 and
K = 100. Scores are averaged for two sources and 10 trials withdifferent pseudorandom seeds. . . . . . . . . . . . . . . . . . . . .

86

3.30 Averaged SDR improvements averaged for music signal song ID4
with impulse response JR2 (reverberation time is 470 ms): (a)L = 5 and K = 10, (b) L = 10 and K = 20, (c) L = 20 and K = 40,(d) L = 30 and K = 60, (e) L = 40 and K = 80, and (f) L = 50 and
K = 100. Scores are averaged for two sources and 10 trials withdifferent pseudorandom seeds. . . . . . . . . . . . . . . . . . . . .

87

3.31 Number of bases in power spectrogram of each source when
its cumulative singular value reaches 80% or 90%: (a) song ID1,
guitar (source 1) and vocals (source 2), (b) song ID2, guitar (source1) and vocals (source 2), (c) song ID3, violins synth. (source 1) andvocals (source 2), and (d) song ID4, guitar (source 1) and synth.
(source 2). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

88

List of Figuresxxi

3.32 Mixing system of each spectrogram slot when N = M = 2; (a) has alinear time-invariant mixing system and there is no reverberation;(b) has some leaked components from the previous frame becauseof reverberation. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

90

3.33 Algorithms of (a) conventional and (b) proposed methods (N = 2,
M = 4, and Q = 2), where subscripts for i and j are omitted. . . . .

92

3.34 Hierarchical clustering using correlation cor (N = 2, M = 4, andQ = 2), where all sets must have the same number of signals andsubscripts for i and j are omitted.. . . . . . . . . . . . . . . . . . .

93

3.35 Recording condition of impulse response used in experiment ofreverberant signals. . . . . . . . . . . . . . . . . . . . . . . . . . . .

94

3.36 Average SDR improvements for song ID1 used in experiment ofreverberant signals. . . . . . . . . . . . . . . . . . . . . . . . . . . .

97

3.37 Average SDR improvements for song ID2 used in experiment ofreverberant signals. . . . . . . . . . . . . . . . . . . . . . . . . . . .

97

3.38 Average SDR improvements for song ID3 used in experiment ofreverberant signals. . . . . . . . . . . . . . . . . . . . . . . . . . . .

98

3.39 Average SDR improvements for song ID4 used in experiment ofreverberant signals. . . . . . . . . . . . . . . . . . . . . . . . . . . .

98

4.1

Training and separation stages in (a) FSNMF and (b) SSNMF. . . . 104

4.2

Difference between (a) conventional and (b) proposed algorithms
of separation, where black components correspond to target
source and gray components correspond to interfering source.
Proposed method utilizes discriminative bases (F 0) that has
unique component of target source for separation, and target
source is synthesized using reconstructive bases (F) that hascomplete spectral components. . . . . . . . . . . . . . . . . . . . . 110

4.3

Spectral bases obtained from simple NMF: (a) C5 piano tone and(b) C6 flute tone. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

4.4

Spectral bases obtained from (4.16), where (a) is discriminativebasis F 0 of piano tone and (b) and (c) are the other bases in T . . . 114

xxii

List of Figures

4.5

Average SDR improvement for each mixture and each number of
iterations in (4.16). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

5.1

Example of SDR improvements of music source separation, where
FSNMF initialized by random values with various pseudorandom
seeds is performed. . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

5.2

Geometry of (a) optimal, (b), orthogonal, and (c) close bases,
where black dots indicate observed data points in positive orthant,gray area indicates cone defined by data points, broken lines
indicate edges of cone, f k denotes kth NMF basis, Φ = K = 2, andΨ = 10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

5.3

Assumption of proposed method, where nonnegative activations
are assumed to be independent of each other. . . . . . . . . . . . . 125

5.4

Convergence of cost function in (a) NICA and (b) ICA. . . . . . . . 129

5.5

Convergences of cost function in EUNMF, where NICA1–NICA3
are depicted in (a), ICA1–ICA3 are depicted in (b), and conventional methods are depicted in both. . . . . . . . . . . . . . . . . . 130

5.6

Convergences of cost function in KLNMF, where NICA1–NICA3
are depicted in (a), ICA1–ICA3 are depicted in (b), and conventional methods are depicted in both. . . . . . . . . . . . . . . . . . 131

5.7

Convergences of cost function in ISNMF, where NICA1–NICA3 aredepicted in (a), ICA1–ICA3 are depicted in (b), and conventional
methods are depicted in both. . . . . . . . . . . . . . . . . . . . . . 132

5.8

SDR improvement of supervised NMF for (a) vocals and (b) other. 135

5.9

Process flow of BSS based on ILRMA with NMF initialization
method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

5.10 Average SDR improvement of ID1 for each number of iterations in(4.16) with various NMF initializations. . . . . . . . . . . . . . . . 139
5.11 Average SDR improvement of ID2 for each number of iterations in(4.16) with various NMF initializations. . . . . . . . . . . . . . . . 139
5.12 Average SDR improvement of ID3 for each number of iterations in(4.16) with various NMF initializations. . . . . . . . . . . . . . . . 140

List of Figuresxxiii

5.13 Average SDR improvement of ID4 for each number of iterations in(4.16) with various NMF initializations. . . . . . . . . . . . . . . . 140
5.14 Average SDR improvement of ID5 for each number of iterations in(4.16) with various NMF initializations. . . . . . . . . . . . . . . . 141
5.15 Average SDR improvement of ID6 for each number of iterations in(4.16) with various NMF initializations. . . . . . . . . . . . . . . . 141

xxv

List of Tables

Categorization of typical existing techniques for audio source
separation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

Models of mixing system, spatial covariance, power spectrogram,and their optimization in each method . . . . . . . . . . . . . . . .

53

Estimated values of shape parameter κ so that kurtosis of F G isadjusted to 50 for each R . . . . . . . . . . . . . . . . . . . . . . . .

59

3.3

Music sources for two-source case . . . . . . . . . . . . . . . . . .

65

3.4

Experimental conditions . . . . . . . . . . . . . . . . . . . . . . . .

66

3.5

Experimental conditions used in Ozerov’s MNMF . . . . . . . . .

69

3.6

Averaged SDR improvements over various speech signals andsources with same recording conditions in two-source case . . . .

75

Averaged SDR improvements over various music signals andsources with same impulse response in two-source case . . . . . .

75

3.8

Music sources for three-source case . . . . . . . . . . . . . . . . . .

79

3.9

Averaged SDR improvements over various music signals andsources with same impulse response in three-source case . . . . .

80

3.10 Computational times (s) for separation of song ID1 with impulse
response E2A in three-source case . . . . . . . . . . . . . . . . . .

82

2.1

3.1
3.2

3.7

3.11 Experimental conditions used in analysis of optimal window length 82
3.12 Music sources used in experiment of reverberant signals . . . . .

95

3.13 Characteristics of each method used in experiment of reverberant
signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

95

3.14 Experimental conditions used in experiment of reverberant signals 96

xxvi

List of Tables

3.15 Computational times for separation of song ID3 used in experimentof reverberant signals (s) . . . . . . . . . . . . . . . . . . . . . . . .

99

4.1
4.2

Mixture ∆ with target and non-target sources . . . . . . . . . . . . 115(train)Sample sounds of non-target source N (train) for preparing Smix
. 115

5.1
5.2

Examples of computational time in each process (s) . . . . . . . . 133
Averaged SDR improvements over various speech signals andsources with same recording conditions, where these scores are
obtained by ILRMA w/o partitioning function with various NMF
initializations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
Averaged SDR improvements over various music signals andsources with same impulse response, where these scores are
obtained by ILRMA w/o partitioning function with various NMF
initializations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

5.3

1

1

Introduction

1.1

Background

Audio source separation is a technique for separating specific source signals
from a mixture signal and has been intensively studied for several decades. This
technique mainly focuses on mixed signals of speech, which can be used formany applications including speech enhancement, automatic speech recognition,
reduction of undesired background noise, and hearing aid systems. Many
applications assume the use of microphone array, which consists of several
synchronized microphones, for recording sound sources in a multichannel
format. In particular, when the number of microphones (channels) is equal to or
grater than the number of sources, which is called determined or overdeterminedsituation, blind source separation (BSS) technique is often applied to solve theaudio source separation problem. The benefit of using BSS is accessibility formany systems because BSS techniques do not require any information about
the recording environment, mixing system, or source locations. Audio source

2

Chapter 1. Introduction

separation for underdetermined or single-channel signals is also a potential
feature because the variety of its applications is wider than the source separation
techniques using microphone array. Since there is no satisfactory information in
the observations, this problem is tougher than the determined problem, andsome assumptions or training data for sources in a mixture are required to solve
the separation effectively.
The audio source separation for music signals has also attracted considerable
interest in recent yeas. This is also used for many applications, for example,
remixing of existing music, automatic music transcription, music search andrecommendation systems, sound field reproduction of a live concert, andeducation for musical instruments. When the music signal is recorded by
the microphone array (in a determined or an overdetermined situation), BSS
techniques can be applied similar to the speech source separation. However,almost all existing music signals are mixed down and provided in a stereo
format even though it includes more than two sources. For such signals, theunderdetermined or single-channel source separation techniques must be
applied to achieve the source separation. As the advantage in music source
separation, some prior information of the instrumental sources are often available,
for example, a music score, solo-played instrumental signals, and synthetic orsample-recorded sound dataset of specific musical instruments. For this reason,supervised approach of music source separation has also been an active area of
research.

1.2

Prior Work

The source separation problem includes several situations depending on its
assumptions and conditions, e.g., the number of recoding channels, recording
environment, presence of prior information and training data, characteristics of
mixed sources (stability and temporal or harmonic structure), and availabilityof other sensing data (multimodality). BSS for the determined observationis one of the most basic theory in these problems. In particular, independent
component analysis (ICA) [1] has been well studied since the mid 1990s not only inthe filed of acoustic signal processing but also in fields of brain science, radio1.2 Prior Work

3engineering, financial engineering, and image signal processing. Since a mixing
system of acoustic signals becomes a convolutive mixture due to effect of room
reverberation, and it is a more difficult problem than the instantaneous mixing
system, ICA has been developed mainly in the field of acoustic signal processing.
In the late 1990s, frequency-domain ICA (FDICA) [2] was established to deal withsuch convolutive mixture using Fourier transform. By the advent of independentvector analysis (IVA) [3, 4] in 2006, which is an extension of FDICA, a high-quality
blind speech separation was achieved.

On the other hand, a new theory of matrix decomposition called nonnegative
matrix factorization (NMF) [5] was invented in 1999. NMF can extract somemeaningful features in an observed data matrix as non-orthogonal basis vectors
and is applied to a spectrogram of an acoustic signal, which enables one to
extract specific spectral patterns of sources. On the basis of this theory, manytechniques for solving the underdetermined and single-channel audio source
separation have been proposed, and they have substantially been progressed
during the 2000s and 2010s. In particular, multichannel NMF (MNMF) [6],which deals with the multichannel signal and simultaneously models the
spectral patterns and spatial information (differences between channels) of therecording environment, results in a major contribution to the underdeterminedsource separation problem. Also, a supervised approach based on NMF [7]was introduced for the single-channel source separation, where training data
for the sources in mixture signal are directly exploited for preparing spectral
dictionaries of each source. In the very recent past, a method of discriminative
training for supervised NMF has actively been addressed [8, 9, 10] to improveseparation performance of supervised NMF. As another issue, the effective
initialization method for NMF has been investigated since NMF appeared,
particularly in the field of machine learning. Since an optimization algorithm in
NMF requires initial values for the variables and the result of decomposition
strongly depends on their initialization, the effective initialization is one of theattractive research topics.

4

Chapter 1. Introduction

1.3

Contributions

The objective of this dissertation is to develop an effective optimization
algorithm for the music source separation and to achieve satisfactory separation
performance. On the basis of the nature in music signals, such as low-rank
and discrete structure of the music spectrogram, NMF is suitably exploited formodeling these structures throughout this dissertation. The following two major
problems in music source separation are mainly addressed here:• Determined and overdetermined BSS• Single-channel semi-supervised source separation
For the former issue, an effective unified method of NMF and IVA is proposed as
an extension of traditional ICA algorithms. In addition, intriguing relationships
between IVA, MNMF, and the proposed method are theoretically revealed.
For the latter research topic, a new optimization algorithm for discriminative
training of supervised bases is developed, which can be applied even insemi-supervised situations. The algorithm approximately solves a conventional
bilevel optimization problem for obtaining discriminative bases. Furthermore,
an effective initialization scheme based on statistical independence of NMF
components is considered and applied to the two proposed methods describedabove.

1.4

Outline

Chapter 2 presents some basic preliminaries of audio source separation,
which include mathematical formulation, overview of conventional techniques,
motivations for developing new algorithms, and principle of NMF. In Chap. 3,
I propose a new effective algorithm for determined BSS task and discuss
the relationship between conventional and the proposed methods. Also, theextended algorithm for overdetermined and reverberant signals is proposed. The
efficacies of these methods are validated via experiments. Chapter 4 deals with asingle-channel semi-supervised source separation problem. After explaining the1.4 Outline

5

conventional approaches for training discriminative features in full-supervised
situation, a new discriminative training algorithm in semi-supervised situation
is developed. In Chap. 5 an initialization problem for general NMF optimization
problem is discussed, and an efficient initialization scheme based on statisticalindependence is proposed. Also, the efficacy of the proposed initialization for
NMF-based source separation task is experimentally confirmed. Finally, Chap. 6
concludes the whole contents and contributions in this dissertation.

72Preliminaries

2.1

Introduction

In this chapter, I provide some preliminaries, which are necessary for later
discussion. After giving a mathematical formulation of general source separation,
I explain an overview of existing conventional techniques and categorize themin terms of prior conditions (assumptions) for solving the source separation
problems. Next, motivations for developing new algorithms of music source
separation are clarified, then I introduce a key ingredient of this dissertation,
which is a matrix decomposition algorithm called NMF [11, 5, 12, 13, 14]. Finally,
I summarize the contents in this chapter.

8

Chapter 2. Preliminaries

2.2

Mathematical Formulation

2.2.1

Multichannel Mixing and Demixing Systems

We consider a mixing system of N sound sources and M channels (microphones)
as the following expression [15]:x̃m (τ) =

N
Õ

(2.1)

c̃nm (τ)

n=1=

N Lfilter
Õ
Õ−1
n=1

ãnm (τ)s̃n (τ − τ0),(2.2)

τ 0 =0where τ = 1, 2, · · · , τend is the integral index of discrete-time, n = 1, 2, · · · , N andm = 1, 2, · · · , M are the integral indices of sources and microphones, respectively,x̃m (τ) is the observed (recorded) time-domain signal of the mth microphone,c̃nm (τ) is the observed nth source signal obtained by the mth microphone, ãnm (τ)
is a filter coefficient of impulse response that models the acoustic path from thenth source to the mth microphone, Lfilter is the filter length of ãnm , and s̃n (τ) is thetime-domain signal of the nth original source. The mixing model can also be
expressed by vector form as follows:x̃(τ) =

N
Õ

(2.3)

c̃n (τ)n=1=

N Lfilter
Õ
Õ−1
n=1

ãn (τ)s̃n (τ − τ0),(2.4)

τ 0 =0where x̃(τ) = ( x̃1 (τ) · · · x̃ M (τ))T is the observed multichannel vector, c̃n (τ) =(c̃n1 (τ) · · · c̃nM (τ))T is the observed multichannel vector of nth source, which isoften called spatial source image [16], ãn (τ) = (ãn1 (τ) · · · ãnM (τ))T includes thefilter coefficients of acoustic paths from nth source to all the microphones, and ·T
denotes the vector or matrix transpose. The equations (2.2) and (2.4) show theconvolutive mixture of original N sources, which simulates the reverberant
mixture in a recording environment, and the filter length Lfilter corresponds tothe length of the reverberation time. In an anechoic case (Lfilter = 1), the mixing

2.2 Mathematical Formulation

9

Recording environment

SourcesAcoustic
paths

Microphones

Figure 2.1: Mixing system when N = 3 and M = 2, where only direct paths are
depicted as arrows.

system represented above becomes an instantaneous mixture in a time domain.
Figure 2.1 shows the mixing system, where only the direct paths between sources
and microphones are depicted as the arrows. Note that since ãnm is a filter withlength Lfilter , there are many other paths with delays due to the reflection in therecording environment. The objective of source separation is not the estimationof original dry source s̃n (τ) but the estimation of the separated observationc̃nm (τ) or c̃n (τ). The estimation problem of s̃n (τ) from c̃nm (τ) or c̃n (τ) is generally
called dereverberation [17, 18, 19], which is out of the scope and is not treated inthis dissertation.
The mixing system can be transformed via short-time Fourier transform(STFT) [20, 21] with an analysis window. In particular, when the length of theanalysis window is sufficiently longer than Lfilter , the convolutive mixture model
can be transformed into the instantaneous mixture model in the time-frequency

10

Chapter 2. Preliminaries

domain as
xi j,m =NÕ

ci j,nm =NÕai,nm si j,n,

(2.5)

ai,n si j,n,

(2.6)

n=1

n=1

N
Õ

N
Õ

or in the vector form
xi j =

ci j,n =n=1

n=1

where i = 1, 2, · · · , I and j = 1, 2, · · · , J are integral the indices of frequency bins
and time frames obtained via STFT, respectively, and xi j,m , ci j,nm , ai,nm , and si j,nare the complex-valued STFT coefficients of the time domain signals x̃m , c̃nm , ãnm ,and s̃i j,n , respectively. The vectors in (2.6) are defined as xi j = (xi j,1 · · · xi j,M )T ,ci j,n = (ci j,n1 · · · ci j,nM )T , ai,n = (ai,n1 · · · ai,nM )T , and si j = (si j,1 · · · si j,N )T , where
note that xi j , ci j,n , and ai j,n are the multichannel (M × 1) vectors and si j is themultisource (N × 1) vector. The vector ai,n is called array manifold vector [22]or steering vector [23, 24], which models the acoustic paths for nth source infrequency domain. We here consider that the steering vector ai,n is time-invariant,namely, all the spatial locations of sources and microphones do not change along
the time frame. The time-invariant instantaneous mixture in the time-frequency
domain leads to the following simple mixing representation:
xi j = Ai si j ,

(2.7)

where


Ai = ai,1 · · · ai,N© ai,11 · · · ai,N1 ª
­ .
.. ®
..
= ­ ..
.
. ®®
­«ai,1M · · · ai,N M ¬(2.8)

is called mixing matrix whose size is M × N. If the mixing matrix is a full-ranksquare matrix (M = N), we can define an inverse matrix of Ai that separates the2.2 Mathematical Formulation

11

sources in xi j as
yi j = Wi xi j ,

(2.9)

H
yi j,n = wi,nxi j ,(2.10)

where yi j = (yi j,1 · · · yi j,N )T is the estimated (separated) multisource vector,∗∗H© wi,1 ª © wi,11 · · · wi,1M ª­ . ® ­ .
.. ®
..
Wi = ­ .. ® = ­ ..
.
. ®®
­
® ­
∗
∗
H
«wi,N ¬ «wi,N1 · · · wi,N M ¬(2.11)

is called demixing matrix, wi,n = (wi,n1 · · · wi,nM )T is called demixing filter for nth
source, ·H denotes the Hermitian transpose, and ·∗ denotes the conjugate ofcomplex value. The estimated time-domain signal ỹn (τ) can be calculated byapplying inverse STFT with overlap-save [25] and the appropriate synthesis
window [26, 27] to yi j,n . The estimation problem of the separated signals ỹn (τ)
without knowing any information about the mixing system ãn (τ) is often called
BSS. In this problem, calibration of the microphones is not required because wecannot distinguish the frequency responses of the microphones and the effects
caused by the filter coefficients ãn (τ), namely, the difference of characteristics
between the microphones is absorbed by ãn (τ).

Hereafter, I denote the complex-valued spectrograms (I × J time-frequency
matrix) of, nth source signal, mth observed signal, and nth separated signal asSn ∈ CI×J , Xm ∈ CI×J , and Yn ∈ CI×J , respectively. Also, the third-order tensor ofsource, observed, and separated signals are denoted using sans-serif upright fontas S ∈ CI×J×N , X ∈ CI×J×M , and Y ∈ CI×J×N , respectively. Moreover, the third-ordertensor with a subscript denotes the sliced matrix or the fiber vector [28] in theoriginal tensor. For example, Xi:: , X: j: , and X::m denote the J × M, I × M, and I × J
sliced matrices in X, respectively. Also, Xi j: , Xi:m , and X: jm denote the M × 1, J × 1,
and I × 1 fiber (column) vectors in X, respectively.

12

2.2.2

Chapter 2. Preliminaries

Single-Channel Mixing System

For the case of M = 1, the mixing system can be defined asx̃(τ) =

N
Õ

(2.12)c̃n (τ)n=1=

N Lfilter
Õ
Õ−1
n=1

ãn (τ)s̃n (τ − τ0).

(2.13)

τ 0 =0In the time-frequency domain, (2.12) and (2.13) are transformed as
xi j =
=

N
Õ
n=1
N
Õ

ci j,n(2.14)

ai,n si j,n .

(2.15)

n=1

The single-channel source separation is more difficult problem than that for
multichannel signals because differences of amplitudes and phases between
channels cannot be utilized. Thus, typical single-channel source separation
techniques employ some strong constraints or a powerful a priori knowledge forachieving the objective.

2.3

Existing Conventional Techniques and Their Categorization

Audio source separation techniques can roughly be divided in terms of two
aspects; determinacy of mixing system and presence of external supervised
information. The former issue is related to the numbers of sources and channels(N and M). Since the source separation is an inverse problem, its difficulty directly
depends on these conditions. When the number of sources is equal or less than
the number of channels (N ≤ M), the source separation becomes determined or
overdetermined problem. On the other hand, when the number of sources
is grater than the number of channels (N > M), it is called underdetermined2.3 Existing Conventional Techniques and Their Categorization

13

Table 2.1: Categorization of typical existing techniques for audio source
separation
Situations

Blind

Determined or
overdetermined

FDICA
IVA

UnderdeterminedSparse coding
TFM
TDOA clustering
MNMF

Single-channel

TFM
REPET
KAM

Supervised
Spectral supervision
Spatial supervisionSound examples
Source activities
Source locations
Steering vectors
Fixed BF
Multichannel DNN
User-guided IVA
Robust adaptive BF
Adaptive BF
Spectral supervision
Spatial supervisionSound examples
Source activities
Source locations
Steering vectors
Multichannel DNN
User-guided MNMF
TFM
Dictionary-based MSM
Hybrid method
Spectral supervision
Spatial supervisionSound examples
Source activities
Source locations
Steering vectors
Supervised NMF
Informed NMF––
DAE

problem, which is tougher than the previous situation. The latter issue is related
to a presence of a priori knowledge for sources. For example, source locations(spatial positions) can be used for a multichannel source separation techniques.
For the music separation, scores are powerful prior information for estimating
source activities. Also, some instrumental sequences may be available in advanceto train the specific source spectra. Table 2.1 summarizes a categorizationof typical existing techniques for audio source separation techniques, where
they are categorized from the viewpoints of the problem determinacy andthe presence of external supervised information. I explain the details of thesetypical techniques below.
In the determined and overdetermined situation, ICA [1, 29, 30, 31, 32, 33, 34,35] has been the most successful algorithm for the source separation problem.
ICA utilizes the assumption of statistical independence between sources andestimates the demixing filters from the mixture observations in a fully blindfashion, which is called BSS. For BSS in audio signals, the sources are convolvedby the room reverberation as (2.4). Many ICA-based separation techniques
for delayed and convolved sources were proposed [36, 37, 38, 39, 40]. Also,
FDICA [2, 41, 42, 34, 43, 35] was developed as another approach for solving thesignal deconvolution using STFT. In FDICA, the demixing matrix in the frequency
domain, Wi , is estimated for the separation. This method is more stable andmore efficient compared with ICA deconvolution in the time domain because we
can easily treat the convolutive mixing system as the simple instantaneous

14

Chapter 2. Preliminaries

mixture (2.7) by applying STFT to the signals.
For supervised source separation in determined and overdeterminedsituations, both fixed and adaptive beamforming (BF) techniques have been
widely used. The source separation based on fixed BF assumes that the locationsof all sources and microphones are known, where the number of microphones
should be large for the accurate separation. Adaptive BF [44, 45, 46, 24] exploits
additional criteria, such as minimum variance and distortionless constraint, toadaptively reduce the background diffuse noise and extract the target sources. It
is revealed that the estimation in adaptive BF are essentially equivalent to that in
BSS based on FDICA [47] whereas FDICA is a blind (unsupervised) technique.
However, even if the locations of sources and microphones are known, BF-based
methods fail to accurately separate the sources when the source signals areconvolved by room reverberations, which particularly arise in audio recording.
This is because the directions of steering vectors spatially spread around thetrue source direction, and the distortionless constraint in adaptive BF cannot
ensure the quality of estimated sources. In some limited cases, the pretrained
steering vectors can be used for BFs as “strict” spatial supervision including
spatial spreads of each source, where the mismatch between the trained steeringvectors and an observation becomes further important problem. Robust adaptive
BF [48, 49, 50, 51, 52] was developed to improve the robustness against suchmismatch of steering vectors.
On the other hand, in the underdetermined situation, ICA has been used
to estimate not the demixing filters but the ICA bases, which is known as an
estimation of overcomplete bases [30, 53, 32]. This approach develops to new
methods such as a sparse coding [54, 55, 56, 57] and a time-frequency masking(TFM) [58, 59, 60], which are related to the methods in the machine learning or
pattern recognition field. These methods are based on the sparseness of signals,
which is a strong and practical assumption, and can solve the BSS problem evenin the underdetermined situation. However, since the sparseness assumption
does not always hold completely, the sound quality of separated signals ismarkedly degraded owing to the generation of artificial noise. Reticently, TFM is
utilized for the estimation of steering vectors [61, 62], and it is reported that thishybrid approach gives better suppression of diffuse noise in the overdetermined2.3 Existing Conventional Techniques and Their Categorization

15

situation. Clustering-based underdetermined source separation is developed for
multichannel observations [58, 63, 64, 65, 60, 66], where these techniques relyon time-difference-of-arrival (TDOA) estimations for each source at multiple
microphones. However, under severe reverberant conditions, TDOA estimations
become unreliable and these clustering-based techniques do not work well.
As another approach, NMF [11, 5, 12, 13, 14] has been introduced forsingle-channel BSS [67, 68, 69, 70, 71, 72, 73, 74, 75, 76]. NMF is a low-rankapproximation of an observed nonnegative matrix under nonnegative constraint,
and a small number of meaningful bases can be extracted from the observed
matrix. For acoustic signals, an amplitude or a power spectrogram is usedas the observed matrix in NMF, and the source separation is achieved byclustering the decomposed bases into each source. In order to utilize spatial
information of each source as a criterion of the clustering, NMF is extended tomultichannel model [77, 78, 79, 80, 6, 81, 82, 83, 84], which has a potential tosolve BSS even in the underdetermined situation. In addition, in recent years,
several new approaches based on a repetitive structure of music signals were
proposed for single-channel blind situation, which are called repeating pattern
extraction technique (REPET) [85, 86, 87, 88, 89, 90] and kernel additive model(KAM) [91, 92, 93, 94].
As supervised approaches for single-channel signals, supervised NMF [95, 7,
96, 97, 98] is the most reliable method, which directly utilizes training sequencesof each source for clustering the NMF bases. For the multichannel signals, ahybrid method of binary TFM and supervised NMF was proposed [99], where
the sources are first separated based on spatial cues, then supervised NMF iseffectively applied. A score-informed or user-guided approach with NMF or
MNMF [100, 82, 101, 102, 103] has also been a popular technique for providing
better separation exploiting an external information about the source activities
as supervision. This supervised approach is also applied to the determined BSS,
e.g., user-guided IVA [104]. In addition, multichannel sparse modeling (MSM)
with large-scale spatial dictionary [105, 106] is another approach for spatiallyinformed underdetermined source separation. In this method, acoustic paths
between many spatial locations and microphones are measured or calculated
in advance to prepare a large-scale spatial dictionary. The mixing system is

16

Chapter 2. Preliminaries

adaptively estimated using sparse modeling and the dictionary. The mismatch
between the dictionary and actual observation is also estimated for the robust
separation [106].
In recent years, underdetermined or single-channel source separation basedon deep neural network (DNN) or denoising autoencoder (DAE) has been a very
active research topic [107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117]. Many
literatures investigate the DNN-based source separation for speech signals [107,108, 109, 110, 112, 113, 114, 116, 117, 118] and music signals [111, 113, 115] so far.
In the method [118], the NMF decomposition for source modeling in MNMF isreplaced to DNN, resulting in developing a new multichannel DNN-based
audio source separation with spectral supervision. For spatial supervision,
multichannel features are exploited as an input data of DAE to train the spatial
information [119, 120, 114]. However, for the convolutive BSS in time domain,
thousands of coefficients in the separation filters must be trained even with
300 ms reverberation and 8-kHz sampling rate. Thus, to train the spatial featuresusing DNN with practical dataset may be almost impossible in a real situation.
In addition, DAE-based methods require many pairs of clean and contaminatedsource signals. This is a crucial problem in music source separation because wecannot prepare sufficient number of such pairs in a practical situation.
The main objective of this dissertation is to advance the audio source
separation techniques and develop more practical algorithms that give us better
separation performance. In addition, this dissertation mainly focuses on theseparation of music signals. For this reason, I will aim to only the following
situations:• Determined and overdetermined BSS• Single-channel semi-supervised source separation
The first issue, which will be addressed in Chap. 3, is a classical BSS problem,
where we do not assume the spatial supervision. As a motivation to treat BSS,
in a realistic situation, it is almost impossible to accurately train the whole
spatial information including source locations and spatial spreads caused byreverberations. The almost all audio recordings are always the only once,2.4 Motivations for Developing New Algorithms

17and completely the same situation (mixing system) is never reproduced. The
mismatch-robust approach such as robust adaptive BF is one of the possible
solution, but the blind estimation is most preferable if it can solve the problem.
For the determined and overdetermined observations, ICA-based approaches
have a potential to blindly solve the source separation and have been successful.
However, it usually aims to the separation of speech signals, and the musicsignals have not often been treated so far. This might be because the conventional
ICA-based BSS does not provide satisfactory performance for music signals. If
we achieve a better music separation in this situation, various applications using
microphone array can be realized, e.g., remixing or editing of live-recorded
music.
The second issue, which will be addressed in Chap. 4, is also important forthe music source separation because almost all music signals are provided as astereo format including more than two sources, namely, the underdeterminedsituation. In particular, source separation for single-channel signals is the mostbasic framework and can easily be applied for the other situations. Moreover,for music signals, the users can easily produce synthetic instrumental sounds
using musical instrument digital interface (MIDI) synthesizer or easily record
the actual instrumental sounds of their interesting source part. Therefore, thespectral supervised approach can also be applied for music source separation.
This dissertation is mainly focused on a semi-supervised method, which utilizes
a training sequence for only the target instruments. This approach is morepractical for many applications than preparing the training sequences for all thesources, namely, a full-supervised method.

2.4

Motivations for Developing New Algorithms

For determined BSS of speech signals, the conventional ICA-based methods arewell investigated and can achieve better separation performance. However,for music signals, the separation accuracy tends to be degraded. The main
reason is that typical music signal frequently contains co-occurring sources withmany overlapped spectra, which results in harmony with multiple instrumental
and vocal sources. In such signals, the statistical independence between the

18

Chapter 2. Preliminaries

Limited number of
discrete pitches

Limited number of
discrete note values
Figure 2.2: Discreteness in music signals, where score is beginning of Prelude(op. 28, no. 7) by Frédéric Chopin. Typical music consists of limited number of
discrete parts.

sources is weakened, and ICA-based separation often causes errors of capturing
each source. Also, the conventional ICA-based methods utilize only the source
distribution p(Yn ) for the source model, and it has been empirically determined(e.g., Laplace distribution as a source distribution of speech signals [121]) without
any information about the specific time-frequency structure of each source.
Since BSS is the inverse problem, the separation performance directly depends
on the accuracy of the source distribution p(Yn ) or the source model. We can
expect that if we employ stricter source model in ICA algorithm, the separation
performance for music signals would be improved.
In this dissertation, I focus on a property of music signals to effectively
model their spectrograms (source models). Typical music signals include many
reiteration of similar or the same instrumental timbres, melody patterns, chords,
harmonies, and refrains with a stable rhythm. Also, the music signals typicallyconsist of limited number of components, for example, steady musical tones,
discrete pitches, and discrete notes as shown in Fig. 2.2. This property meansthat the spectrogram of a music signal tends to be a low-rank matrix comparedwith a speech spectrogram. Here, I explain this property with some example
music and speech signals.
Figures 2.3–2.6 show the power spectrograms of drums, guitar, vocals,
and male speech signals, respectively. These audio signals are obtained fromSiSEC2011 dataset [122]. The power spectrograms are calculated by STFT using2.4 Motivations for Developing New Algorithms

19

Figure 2.3: Example of power spectrogram of drums sound obtained from“another dreamer-the ones we love” in SiSEC2011 dataset, where grayscale
indicates spectral power and white is stronger than black.

Figure 2.4: Example of power spectrogram of guitar sound obtained from“another dreamer-the ones we love” in SiSEC2011 dataset, where grayscale
indicates spectral power and white is stronger than black.

20

Chapter 2. Preliminaries

Figure 2.5: Example of power spectrogram of vocals sound obtained from“another dreamer-the ones we love” in SiSEC2011 dataset, where grayscale
indicates spectral power and white is stronger than black.

Figure 2.6: Example of power spectrogram of male speech sound obtained
from “dev1_male3_src” in SiSEC2011 dataset, where grayscale indicates spectral
power and white is stronger than black.

2.4 Motivations for Developing New Algorithms

21

Figure 2.7: Example of cumulative singular values of music and speech
spectrograms, where all signals are truncated to be the same signals length.

256-ms-long Hamming window with 192-ms-long overlap. Also, Fig. 2.7 showstheir cumulative singular values. It is obviously confirmed that the drums andguitar signals contain many similar or the same spectral patterns compared withthe vocals and speech signals, which results in reducing the rank of the power
spectrogram as shown in Fig. 2.7. Therefore, if we can extract a limited numberof significant spectral patterns from the low-rank music spectrogram as thebases, the signal can effectively be modeled using such bases as the “low-rankapproximation.” The approximated model spectrogram of each source candirectly be used as the strict source model in conventional ICA algorithm.
For this reason, I consider that NMF decomposition is certainly suitable
for modeling the spectrogram of music signals because we can blindly extract
significant nonnegative bases and their coefficients, which correspond to thefrequently appearing spectral patterns and their time-varying gains. Indeed,
many tasks related to the music signals have been addressed using NMF,
e.g., [123, 124, 79, 125, 71, 126, 127, 100, 128, 129, 130, 131, 132, 98, 99, 133].
Therefore, the unification of ICA-based source separation and the NMF-basedsource modeling will provide better performance for the multichannel musicsource separation task.

22

Chapter 2. Preliminaries

As another issue, the conventional single-channel source separation based onsemi-supervised or full-supervised NMF (SSNMF or FSNMF) does not always
provide satisfactory separation although it utilizes the training sequence fortarget source or all the sources. This is because the pre-trained NMF bases(supervised bases) may represent not only the relevant source but also different
sources when the sources share similar or the same spectral patterns. Even if thesupervised bases are prepared for all the sources in FSNMF, this problem mayoccur, resulting in the degradation of separation performance. As a means of
avoiding this problem, several techniques for training discriminative supervisedbases were proposed [134, 135, 8, 9, 136, 137, 10]. However, these methods are
only applicable to the full-supervised situation, and no one investigates thediscriminative training for SSNMF.
In this dissertation, I contribute the above-mentioned issues, namely, aunified algorithm of ICA-based BSS and NMF source model for the determined
and overdetermined BSS problems in Chap. 3 and a discriminative NMF basis
training for single-channel semi-supervised source separation problem in
Chap. 4. Since both contributions are based on NMF algorithm, in the next
section, I provide a basic principle of NMF as a preliminary for the later chapters.

2.5

Basic Principle of NMF

NMF is an unsupervised data decomposition technique and a latent variable
analysis that can be used for, e.g., topics recovery, feature learning, clustering,
temporal segmentation, filtering and source separation, and coding. There has
been a variety of successful applications: text mining [138, 139], image signal
processing (e.g., object discovery [140], face recognition [141], tagging [142],
denoising and inpainting [143], texture classification [144], hashing [145], andwatermarking [146]), feature extraction [147, 148] and artifact rejection [149, 150]
for electroencephalography signals, and even in the bioinformatics field [151, 152,
153]. In audio signal processing, source separation based on NMF for both speech
and music signals is well-studied and still a growing research topic as describedin Sect. 2.3. Moreover, audio denoising [154, 155], audio inpainting [156, 157],
compression [158], and music transcription [123, 159, 160, 161, 125, 162, 163] are2.5 Basic Principle of NMF

23

also investigated with NMF.
NMF approximately decomposes an observed nonnegative data matrix∆ ∈ RΦ×Ψ
into a product of two nonnegative matrices F ∈ RΦ×K
and G ∈ RK×Ψ
as
≥0
≥0
≥0
∆ ≈ ∆ˆ = F G,
δφψ ≈ δ̂φψ =K
Õ

(2.16)
fφk gkψ,(2.17)

k=1

where F = ( f1 · · · f K ) is called basis matrix that involves NMF bases f k as
the column vectors, G = (g1 · · · gK )T is called activation matrix that involvescoefficient vectors g k for each basis as the row vectors, δφψ , δ̂φψ , fφk , and gkψ areˆ F, and G, respectively, Φ and Ψ are
the nonnegative entries of the matrices ∆, ∆,the numbers of rows and columns of the matrix ∆, respectively, K is the numberof bases, and φ = 1, 2, · · · , Φ, ψ = 1, 2, · · · , Ψ, and k = 1, 2, · · · , K are the integral
indices for the rows, columns, and bases. Since the main objective of NMF is to
reduce the dimensionality and find a low-rank representation with nonnegative
parts, the number of bases K should be set to a small value as K  min(Φ, Ψ).
The basis vectors can be considered as nonnegative parts representing theobserved data, where only linear combinations with nonnegative coefficients areallowed. As a result, the obtained bases and activations can often be interpreted
intuitively, and they should be the latent and meaningful features in the observed
data matrix ∆. For audio signals, a magnitude or power spectrogram obtained
via STFT is often used as the input nonnegative data matrix ∆. In this case, thebases vectors correspond to the frequently-appearing spectral patterns in the
spectrogram, and the activation vectors represent time-varying gains of each
spectral pattern. Figure 2.8 shows an example of NMF decomposition for audio
spectrogram data, where ∆ includes two harmonic tones with some overlaps.
The obtained basis vectors in F correctly capture the spectra of each tone, andtheir gains are represented as the activation vectors in G. In this decomposition,
the two rank-1 matrices, f1 g1T and f2 g2T , represent each tonal spectrogram, andthey are superposed for approximately representing the observed spectrogramas ∆ ≈ f1 g1T + f2 g2T . Therefore, the source separation based on NMF can beconsidered as a clustering problem of K bases into N sources.

Chapter 2. Preliminaries

Frequency

Frequency

Amplitude

24

Time

Activation matrix(Time-varying gain)

Time

Amplitude

Data matrix(Spectrogram)

Basis matrix(Spectral patterns)

Figure 2.8: Decomposition model of simple NMF, where K = 2. Basis matrix
involves representative spectral patterns, and activation matrix represents
time-varying gains for each basis.

In NMF, we find the decomposed bases and their activations by minimizationof the distance or divergence between the observed data ∆ and an approximatedmodel ∆ˆ = F G as follows:ˆ s.t. fφk , gkψ ≥ 0 ∀φ, ψ, k,
min D(∆k ∆)F,G(2.18)

where D(·k·) denotes an arbitrary divergence (dissimilarity). Various divergences
have been utilized, e.g., the squared Euclidean distance (EU distance), generalized
Kullback–Leibler divergence (KL divergence) [164], and Itakura–Saito divergence(IS divergence) [165]. As a generalized criterion, β divergence [166] D β (·k·) isalso introduced into NMF:
β
β
β−1

Õ © δφψδ̂φψ δφψ δ̂φψ ª



+
−
­
® (β ∈ R\{0, 1} )


(β
β
−
1)
β
β
−
1


φ,ψ «

¬



δφψ
Õ ©

ª
ˆ =
D β (∆k ∆).
+ δ̂φψ − δφψ ®
(β = 1)­δφψ log

δ̂φψ

φ,ψ «

¬




Õ
δ
δ

φψ
© φψ
ª


− log
− 1®
(β = 0)­ φ,ψ δ̂φψ
δ̂φψ

«
¬

(2.19)

2.5 Basic Principle of NMF

25

When β = 2, β = 1, and β = 0, the β divergence becomes identical to EU distance,
KL divergence, and IS divergence, respectively. In the context of audio source
separation, KL divergence and IS divergence are often used because they usually
give us a better separation performance than EU distance.

Since the simultaneous minimization of F and G based on (2.18) is notconvex, regardless of the type of divergence, the closed-form solution for (2.18)
have yet to be found. However, for the alternating optimization of F and G,
efficient iterative update rules have been derived [5, 12, 167, 168] using theauxiliary function technique [12], which is the generalized optimization approachof expectation-maximization (EM) algorithm [169] and is also known as the
majorization-minimization algorithm [170, 171, 172]. For β-divergence-based
NMF, the following multiplicative update (MU) rules efficiently minimize thevalue of cost function (2.18):
 β−2 # ϕ(β)Í
0 gk 0ψ
δ
g
f
0
φψ
kψ
φk
ψ
k
fφk ← fφk
,
 β−1
Í
Í
0
0
g
f
g
0
ψ kψ
k φk k ψ"Í β−2 # ϕ(β)Í
φ fφk δφψ
k 0 fφk 0 gk 0 ψ,
gkψ ← gkψ
 β−1
Í
Í
φ fφk
k 0 fφk 0 gk 0 ψ"Í(2.20)(2.21)

where ϕ(β) is given by
1






2 − βϕ(β) = 1



1



β−1

(β < 1)(1 ≤ β ≤ 2) .
(β > 2)(2.22)

26

Chapter 2. Preliminaries

The update rules (2.20)–(2.21) can also be rewritten in a matrix form as follows:
h
i
.ϕ(β)
.(β−2)



(F
∆
◦
G)
GT 




h
i
F←F◦
,

 (F G).(β−1) GT hi .ϕ(β).(β−2) 

T





 F ∆ ◦ (F G)
h
i
G←G◦
,


 F T (F G).(β−1) (2.23)(2.24)

where ◦ and the quotient symbol for matrices denote the Hadamard product(entrywise multiplication) and entrywise division, respectively, and the dotted
exponent for matrices denotes entrywise exponent. The iteration of these update
rules ensures the monotonic decrease of the cost function. In (2.20) and (2.21),
the initial values for fφk and gkψ must be given for all φ, ψ, and k. This fact meansthat the initial values influence the decomposed solution because there exist
many local minimum solutions. Since all the results in NMF-based applications
including the methods treated in Chaps. 3 and 4 directly depend on these initialvalues, the effective initialization method for NMF is one of the big problems.
This issue will be treated in Chap. 5.

2.6

Summary

In this chapter, a mathematical formulation for general source separation
problems was introduced. Next, typical and popular source separation methods
were reviewed. In addition, motivations for developing new source separation
algorithms were explained. Finally, a key ingredient of this dissertation,
NMF, was introduced with theoretical and mathematical principles. In thefollowing chapters, the main contribution of this dissertation will be discussed;
determined and overdetermined BSS will be addressed in Chap. 3, single-channel
semi-supervised source separation will be treated in Chap. 4, and a better
initialization for NMF will be addressed in Chap. 5.

27

3

Determined and Overdetermined
Blind Source Separation Based on
Independent Low-Rank Matrix
Analysis

3.1

Introduction

In this chapter, I address the determined BSS problem and propose a new efficientalgorithm that unifies conventional ICA-based BSS and NMF-based source
model. First, I introduce some basic principles of ICA, FDICA, IVA [3, 4, 173],
and NMF based on IS divergence (hereafter referred to as ISNMF) [125], which
are necessary for the main contribution in this chapter. Next, a new efficient
BSS technique is described with its motivations and optimization algorithms.

Chapter 3. Determined and Overdetermined Blind Source Separation Based28
on Independent Low-Rank Matrix Analysis
After giving an explanation of the relationship between IVA, MNMF, and the
proposed method, the efficacy of the proposed method for BSS task is validated
via experimental analysis and comparison. In addition, further extension of the
proposed method for overdetermined BSS problem is addressed. Finally, thewhole contents in this chapter are summarized.

3.2

Basic Principles of ICA, FDICA, IVA, and ISNMF

While many text books for ICA have been published [30, 31, 32, 33] so far, I
briefly introduce some basic principles of ICA and FDICA. Also, IVA, and ISNMF
are explained with their motivations. These algorithms and their statisticalbackgrounds are important and necessary for introducing the motivations of the
proposed method.

3.2.1

ICA and FDICA

ICA, which is sometimes regarded as synonymous with BSS, relies on nonGaussianity, namely, the independent sources s̃n (τ) in the observed mixture x̃m (τ)
are inherently generated from non-Gaussian distributions. This assumption
makes the BSS problem solvable because the mixture of sources tends towards a
Gaussian distribution even if the original sources themselves obey non-Gaussian
distributions, which is known as the central limit theorem. Therefore, if wedetermine the statistical model of separated signals ỹn (τ) as p( ỹn ) and if M = N(determined) and Lfilter = 1 (instantaneous mixture), we can basically estimate
the original ỹn (τ) from the mixture x̃(τ) by ICA, while the sources must truly
obey the non-Gaussian distributions.
Let us assume that M = N, Lfilter = 1, and the mixing system is described as
x = As

(3.1)

≡ Ay,

(3.2)

where tildes and the time index τ are omitted for the simplicity, and all the3.2 Basic Principles of ICA, FDICA, IVA, and ISNMF

29

variables are real values. The probability p(x) can be written using Jacobian∂y= | det A−1 |,
∂x

(3.3)

as
p(x) = | det A−1 | · p(y)Ö
= | det W | ·p(yn ),

(3.4)
(3.5)

n

Î
where W = A−1 , and p(s) = n p(sn ) are obtained by assuming mutual independence between yn for all the sources. Since the signals x, s, and y have τend
samples, the likelihood of W is given byL(W ) =

ÖÖ
τp(wnT x(τ))| det W |,

(3.6)

n

where wn is a row vector in W . Also, the log-likelihood function is obtained aslog L(W ) =ÕÕτp(wnT x(τ)) + τend log | det W |.

(3.7)

n

By replacing the sum of τ to the expectation operator E[·], we can rewrite (3.7) as
1
τend

"
log L(W ) = E#
Õp(wnT x) + log | det W |.

(3.8)

n

The maximum likelihood (ML) estimation in ICA can be obtained by differentiating (3.8) with respect to W . The gradient of (3.8) can easily be calculated
as
 −1

 
1 ∂ log L(W )= E S(W x)x T + W T ,
τend
∂W

(3.9)

where
S(y) =∂ log p(y)∂y(3.10)

Chapter 3. Determined and Overdetermined Blind Source Separation Based30
on Independent Low-Rank Matrix Analysis
is called the score function or nonlinear function in the context of ICA optimization.
From (3.9), the steepest gradient descent for ICA can be derived as

W ← W + η E S(W x)x


T





+ W

T

 −1 

,(3.11)

where η is a stepsize parameter. This algorithm is often called Bell–Sejnowski
algorithm, which is firstly derived from another ICA principle called Infomax
approach [174]. The algorithm (3.11) is extended to a natural gradient [175, 33]
method that is based on a geometric structure in the parameter space, as W ← W + η E S(W y)y T + I W .

(3.12)

This algorithm is more efficient than (3.11) because the gradient for the parameters
are defined in Riemannian metric, and the inverse calculation of W is omitted.
In addition, for the minimization of (3.9), fast and stable update rules called
iterative projection (IP) based on the auxiliary function technique have been
proposed [176]. Note that there exists signal permutation and scaling ambiguities
in ICA solution, namely,y ← ΛΥy(3.13)is also a solution for any permutation matrix Υ and diagonal matrix Λ. This isbecause the solution in ICA is only based on a statistic source model p( ỹn ) and its
independence between the sources. In many ICA applications, the source modelp( ỹn ) or its score function is empirically set to the appropriate ones, e.g., Laplace
distribution for super-Gaussian sources.
For the BSS problem of acoustic signals, the sources are convolved withreverberation in a recording environment as represented in (2.4), and it becomesa deconvolution problem. Many ICA-based deconvolution techniques forsolving BSS problem in time-domain were proposed [36, 37, 38, 39, 40]. However,
the estimation of inverse filters in the time domain is still a tough problem
because the number of parameters drastically increases when the filter length
Lfilter becomes large. Instead of solving the time-domain deconvolution,

3.2 Basic Principles of ICA, FDICA, IVA, and ISNMF

31

FDICA [2, 41, 42, 34, 43, 35] was proposed. In this approach, the instantaneous
mixture in the frequency domain is assumed as (2.6). When the mixing filter
length Lfilter is much shorter than the length of analysis window in STFT,
this assumption becomes valid. In FDICA, the simple ICA is independently
applied in each of frequency bins, and the frequency-wise demixing matrix Wi is
estimated for the separation. The optimization algorithm for FDICA is identicalto that for simple ICA with complex-valued signals [121, 177]. However, in
FDICA, the permutation ambiguity of separated signals becomes a serious
problem because the separated components in each frequency bin must be
correctly aligned as shown in Fig. 3.1. This alignment problem is often called
permutation problem, and many criteria for solving this ambiguity have been
proposed [178, 179, 180, 181]. The popular permutation solver is using directionof arrival (DOA) of each source [178], where DOA can be calculated from theestimated demixing filter wn . In [180], the correlations between frequency bins
are simultaneously exploited with DOA information for solving the permutation
problem. The scaling ambiguity should also be solved after the estimation of Wi
in FDICA. The simplest way for recovering signal scales is projecting them to theobserved signals, asĉi j,n ← Wi−1 (en ◦ y) ,(3.14)

where ĉi j,n = (ĉi j,n1 · · · ĉi j,nM )T is an estimated source image whose scale is fitted
to the observed signals at each microphone, and en denotes the M × 1 unit vector
with the nth element equal to unity. This calculation is called the back projection
technique [179].

3.2.2

IVA

IVA [3, 4, 173] is a multivariate extension of FDICA and can solve the BSS
problem while avoiding the permutation problem. In IVA, we assume themultivariate source vector s j,n , observed vector x j,m , and separated vector y j,n ,Chapter 3. Determined and Overdetermined Blind Source Separation Based32
on Independent Low-Rank Matrix Analysis
Separated signal 1

Sourcesignal 1

Observed
signal 1

ICA
Sourcesignal 2

Time

Permutation
solver

Separated signal 2

Observed
signal 2

Figure 3.1: Permutation problem in FDICA and its solver, where N = M = 2.

which consist of all the frequency bins, as
s j,n = (s1 j,n · · · s I j,n )T,(3.15)

x j,m = (x1 j,m · · · x I j,m )T,(3.16)

y j,n = (y1 j,n · · · y I j,n )T .

(3.17)

Figure 3.2 shows the mixing and demixing model in IVA, where N = M = 2.
In IVA, all the source, observed, and separated signals are represented as
frequency vector variables, whereas FDICA independently models each of
the frequency components resulting in the permutation problem. In addition,higher-order correlations between the frequency components in each source (orseparated) vector are introduced by assuming spherically symmetric multivariate
source distributions p(s j,n ) ≈ p(y j,n ) = p(y1 j,n, · · · , y I j,n ), where the sphericallysymmetric property means that the distribution is a function of only the norm ofmultivariate vector variable, i.e., p(y j,n ) = f (k y j,n k).
In the literature [3, 4, 173], a spherically symmetric multivariate Laplace
distribution [182, 183] was exploited as a super-Gaussian source distribution for3.2 Basic Principles of ICA, FDICA, IVA, and ISNMF
SourcesignalMixing matrix

Observed
signal33

Demixing matrix

Separated
signal

…

…

…

…

…

Figure 3.2: Mixing and demixing model in IVA, where N = M = 2.

modeling speech sources. This distribution is shown in Fig. 3.3 and is defined asvtÕ yi j,n 2 ª©
p(s j,n ) ≈ p(y j,n ) = ρ exp ­−
®,
ri,n
i
«
¬

(3.18)

where ρ is a normalization term and ri,n is the variance, which determines thesignal scale of yi j,n . Since the source distribution has a spherically symmetricproperty, higher-order correlations between the frequency components ineach source are assumed, which results in avoiding the permutation problem.
Hereafter, IVA based on the source distribution (3.18) is referred to as Laplace IVA.
From the generative source model p(s j,1, · · · , s j,N ) ≈ p(y j,1, · · · , y j,N ) andthe demixing system (2.9), p(x j,1, · · · , x j,M ) can be obtained by multiplyingp(y j,1, · · · , y j,N ) by the Jacobian∂(y j,1, · · · , y j,N ) Ö| det Wi | 2 ;=∂(x j,1, · · · , x j,M )i(3.19)

note that the Jacobian for a complex-valued variable is the square of the Jacobian
for a real-valued variable [177]. Therefore, the likelihood function L(W ) of theChapter 3. Determined and Overdetermined Blind Source Separation Based34
on Independent Low-Rank Matrix Analysis

0.15

0.10

0.05

0
4
2
0
-2
-4

0

-2

-4

2

4

Figure 3.3: Spherically symmetric multivariate Laplace distribution, where s̄i j,ncan be considered as either real or imaginary part of si j,n and I = 2 (bivariatecase). Two frequency components s1 j,n and s2 j,n are uncorrelated but have mutual
dependences, which is called higher-order correlation.

parameter set W = {Wi |i = 1, · · · , I} is given as
L(W ) =

Öp(x j,1, · · · , x j,M |W )(3.20)

j

"
=

Ö

#
p(y j,1, · · · , y j,N ) ·Öj(3.21)i("=| det Wi | 2

#

Ö

Ö

j

n

p(y j,n ) ·)Ö| det Wi | 2 ,(3.22)iÎ
where p(y j,1, · · · , y j,N ) = n p(y j,n ) is obtained by assuming mutual independence
between y j,n for all the sources. The negative log-likelihood function can be3.2 Basic Principles of ICA, FDICA, IVA, and ISNMF

35

calculated as− log L(W ) = −Õ

log | det Wi | 2 −i, j= − 2JÕlog p(y j,n )(3.23)

Õ(3.24)j,nÕ

log | det Wi | +i

G(y j,n ),j,nwhere G(y j,n ) = − log p(y j,n ) is called the contrast function, which depends on
H x , the separated
the source distribution p(y j,n ). Note that since yi j,n = wi,nij
signal y j,n includes the optimization variable Wi . The ML estimation based on(3.24) is equivalent to the well-known estimation [29, 32] that maximizes theindependence between all the sources with the KL divergence DKL as follows:!
Õ

DKL p(y j,1, · · · , y j,N )kÖp(y j,n )njp(y j,1, · · · , y j,N )Î
d y j,1 · · · d y j,N (3.25)
n p(y j,n )jÕÕ= const. − 2J
log | det Wi | +G(y j,n ).
(3.26)=

Õ∫p(y j,1, · · · , y j,N ) logij,nOn the basis of the source distribution (3.18), the contrast function G(y j,n ) andthe cost function in Laplace IVA can be obtained as follows:G(y j,n ) = − log ρ + k y j,n k2,ÕÕ− log L(W ) = const. − 2J
log | det Wi | +k y j,n k2,
i(3.27)(3.28)

j,n

where k · k2 denotes the `2 norm. Also, the variance is set to ri,n = 1 for all i and nbecause the scales of separated signals cannot be determined by ICA or IVA,
and they can be recovered by the back-projection technique (3.14) after the
separation. Similar to the simple ICA, IP-based efficient update rules have been
proposed [184, 185, 186].
Figure 3.4 shows the principles of source estimation in FDICA and Laplace
IVA. The demixing matrix Wi is optimized so that the estimated signals yi j,nobay the assumed non-Gaussian source model. Whereas FDICA assumes theChapter 3. Determined and Overdetermined Blind Source Separation Based36
on Independent Low-Rank Matrix Analysis

Estimated
signal

Separation
filter

STFT

Time

Optimize separation filter so that estimatedsignals obey non-Gaussian source model

Time

(a)

Frequency

Vector random variables
Estimated
signal

Separation
filter

STFT

Empirical
distribution of Non-Gaussian
estimated signal source modelTime

Frequency

Observed
signalEmpirical
distribution of Non-Gaussian
estimated signal source model

Frequency

Observed
signalFrequency

Scalar random variables

Optimize separation filter so that estimatedsignals obey non-Gaussian source model

(b)

Time

Figure 3.4: Principles of source estimation in (a) FDICA and (b) Laplace IVA.
Separation filter (demixing matrix) is optimized so that estimated signals obey
non-Gaussian source model. Whereas FDICA assumes non-Gaussian source
distribution p(si j,n ) for each frequency component, IVA assumes non-Gaussian
multivariate source distribution p(s j,n ) that has spherically symmetric property.

non-Gaussian source distribution p(si j,n ) for each frequency component, IVA
assumes the non-Gaussian spherically symmetric source distribution p(s j,n ) for
the frequency vector variables.

3.2.3

ISNMF

When we apply NMF to an acoustic signal, the power spectrogram obtained via
STFT is considered as an observed nonnegative matrix and can be decomposed
into two nonnegative matrices as|D| .2 ≈ T V,

(3.29)

3.2 Basic Principles of ICA, FDICA, IVA, and ISNMF

37

0.15

0.10

0.05

0
4
2
0
-2
-4

-4

-2

0

2

4

Figure 3.5: Circularly symmetric complex Gaussian distribution. Probability
does not depend on phase arg(qi jl ) but only depends on amplitude |qi jl | or
power |qi jl | 2 because of circularly symmetric property.

where D ∈ CI×J is a complex-valued spectrogram, the absolute | · | for matrices
L×J
is thedenote the entrywise absolute value, T ∈ R≥I×L
0 is the basis matrix, V ∈ R ≥ 0
activation matrix, and L is the number of bases.
Since the transformation of complex spectrograms into power (or amplitude)
spectrograms is nonlinear, power spectrograms are non-additive, namely, thepower spectrum of the sum of two waveforms is not equal to the sum of thepower spectra of the two waveforms. This implies that decomposing a power
spectrogram into the sum of additive components does not necessarily lead to an
appropriate decomposition of the audio signal. However, the decomposition
based on ISNMF applied to the power spectrogram ensues such spectral
additivity in the expectation sense, which has been given by Févotte, et al. [125].
Let us assume that L complex-valued spectrograms qi j1, · · · , qi j L are generated
from circularly symmetric complex Gaussian distribution [187], which areChapter 3. Determined and Overdetermined Blind Source Separation Based38
on Independent Low-Rank Matrix Analysis
independently defined in each time-frequency slot as follows:!
|qi jl | 21
p(qi jl ) =exp −,πri jlri jl(3.30)

where l = 1, · · · , L is the integral index of L components and ri jl is thenonnegative variance of each distribution. Figure 3.5 shows the circularly
symmetric complex Gaussian distribution. Since the distribution has a circularly
symmetric property in the complex plane, the probability does not depend on
the phase arg(qi jl ) and only depends on the amplitude |qi jl | or power |qi jl | 2 . Notethat the variance ri jl corresponds to the expectation value of the power spectrum|qi jl | 2 , namely, ri jl = E[|qi jl | 2 ]. When the variance ri jl is large, the distribution
becomes wider, and the complex-valued spectrum qi jl with a large power caneasily be generated, while the phase of qi jl is always uniformly distributed. In
addition, if we assume that the observation di j , which is the complex-valuedÍ
entry of D, is the sum of the components qi jl , namely, di j = l qi jl , the following
generative model can also be assumed because of the reproductive property incomplex Gaussian distributions:p(D) =

Öp(di j )(3.31)i, j!
Ö 1|di j | 2
=
exp −
,
πri j
ri j
i, j

(3.32)

Í
where ri j = l ri jl . This fact means that the additivity of power spectra |qi jl | 2 isÍheld only in the expectation sense, and the superposed component di j = l qi jl isalso assumed to obey the circularly symmetric complex Gaussian distribution
Íwith the superposed variance ri j = l ri jl . In [188], the model of power
spectrogram (3.32) is extended to a maximum a posteriori framework using
inverse Gamma prior for the variances.

The likelihood function of T and V can be obtained as follows by putting3.2 Basic Principles of ICA, FDICA, IVA, and ISNMF

39

ri jl = til vl j ;L(T, V ) = p(D|T, V )(3.33)
!

Ö

=

i, j

π

Í

|di j | 2
1
exp − Í,l til vl jl til vl j

(3.34)

where til and vl j are the nonnegative entries of T and V , respectively. The
negative log-likelihood function is− log L(T, V ) =Õ

log π + log

Õ

i, j

l

!
|di j | 2
til vl j + Í.
l til vl j

(3.35)

It is clear that the ML estimation based on (3.35) is equivalent to the minimizationof IS divergence DIS [165] between |D| .2 and T V :

DIS |D| .2 kT V =|di j | 2
|di j | 2
Í
− log Í−1l til vl jl til vl jÕi, j= const. +

Õ
i, j

!

!
Õ
|di j | 2Í+ log
til vl j .
l til vl jl(3.36)(3.37)

Thus, when ISNMF is applied to the observed power spectrogram |D| .2 , it isassumed that di j follows the generative model (3.32) and the components qi jl
are mutually independent. The multiplicative update rules for T and V thatminimize (3.35) or (3.37) are given by [167]vutÍtil ← til2
j |di j | vl j

 −2

l 0 til 0 vl 0 j −1ÍÍj vl jl 0 til 0 vl 0 jvutÍvl j ← vl jÍ2i |di j | til

Í

 −2

l 0 til 0 vl 0 j
 −1
Í
Í
i til
l 0 til 0 vl 0 j,(3.38).

(3.39)

These MU rules are identical to (2.20) and (2.21) when we set β = 0.

Chapter 3. Determined and Overdetermined Blind Source Separation Based40
on Independent Low-Rank Matrix Analysis

3.2.4

Time-Varying Gaussian IVA

Laplace IVA employs the spherically symmetric Laplace distribution as asuper-Gaussian source distribution. The model ensures that all the frequency
components in the same source have higher-order correlation. As anothersuper-Gaussian source model with the higher-order correlation, in [104], thecircularly symmetric complex Gaussian distribution with time-varying variancer j,n is introduced to conventional IVA instead of the stationary distribution:p(y1,n, · · · , y J,n ) =Öp(y j,n )j!
Ö 1
k y j,n k22
exp −
,
=
πr j,nr j,nj(3.40)

where the time-varying variance r j,n is shared over the frequency bins in eachtime frame. Similar to (3.18), the distribution (3.40) has the spherically symmetricproperty for the multivariate vector y j,n because p(y j,n ) only depends on thevector norm k y j,n k2 . Also, the distribution is assumed to be mutually independent
for time frames and sources. Whereas the temporal source model p(y j,n ) is basedon the Gaussian distribution, the global source model p(Yn ) = p(y1,n, · · · , y J,n )becomes the super-Gaussian distribution because of the time-varying variancer j,n [186]. This time-varying Gaussian source model has been adopted for manytechniques, e.g., BSS [189, 190] and dereverberation of speech signals [191].
Hereafter, IVA based on the source distribution (3.40) is referred to as time-varying
Gaussian IVA.
Figure 3.6 (a) shows the source model (variance structure in a time-frequency
region) assumed in time-varying Gaussian IVA. Since the variance r j,n is sharedover the frequency bins, it can be interpreted as an uniform (flat) spectral
basis. On the other hand, ISNMF has a more flexible source model because
the variance ri j is independently defined in each time-frequency slot as shownin Fig. 3.6 (b). It allows us to model the specific time-frequency structurewith limited numbers of bases and activations. Similar to (3.40) and Fig. 3.6(a), the time-frequency-varying source model (3.32) and Fig. 3.6 (b) are thesuper-Gaussian distribution because of the time-frequency-varying variance ri j,n .

3.3 Independent Low-Rank Matrix Analysis

41

Uniform variance over frequency bins
Time

Vector-activated timefrequency source model

Frequency

Frequency

0.15

(a)

0.10
0.05
0
4

Time

2

0

-2

-4

-4

-2

0

2

4

Arbitrary number of bases

Basis

Time
Low-rank timefrequency source model

Frequency

Basis

Frequency

0.15

(b)

0.10
0.05
0
4

Time

2

0

-2

-4

-4

-2

0

2

4

Figure 3.6: Comparison of source models (variance structures) in (a) IVA and (b)ISNMF, where grayscale in each time-frequency slot indicates scale of variance.
IVA has uniform variance over frequency bins, and all the frequency bins havethe same activations (time-varying gains), whereas ISNMF employs limited
number of bases to capture low-rank structure resulting in more flexible source
model.

Thus, it can also be used as a source distribution in ICA-based methods.

3.3

Independent Low-Rank Matrix Analysis

3.3.1

Motivation and Strategy

For speech signal separation, Laplace IVA or time-varying Gaussian IVA canachieve better performance than FDICA. However, since only the higher-ordercorrelation defined in (3.18) or (3.40) is utilized as a spectral structure in thesource model, IVA cannot treat the specific harmonic structures of each source
and lacks flexibility, as shown in Fig. 3.6. For this reason, IVA is not suitable forsources that have characteristic (specific) spectral structures, such as instrumental
sounds or music signals. NMF decomposition is suitable for modeling thespectrogram of music or instrumental signals because such signals typicallyChapter 3. Determined and Overdetermined Blind Source Separation Based42
on Independent Low-Rank Matrix Analysis
consist of a limited number of components, for example, steady musical tones,
discrete pitches, and discrete notes, as described in Sect. 2.4. This propertymeans that the spectrogram of a music signal tends to be a low-rank matrix
compared with a speech spectrogram.
In [104], the temporal power variation of sources provided by a user is
exploited as the prior distribution of the time-varying gain r j,n , which is defined
as an inverse gamma distribution. In [192], a new multichannel source separation
method with external model information has been proposed, which is called
model-based IVA. In this approach, we consider that the time-frequency varianceri j,n for each source is given by another technique (e.g., single-channel spectral
subtraction, voice activity detection, or time-frequency binary masking) appliedin advance. The demixing matrix Wi is estimated on the basis of the independence
between sources taking the given variance ri j,n into account. These approaches
show that the estimation of Wi based on a correct and precise variance will
provide better separation performance.
On the basis of these ideas, in this chapter, I introduce ISNMF to IVA for
decomposing the sourcewise variance ri j,n using a limited number of NMF bases,
where the demixing matrix Wi and the source model p(y j,1, · · · , y j,N ) with the
NMF variables are simultaneously estimated in a fully blind fashion. Thisapproach is a natural extension of time-varying Gaussian IVA because we
extend the vector source model (frequency-uniform variance) to the low-rankmatrix source model (NMF decomposition) as shown in Fig. 3.6. For thisreason, hereafter, I call the proposed method independent low-rank matrix
analysis (ILRMA) [193, 194]. Similarly to standard FDICA or IVA, ILRMA isapplicable to the determined case (M = N). In the overdetermined case (M > N),
dimensionality reduction using principal component analysis (PCA) should beapplied so that M = N.

3.3.2

Derivation of Cost Function

In ILRMA, similarly to ISNMF, the circularly symmetric complex Gaussian
distribution is independently assumed to be as follows in each time-frequency

3.3 Independent Low-Rank Matrix Analysis

43

slot as the source model of the separated signal;p(Y1, · · · , Yn ) =

Ö

=

Ö

p(y j,1, · · · , y j,N )(3.41)p(y j,n )(3.42)jn, j!
|yi j,n | 2
1
exp −
,
=
πr
r
i
j,n
i
j,n
n,i, jÖ(3.43)

where ri j,n is the sourcewise variance that corresponds to the expectation of
the power spectrogram, namely, ri j,n = E[|yi j,n | 2 ]. The contrast function andthe negative log-likelihood function of the parameter set W and R = {ri j,n |i =1, · · · , I; j = 1, · · · J; n = 1, · · · N } are given as
G(y j,n ) =Õi|yi j,n | 2log πri j,n +ri j,n= I log π +(3.44)!
|yi j,n | 2,
log ri j,n +ri j,nÕi− log L(W, R) = const. − 2J!

Õlog | det Wi | +Õ

ii, j,n≡ LILRMA .

(3.45)|yi j,n | 2
log ri j,n +ri j,n!
(3.46)(3.47)

Here, I consider two types of ri j,n decomposition depending on the presence of apartitioning function:
ri j,n =ri j,n =ÕlÕtil,n vl j,n,(3.48)

znk tik vk j ,(3.49)k
I×L
where til,n and vl j,n are the nonnegative entries of Tn ∈ R≥I×L
0 and Vn ∈ R ≥ 0 that are
the sourcewise basis and activation matrices, and tik and vk j are the nonnegative
entries of T and V that include K bases and activations, respectively. Moreover,
N×K
znk ∈ [0, 1] is the entry of Z = (z 1 · · · z N )T ∈ R{0,1} , which is a partitioning function

Chapter 3. Determined and Overdetermined Blind Source Separation Based44
on Independent Low-Rank Matrix Analysis
Íthat clusters K bases into N sources and satisfies n znk = 1, and k = 1, · · · , K
is the new basis index. In (3.48), a fixed number of bases, L, is utilized todecompose each separated source spectrogram |yi j,n | 2 . On the other hand,
we can adaptively determine the number of bases for each separated source
spectrogram by employing the partitioning function znk as (3.49). In this model,
we only set the total number of bases to K. This approach is reasonable because
the optimal number of bases will depend on the time-frequency structure ofeach source. For a source that consists of a low-rank power spectrogram, such as
an instrumental signal, the number of bases should be small, whereas a speech
or vocal spectrogram may require more bases for its precise representation. The
cost function in ILRMA can be obtained by substituting (3.48) or (3.49) into (3.46).
In Laplace IVA, the variance ri,n is uniformly set to unity over the frequency
bins, and is not estimated. This is because the variance only determines thesignal scale of yi j,n , and it can be restored by the back-projection technique. In
time-varying Gaussian IVA, only the activation for the uniform variance isestimated based on the prior information given by users. On the other hand, thevariance in ILRMA, ri j,n , is blindly estimated by low-rank decomposition using
NMF (3.48) or (3.49) to capture the time-frequency structure as shown in Fig. 3.6(b). It is clear that when the number of bases is set to one for every source and allbases have a flat spectrum, the source models in time-varying Gaussian IVA andILRMA become identical. This fact shows that ILRMA includes time-varying
Gaussian IVA as a special case, which will be discussed in Sect. 3.5.

3.3.3

Update Rules

For the optimization of ICA or IVA, IP-based update rules, which can be derived
using the auxiliary function technique, have been proposed [176, 184, 185, 186,
104, 192], and it has been reported that these update rules are faster and morestable than those for a conventional update scheme (e.g., natural gradientmethod [175, 33]) and that the step size parameter can be omitted in each
iteration. Regarding the estimation of Wi , the differential of (3.46) w.r.t. Wi
becomes equivalent to that of the auxiliary bounding function in Laplace
IVA [184]. For this reason, the update rules of Wi based on IP can easily be3.3 Independent Low-Rank Matrix Analysis

45

derived as follows:1Õ 1
xi j xiHj ,J j ri j,n −1wi,n ← Wi Vi,nen,
 − 12H.
wi,n ← wi,n wi,n Vi,n wi,nVi,n =(3.50)(3.51)(3.52)

After the update of Wi , the separated signal yi j should be updated asHyi j,n ← wi,nxi j .

(3.53)

If we eliminate the partitioning function znk , which is ILRMA with (3.48), thedifferential of (3.46) w.r.t. til,n or vl j,n becomes identical to the differential of thecost function in ISNMF (3.37). Therefore, the update rules of til,n and vl j,n are
given asvutÍtil,n ← til,n−22
j |yi j,n | vl j,n ri j,nÍ−1
j vl j,n ri j,n,(3.54).

(3.55)vutÍvl j,n ← vl j,n−2
2
i |yi j,n | til,n ri j,n−1
i til,n ri j,nÍ

The estimated source model ri j,n should be updated by (3.48) after each updateof til,n and vl j,n .
Alternatively, if we employ the partitioning function znk to cluster K bases
into N specific sources, which is ILRMA with (3.49), we can derive the auxiliaryfunction-based update rules of znk , tik , and vk j by minimizing (3.46) in a similar
way to in [12, 167].
Here, I design an upper bound function of (3.46) as the auxiliary function.
The first term in (3.46) is a convex function for the variables. Applying
Jensen’s inequality to this term with an auxiliary variable αi j k,n ≥ 0 that satisfies

Chapter 3. Determined and Overdetermined Blind Source Separation Based46
on Independent Low-Rank Matrix Analysis
Ík αi j k,n = 1, we have

Õ αi2j k,n1
Í≤.
znk tik vk jk znk tik v k jk(3.56)

Also, the third term in (3.46) is a concave function, and we can apply the tangent
line inequality to this term with an auxiliary variable βi j,n ≥ 0 aslogÕ

znk tik vk j ≤

k

!

1

Õ

βi j,nkznk tik vk j − βi j,n + log βi j,n .

(3.57)The equality of (3.56) and (3.57) holds if and only if the auxiliary variables are set
as follows:
znk tik vk j,
k 0 znk 0 tik 0 v k 0 j(3.58)

k znk tik v k j .

(3.59)

αi j k,n = Í
βi j,n =ÍUsing these upper bounds, we can design the auxiliary function of (3.46) as+
LILRMA ≤ LILRMA="Õ Õ |yi j,n | 2 αi2j k,ni, j+znk tik vk jn,k

− 2 log | det Wi |!

1Õβi j,nk#znk tik vk j − βi j,n + log βi j,n .

(3.60)+The update rules for LILRMA
with respect to each variable are determined by+
setting the gradient to zero. From ∂LILRMA/∂znk = 0, we obtain

"
Õ
i, j

−

|yi j,n | 2 αi2j k,n
2 t v
znk
ik k j+1
βi j,n#tik vk j = 0.

(3.61)

By transposing the first term in (3.61) to the right-hand side and multiplying

3.3 Independent Low-Rank Matrix Analysis

47

2 , we haveboth sides by znk

2
znk

Õ 1
Õ |yi j,n | 2 αi2j k,ntik vk j =.
βi j,ntik vk ji, ji, j(3.62)

Finally, the MU rule of znk can be derived by substituting (3.58) and (3.59) into
(3.62) as follows:
v
u
tÍ

−2
2
i, j |yi j,n | tik v k j ri j,nznk ← znk
znk ← Í,−1i, j tik v k j ri j,nÍ(3.63)znk,
n 0 zn 0 k(3.64)

Ín znk = 1. Similarly to (3.63), the update

where (3.64) is calculated to ensurerules of tik and vk j are obtained asvutÍ
tik ← tik−2
2
j,n |yi j,n | znk v k j ri j,nÍ−1j,n znk v k j ri j,n,(3.65),(3.66)vutÍ
vk j ← vk j−2
2
i,n |yi j,n | znk tik ri j,n−1
i,n znk tik ri j,nÍ

The estimated source model ri j,n should be updated by (3.49) after each updateof znk , tik , and vk j .
Thus, we can estimate all the variables that minimize (3.46) by iterating these
update rules. Note that a scale ambiguity exists between Wi and ri j,n because
both of them can determine the scale of the separated signal yi j,n . Therefore, Wior ri j,n has a risk of diverging during the optimization. To avoid this problem,the following normalization should be applied at each iteration:
wi,n ← wi,n λn−1,(3.67)

yi j,n ← yi j,n λn−1,(3.68)

ri j,n ← ri j,n λn−2,(3.69)

Chapter 3. Determined and Overdetermined Blind Source Separation Based48
on Independent Low-Rank Matrix Analysis
andtil,n ← til,n λn−2,(3.70)

should be applied for ILRMA without a partitioning function, ortik ← tikÕznk λn−2,(3.71)nznk ← Íznk λn−2−2
n 0 zn 0 k λn 0,(3.72)

should be applied for ILRMA with a partitioning function, where λn is anarbitrary sourcewise normalization coefficient, such as the sourcewise average
power
λn =

s

1 Õ
|yi j,n | 2 .
I J i, j(3.73)

These normalizations do not change the value of the cost function (3.46). The
scale of the separated signal yi j,n can be restored by applying the back-projection
technique (3.14) after the optimization.

3.3.4

Summary of Algorithm

The detailed algorithm of ILRMA is summarized in Algorithms 1 and 2, wherePn ∈ R≥I×J
0 is the power spectrogram whose entry is pi j,n , max(·, ·) returns a matrix
with the larger elements taken from two inputs in each entry, ε denotes themachine epsilon, 1(size) denotes matrix of ones whose size is denoted as thesuperscript, and P ∈ R≥I×J×N
and R ∈ R≥I×J×N
are third-order tensors whose entries00
are pi j,n and ri j,n , respectively. To avoid division by zero, flooring with themachine epsilon is performed in the update of the NMF variables.
General NMF-based source separation techniques directly use the decomposed components to reconstruct the estimated signals or to calculate a Wiener
filter. Since the NMF decomposition is a nonlinear approximation and the
additivity of power spectrograms are generally invalid, this separation mecha-

3.3 Independent Low-Rank Matrix Analysis

49

nism often causes artificial distortion in the estimated signals and deteriorates
the sound quality. In ILRMA, unlike such NMF-based source separation,
the spectrogram decomposition is utilized only for the estimation of a latentsource model ri j.n , and the source separation is carried out by the linear spatial
demixing filter wi,n resulting in less distorted estimated signals. This issue will
be confirmed by a subjective comparison of ILRMA and the other methods in
Sect. 3.6.3.

Algorithm 1: Algorithm for ILRMA without partitioning function
Initialize Wi with identity matrix and Tn and Vn with nonnegative random values;
Calculate yi j = Wi xi j for all i and j;
3 Calculate P::n = | Y::n | .2 and R::n = Tn Vn for all n, respectively;
4 repeat5
for n = 1 to N do!
h P ◦R.−2 V T i . 21
( ::n ::n ) n,ε ;6Tn ← max Tn ◦R .−1V T
1
2

::n

7

R::n = Tn Vn ;nh T T P ◦R.−2 i . 21
n ( ::n
::n )

!
,ε ;

8

Vn ← max Vn ◦

9
10
11

R::n = Tn Vn ;for i = 1 to I do H
.−1 1(1×M)
Ui,n = J1 Xi::
Xi:: ◦ Ri:n;−1
wi,n ← (Wi Ui,n ) e n ;H U w )− 21 ;wi,n ← wi,n (wi,ni,n i,n
end

12
13
14
15
16

.−1
TnT R::nendCalculate yi j = Wi xi j for all i and j;
17
Calculate P::n = | Y::n | .2 for all n;
18
for n = 1 q
to N doÍ19
λn = I1J i, j pi j,n ;20
for i = 1 to I do
21
wi,n ← wi,n λn−1 ;
22
end
23
P::n ← P::n λn−2 ;24
R::n ← R::n λn−2 ;25Tn ← Tn λn−2 ;26
end
27 until converge;
28 Calculate ŷi j,n = Wi−1 e n ◦ yi j for all i, j, and n;

Chapter 3. Determined and Overdetermined Blind Source Separation Based50
on Independent Low-Rank Matrix Analysis
Algorithm 2: Algorithm for ILRMA with partitioning function
Initialize Wi with identity matrix, T and V with nonnegative random values, and Z withrandom values in range [0, 1]; .−1;
2 Z ← Z ◦ 1(N ×N ) Z
3 Calculate yi j = Wi xi j for all i and j;T ◦ T V for all n, respectively;
4 Calculate P::n = | Y::n | .2 and R::n = 1(I×1) z n
5 repeat6
for n = 1 to N do
 T T P ◦R.−2 ◦V 1(J ×1)  . 21{[ ( ::n ::n )] }7
bn(Z) =;.−1 ◦V 1(J ×1)[(T T R::n
) ]
8
end

T
9
Z ← max Z ◦ B(Z), ε , where B(Z) = (b1(Z) · · · b (Z)
N ) ;

.−1
10
Z ← Z ◦ 1(N ×N )Z;
11
Calculate R::n = 1(I×1) z nT ◦ T V for all n;
12
for i = 1 to I do
 . 12.−2 ◦Z T 1(N ×1){[V (Pi:: ◦Ri::
)] }
(T )
13
bi =
;
.−1 ◦Z T 1( N ×1)[(V Ri::
) ]
14
end

) T
15
T ← max T ◦ B(T ), ε , where B(T ) = (b1(T ) · · · b (TI ) ; (I×1) T 
16
Calculate R::n = 1
z n ◦ T V for all n;
17
for j = 1 to J do
! . 21
nh 
i
o
1

18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

)
=
b (V
j

T T P: j: ◦R:.−2◦Z T 1( N ×1)
j:
h

i
T ( N ×1)
T T R:.−1j: ◦Z 1;endV ← max V ◦ B(V ), ε , where B(V ) = (b1(V ) · · · bJ(V ) );
Calculate R::n = 1(I×1) z nT ◦ T V for all n;
for n = 1 to N do
for i = 1 to I do H
.−1 1(1×M)
Xi:: ◦ Ri:n;
Ui,n = J1 Xi::−1
wi,n ← (Wi Ui,n ) e n ;H U w )− 21 ;wi,n ← wi,n (wi,ni,n i,nend
endCalculate yi j = Wi xi j for all i and j;
Calculate P::n = | Y::n | .2 for all n;
for n = 1 q
to N doÍ
λn = I1J i, j pi j,n ;for i = 1 to I do
wi,n ← wi,n λn−1 ;end
P::n ← P::n λn−2 ;R::n ← R::n λn−2 ;endÍCalculate tik ← tik n znk λn−2 for all i and k;
λ−2Calculate znk ← znk Í 0 z n0 λ−2 for all n and k;
n

n k

n0

until converge;
42 Calculate ŷi j,n = Wi−1 e n ◦ yi j for all i, j, and n;

41

3.4 Relationship between IVA, MNMF, and ILRMA

3.4

51

Relationship between IVA, MNMF, and ILRMAIn NMF-based source separation, the decomposed bases and activations must be
clustered in every source to achieve source separation. To solve this problem,
MNMF has been proposed [77, 78, 80, 6, 81, 82, 83, 84]. In particular, MNMF
methods [6, 81, 82, 84] treat convolutive mixtures similarly to FDICA, IVA, andILRMA and estimate a mixing system for the sources, which is utilized for theclustering of bases. In these MNMFs, the spatial covariance [195, 196], which isthe covariance matrix of a zero-mean multivariate Gaussian distribution, hasbeen utilized to model the mixing conditions of the recording environment. In
this section, the relationships between time-varying Gaussian IVA, ILRMA, and
MNMF are revealed from the viewpoint of their assumed generative models.

3.4.1

Generative Model in MNMF and Spatial Covariance

In MNMF [6, 81, 82, 84] and its related methods [195, 196], the probability
distribution of multichannel STFT coefficients xi j is modeled by a circularly
symmetric multivariate complex Gaussian distribution with a time-frequencyvariant covariance matrix as follows:p(xi j ) =1
H (x) −1
exp
−x
R
x
,
i
j
ij ij
(x)

π M det Ri j

(3.74)

where Ri(x)j is called the spatial covariance [195, 196] of the observed multichannelHsignal xi j , namely, Ri(x)j = E[xi j xi j ]. This spatial covariance can be decomposed(s)into the time-invariant source covariance Ri,n, the time-variant scalar varianceri j,n , and the time-invariant noise covariance Ri(n) that contributes to additional
noise ni j , as

Ri(x)
j =

Õ

(s)
ri j,n Ri,n+ Ri(n) .

(3.75)n(s)
The spatial covariance Ri,nrepresents the spatial position and the spatial spreadof the nth source. In particular, if the mixing system can be modeled by the(s)mixing matrix Ai as (2.7) with a noiseless assumption, the spatial covariance Ri,nChapter 3. Determined and Overdetermined Blind Source Separation Based52
on Independent Low-Rank Matrix Analysis
is equal to the rank-1 matrix(s)H= ai,n ai,n.
Ri,n(3.76)

This mixing model is the called rank-1 spatial model, which is identical to theassumption of an instantaneous mixture in the frequency domain. In contrast, if
the mixing system cannot be modeled by (2.7) owing to, for example, strong(s)reverberation in the recording environment, the rank of Ri,nincreases so that it
becomes a full-rank spatial covariance [195, 196].

3.4.2

Existing MNMF Models

Existing MNMF models and their related works can be characterized in terms of
two features: models of spatial covariance Ri(x)j and source spectrograms. Table 3.1
summarizes the existing methods. The models proposed in [195, 196] have(s)
the most general representations. Several types of Ri,nhave been investigatedincluding rank-1 and full-rank matrices. MNMF in [6] (hereafter referred toas Ozerov’s MNMF) was the first method to model a power spectrogram ri j,nusing NMF decomposition. In this method, the sourcewise spatial covariance(s)
Ri,nis constrained by a rank-1 matrix, and an additive noise component ni j isalso assumed. The update rules of the variables based on both EM and MU
algorithms have been derived. Ozerov’s MNMF was extended to a full-rank
spatial model in [81]. Also, a more flexible source model with a partitioning
function znk was introduced in [82]. As another optimization scheme, an MU
algorithm based on an auxiliary function technique was proposed in [84](s)(hereafter referred to as Sawada’s MNMF). It also employs the full-rank Ri,nand
the flexible source model with znk and NMF variables. Note that all the existing(s)
MNMFs estimate the sourcewise mixing system Ri,nto achieve separation via
multichannel Wiener filtering (MWF) [197], whereas ILRMA estimates the
demixing matrix Wi .

3.4 Relationship between IVA, MNMF, and ILRMA

53

Table 3.1: Models of mixing system, spatial covariance, power spectrogram, andtheir optimization in each method
Model of Ri(x)jSpatial covariancePower spectrogram

Optimization

Ozerov andFévotte [6]

Í

(n)
(s)
n,l til,n vl j,n Ri,n +Ri(xi j = Ai si j + ni j )(s)andRank-1 matrix Ri,ndiagonal matrix Ri(n)NMF w/opartitioning function

EM and MU for Ai ,Ri(n) , Tn , and Vn

Arberetet al. [81]

Í

(n)
(s)
n,l til,n vl j,n Ri,n +Ri(s)andFull-rank matrix Ri,ndiagonal matrix Ri(n)NMF w/opartitioning function(s),
EM for Ri,n(n)
Ri , Tn , and Vnn ri j,n Ri,n(s)
includingSeveral types of Ri,nrank-1 and full-rank matrices

ri j,n(w/o NMF)(s)
EM for Ri,n

(s) Í

(n)
k znk tik v k j +Ri
n Ri,n(xi j = Ai si j + ni j )(s)andRank-1 matrix Ri,ndiagonal matrix Ri(n)NMF with
partitioning function

EM and MU for Ai ,Ri(n) , Z, T , and V

LiteratureDuong
et al. [196]
Ozerov
et al. [82]ÍÍSawadaet al. [84]

Í

(s) Í
k znk tik v k jn Ri,n(s)
Full-rank matrix Ri,n

NMF with
partitioning function(s),
MU for Ri,nZ, T , and V

Kitamura
et al. [194]

Í

(s) Í
n Ri,nk znk tik v k j(xi j = Ai si j )(s)Rank-1 matrix Ri,n

NMF with
partitioning function

IP for Wi = A−1
i and
MU for Z, T , and V

3.4.3

Equivalence between ILRMA and MNMF with Rank-1
Spatial Model

From (3.74), the likelihood function of the observed spatial covariance R(x) ={Ri(x)
j |i = 1, · · · , I; j = 1, · · · , J} is given as
L(R(x) ) =

Ö

=

Ö

p(xi j |Ri(x)j )(3.77)

i, j

1



H (x) −1
exp
−x
R
x
ij ,
ij ij
(x)

(3.78)

Mi, j π det Ri j

and the negative log-likelihood function is− log L(R(x) ) =

Õh

−1

H (x)
M log π + log det Ri(x)xi jj + xi j R i ji(3.79)

i, j= const. +

Õh


i
(x) −1log det Ri(x)
+
tr
X
R
,
ij ij
j

(3.80)

i, j

where Xi j = xi j xiHj is an observed instantaneous covariance matrix. Similar
to ISNMF in Sect. 3.2.3, the ML estimation based on (3.80) is identical to themultichannel IS divergence DMIS [84], which is known as Stein’s loss [198] in thestatistics field or the log-determinant divergence [199] in the machine learning

Chapter 3. Determined and Overdetermined Blind Source Separation Based54
on Independent Low-Rank Matrix Analysis
field:Õi, j

DMIS (Xi j kRi(x)
j )=


i
Õh 
−1
(x) −1
−
log
det
X
R
−
M
tr Xi j Ri(x)ij ijj(3.81)

i, j= const. +

Õh


i
(x) −1+tr
X
R.
log det Ri(x)ij ijj(3.82)i, j

In FDICA, IVA, and ILRMA, the mixing model (2.7) with a noiselessassumption is used, which results in the rank-1 spatial model (3.76). On the basisof this assumption, the covariance matrix Ri(x)j can be rewritten using the mixing
matrix Ai as
Ri(x)
j =

Õ

H
ri j,n ai,n ai,n(3.83)n= Ai Di j AH
i ,(3.84)©ri j,1 0 · · · 0 ª­.. ®
­ 0 ri j,2 . . .
. ®®
Di j = ­­ .
®.
..
..
.
. 0 ®
­ ..
­
®
0
·
·
·
0
r
i j,N ¬«

(3.85)

where3.4 Relationship between IVA, MNMF, and ILRMA

55

If we substitute (3.84) into the cost function in MNMF (3.80), we obtain− log L(R(x)

) = const. +Õ

log det Ai Di j AH
i + tr


Xi j



AH
i

 −1

−1
Di−1
j Ai


(3.86)

i, j= const. +Õlog(det Ai )(det Di j )(det Ai )H

i, j

+ tr



Wi−1 yi j yiHj

= const. +

Õh



Wi−1

H

WiH Di−1
j Wi
(3.87)
i
log | det Ai | 2 + log det Di j + tr Wi Wi−1 yi j yiHj Di−1
j

i, j

"
=const. − 2JÕlog | det Wi | +

Õ

i

log

Ö

(3.88)#


ri j,n + tr yi j yiHj Di−1
j

n

i, j

(3.89)"= const. − 2JÕilog | det Wi | +Õi, j,n#log ri j,n +|yi j,n | 2,
ri j,n(3.90)

where we used xi j = Wi−1 yi j and Wi = A−1
i to transform the variables. Thus, it
is revealed that the cost function in MNMF with the rank-1 spatial model is
identical to (3.46), the cost function in ILRMA, because the same spatial andsource models are assumed.

Figure 3.7 shows the relationship between IVA, ILRMA, and MNMF. MNMF
with a rank-1 spatial model, which assumes an instantaneous mixture in thefrequency domain, is essentially equivalent to ILRMA, which is IVA with aflexible source model using NMF decomposition. Therefore, ILRMA can beconsidered as an intermediate model between IVA and MNMF in terms of themodel flexibility. From the IVA side, we introduced the source model using
NMF with bases to capture the specific spectral patterns, and from the MNMF
side, a rank-1 spatial model was introduced to transform the variable Ai into Wi
and to make the optimization more efficient.

Flexible

MNMF
Introduce rank-1
spatial model

Restricted

Spatial model

Chapter 3. Determined and Overdetermined Blind Source Separation Based56
on Independent Low-Rank Matrix Analysis

Introduce NMF
source model

IVA

Restricted

ILRMA
Source model

Flexible

Figure 3.7: Relationship between IVA, ILRMA, and MNMF from viewpoint offlexibility of spatial and source models.

3.5

Experimental Analysis of ILRMA using Artificial
Observation

In this section, I discuss the inherent difference between FDICA, Laplace IVA,
and ILRMA. In addition, I evaluate them via BSS experiments using artificial
sources and show that ILRMA possesses better flexibility than FDICA and
Laplace IVA for both the source and spatial models.

3.5.1

Difference between Assumption in Source Model

In IVA, as already discussed in Sects. 3.2.2 and 3.2.4, we generally introduce
the spherically symmetric multivariate distribution to ensure the higher-ordercorrelation between frequency bins, where the stationary distribution is assumedin Laplace IVA and the time-varying variance is introduced in time-varying
Gaussian IVA. As shown in Fig. 3.6 (a), all the frequency components are assumed
to have the same activation (time-varying gain) in time-varying Gaussian IVA.
This simple and nonflexible source model can be interpreted as a specific NMF
that has only one frequency-uniform (flat) basis for each source. Therefore, the3.5 Experimental Analysis of ILRMA using Artificial Observation

57

number of bases of a model spectrogram in time-varying Gaussian IVA always
becomes one.
On the other hand, conventional MNMF and proposed ILRMA independently
assume circularly symmetric complex Gaussian distribution for each timefrequency slot [125] because their cost functions are based on IS divergence.
Therefore, the estimated variances ri j,n can explicitly express a model spectrogram
via NMF decomposition with an arbitrary number of bases (see the spectrogramin Fig. 3.6 (b)). For this reason, the source model in ILRMA is more flexible thanthat in IVA. In addition, time-varying Gaussian IVA can be thought of as a
special case of ILRMA. If the number of bases for each source is set to one andthe basis is fixed to a flat spectrum, both methods become essentially equivalent.
For conventional FDICA, its source model depends on how the permutation
problem is solved. The permutation solver utilizing the correlation between
frequency bins [179] is an essentially equivalent approach to IVA. However, thepermutation solver based on DOA of each source [178] is a different approach.
Hereafter, I refer to the combined method of FDICA and DOA-based permutation
solver as FDICA+DOA. FDICA+DOA uses the estimated steering vectors(estimated spatial model), and there is no explicit source model except for
non-Gaussianity in the time series for each frequency bin.

3.5.2

Difference between Assumption in Spatial Model

In IVA and ILRMA, there is no explicit assumption in the spatial model (mixing
system) except for the rank-1 spatial model (3.76). Both methods only use thestatistical independence between source models (model spectrograms) and theobserved multichannel mixtures to estimate the demixing matrix.
In contrast, FDICA+DOA directly uses the difference between the estimatedspatial conditions for each source to solve the permutation problem. Therefore,
the separation performance of FDICA+DOA is sensitive to the spatial setup of
the sources; if the positions of the sources become close or the reverberation
becomes strong, the error of the permutation solver may increase. In summary,
FDICA+DOA is severely affected by the spatial conditions rather than source
modeling, whereas IVA and ILRMA are not.

Chapter 3. Determined and Overdetermined Blind Source Separation Based58
on Independent Low-Rank Matrix Analysis

・・・

・・・

Power
spectrogram

Figure 3.8: Artificial source that consists of R bases.

3.5.3

Experimental Validation

I validate the difference between both the source and spatial models among IVA,
FDICA+DOA, and ILRMA. In this validation, for simplicity, the numbers ofsources and microphones are set to two, namely, N = M = 2.

Design of Artificial Spectrograms with R Bases
From the difference between the source models of IVA and ILRMA, we can
expect that the number of bases (rank) in the power spectrogram will affect the
separation performance for IVA. If the power spectrogram of each source consistsof only one basis, both IVA and ILRMA can separate the sources with highaccuracy. However, if the sources have more complicated power spectrograms,
the source model in IVA cannot represent them in principle, and the separation
performance may decrease.
To investigate this issue, in this experiment, I produce artificial sources whose
power spectrograms can be represented by R bases. Figure 3.8 shows the power
spectrogram that I produced. To simulate a nonnegative sparse spectrogram, I
generate nonnegative random values fir and gr j that obey independent andidentically distributed (i.i.d.) gamma distributions, where r = 1, · · · , R is
the integral index of the basis in matrices F and G. The power spectrogramis a product of F and G and its size is I × J. The gamma distribution can be3.5 Experimental Analysis of ILRMA using Artificial Observation

59

Table 3.2: Estimated values of shape parameter κ so that kurtosis of F G isadjusted to 50 for each R
Number of bases R
1
2
3
4
5
6
7
8

Shape parameter κ
0.838090.549620.43450
0.36929
0.326170.29504
0.271240.25231represented as
Gamma( χ|κ, θ) = χ κ−1

 χ
1
exp
− ,
Γ(κ)θ κ
θ

(3.91)

where χ is a random variable, and κ and θ are shape and scale parameters,
respectively. After producing the power spectrogram F G, I add random phasesthat obey a uniform distribution in the range [0, 2π] to F G, and the produced
complex spectrogram (F G with random phases) is used as an artificial source
whose power spectrogram has R bases. Therefore, in this procedure, I simulate
the variances of zero-mean spherical complex Gaussian distributions with anouter product of variables that obey i.i.d. gamma distributions and their linear
combination.
In this artificial source, it is important to set κ to an appropriate value. For
example, when κ is set to a constant value regardless of R, the random values
in the power spectrogram F G become close to a Gaussian distribution.ThisÍRis because the kurtosis of the element r=1
fir gr j in F G converges to three by
the central limit theorem when R increases. For this reason, the separation
accuracy of ICA-based methods decreases as R increases. To avoid this influence,
I adjust the shape parameter κ for each value of R so that the kurtosis of F G
is always the same value regardless of R. Such a κ can be derived using the
moment-cumulant transform [200] (see Appendix A). The following equation

Frequency ofsource components

Chapter 3. Determined and Overdetermined Blind Source Separation Based60
on Independent Low-Rank Matrix Analysis

DOA
Figure 3.9: Artificial DOA with Gaussian distributions.

gives the shape parameter κ used to adjust the kurtosis of F G:
ζ(κ, R)− kurt = 0,ξ(κ, R)(3.92)

where kurt is the intended value for the kurtosis of F G andζ(κ, R) = 84κ 3 + 174κ 2 + 132κ + 36
+ R 52κ 4 + 60κ 3 + 19κ 2 + R 2 12κ 5 + 6κ 4 + R 3 κ 6,ξ(κ, R) = R 4κ 4 + 4κ 3 + κ 2 + R 2 4κ 5 + 2κ 4 + R 3 κ 6 .

(3.93)(3.94)

Since no closed-form solution exists that satisfies (3.92), I calculate the optimal κ
by a greedy search. Table 3.2 shows the estimated shape parameter values whenkurt = 50. I experimentally confirmed that the kurtosis of the produced power
spectrogram F G is always controlled to be approximately 50 using these shapeparameters.
Design of Artificial Mixing Systems
For a mixing system, I designed an artificial DOA that consists of N = 2 Gaussian
distributions, as shown in Fig. 3.9, where µn and σn are the mean value (positionof the source) and standard deviation of the nth source, respectively. This

3.5 Experimental Analysis of ILRMA using Artificial Observation
Laplace IVA

FDICA+DOA

ILRMA with only one basis

ILRMA with

35

35

30

30

25
20
15
10
5
0
1

bases

supervised bases

SDR improvement [dB]SDR improvement [dB]

ILRMA with

61

25
20
15
10
5

2

3 4 5 6 7
Number of basesfor each source ( )

(a)

8

0
1

2

3 4 5 6 7
Number of basesfor each source ( )8(b)Figure 3.10: Separation results of (a) first source and (b) second source forvarious numbers of bases.

modeling mimics an actual acoustical phenomenon in which the DOAs ofwavefronts in different frequencies i are randomly distributed owing to the room
reverberation effect. I produced steering vectors ai,n that obey the Gaussian
distributions in Fig. 3.9 and prepared an artificial mixing matrix Ai . Finally, I
produced an artificial observed signal xi j with artificial sources and an artificial
mixing system using (2.7).
Experiment on Variational Artificial Spectrogram
In this experiment, I assume the following conditions: I = J = 257, kurt = 50,
θ = 1, µ1 = 5π/12, µ2 = 7π/12, σ12 = σ22 = 0.05, and the interelement spacingof microphones is set to 5.66 cm. Figure 3.10 shows the improvement of thesignal-to-distortion ratio (SDR) [201] for various numbers of bases R, where
SDR indicates the total separation performance and the improvement in the
SDR is the increment from the SDR value of the observed signal. For ILRMA, I

Chapter 3. Determined and Overdetermined Blind Source Separation Based62
on Independent Low-Rank Matrix Analysis
use a simple formulation without the partitioning function. Also, I evaluate
three patterns, namely, the case of L = 1 (ILRMA with only one basis), the caseof a suitable number of spectral bases L = R (ILRMA with R bases), and thecase of a supervised source model by setting T = F and V = G (ILRMA with R
supervised bases). From Fig. 3.10, the separation scores of Laplace IVA andILRMA with only one basis decrease when the number of bases of each source,
R, increases because they cannot capture the exact power spectrograms. In
contrast, ILRMA with R bases can maintain high SDR values because the power
spectrogram of each source can be represented by a model spectrogram using R
spectral bases in T . This clearly demonstrates the flexibility of the source modelin ILRMA.

Experiment on Variational Artificial Mixing Systems
From the difference between the spatial models in FDICA+DOA and ILRMA,
we can expect that the mixing system (spatial conditions of each source) willaffect the separation performance for FDICA+DOA. If the source positions
are close or the variance of the DOAs is large, a large error of DOA clustering
occurs in FDICA+DOA, resulting in marked degradation of the separation.
However, since IVA and ILRMA do not use the explicit properties of the mixing
condition (spatial model), we can expect that their separation performance
will not strongly depend on the source positions or the variance of the DOAs.
To investigate this issue, in this experiment, I produce observed signals withvarious mixing conditions and evaluate the separation performance. I use theartificial sources described in Sect. 3.5.3, where the power spectrograms of thesesources are generated with kurt = 50 and R = 1. The mixing system is producedby the artificial DOA shown in Fig. 3.9 with various µ1 , µ2 , σ12 , and σ22 . Note thatthe experiment in which σ12 and σ22 are changed does not simulate a change
in the reverberation time. It only controls the variance of the DOAs over thefrequencies, and the length of the impulse response does not change. Therefore,
even when using larger σ12 and σ22 , the rank-1 spatial model is always valid inthis simulation. For ILRMA, the number of bases L is set to one, which is equal
to R. The other conditions are the same as those in Sect. 3.5.3.

3.5 Experimental Analysis of ILRMA using Artificial Observation

FDICA+DOA

35

35

30

30
SDR improvement [dB]SDR improvement [dB]

Laplace IVA

25
20
15
10

63

ILRMA

25
20
15
10

5

5

0
0.0 0.1 0.2 0.3 0.4 0.5 0.6
Angle between sources() [rad]0
0.0 0.1 0.2 0.3 0.4 0.5 0.6
Angle between sources() [rad](b)(a)

Figure 3.11: Separation results of (a) first source and (b) second source forvarious angles.

FDICA+DOA

35

35

30

30
SDR improvement [dB]SDR improvement [dB]

Laplace IVA

25
20
15
10

ILRMA

25
20
15
10

5

5

0
0.00 0.05 0.10 0.15 0.20
Variance of sources ()

00.00 0.05 0.10 0.15 0.20
Variance of sources (
)

(a)

(b)

Figure 3.12: Separation results of (a) first source and (b) second source forvarious variances.

Chapter 3. Determined and Overdetermined Blind Source Separation Based64
on Independent Low-Rank Matrix Analysis
Figure 3.11 shows the SDR results for various positions of the sources (µ1 andµ2 ), where the horizontal axis indicates the angle between the two sources,µ2 − µ1 , and the variances are fixed to σ12 = σ22 = 0.05. Also, Fig. 3.12 shows the
SDR results for various variances (σ12 and σ22 ), where σ12 and σ22 are always set tothe same value and the positions of the sources are fixed to µ1 = 5π/12 andµ2 = 7π/12. From these results, we can confirm that the separation performanceof FDICA+DOA is sensitive to the mixing system. In particular, when thesource positions become close (around 0.0 on the horizontal axis in Fig. 3.11)
or the variance of the DOAs. is large (around 0.20 on the horizontal axis in
Fig. 3.12), the permutation solver using the DOA cannot cluster the sources
correctly, resulting in large permutation errors. In contrast, Laplace IVA andILRMA achieve good performance regardless of the mixing system because
these methods do not have explicit spatial constraints. This shows the flexibilityof the spatial model in ILRMA.

3.6

Comparison of Speech and Music Separation Performance

In this section, I confirm the efficacy of ILRMA for determined BSS task bycomparing the separation performance of many techniques.

3.6.1

Datasets

In this experiment, I investigated two cases: speech signal and music signal
cases. In the speech signal case, I used live recorded mixture signals obtained
from an underdetermined BSS task in SiSEC2011 [122]. This dataset includes 12
mixture signals (dev1 and dev2 datasets) with female and male speech, where thereverberation time is 130 ms/250 ms and the microphone spacing is 1 m/5 cm.
Details of the other conditions for this dataset can be found in [122]. Note thatsince this dataset is for underdetermined BSS, three sources (N = 3) are provided
as stereo recordings (M = 2). In this experiment, I used only the first and second
speech sources to make the task determined (M = N = 2). In the music signal

3.6 Comparison of Speech and Music Separation Performance
Impulse response E2A(reverberation time: 300 ms)
Source 1

Source 2

Impulse response JR2(reverberation time: 470 ms)
Source 1

Source 2

2m

50

65

50
5.66 cm

(a)

2m

60

60
5.66 cm(b)

Figure 3.13: Recording conditions of impulse responses (a) E2A and (b) JR2 for
two-source case.

Table 3.3: Music sources for two-source case
ID
1
2
3
4

Song namebearlin-roads
another_dreamer-the_ones_we_love
fort_minor-remember_the_nameultimate_nz_tour

Source (1/2)
acoustic_guit_main/vocals
guitar/vocals
violins_synth/vocals
guitar/synthcase, the observed signals were produced by convoluting the impulse response
E2A or JR2, which was obtained from the RWCP database [202], with eachsource. Figure 3.13 shows the recording conditions of impulse responses E2A
and JR2. As the music sources, I used professionally produced music obtained
from a music separation task in SiSEC2011. The titles of the music and the
instruments used are shown in Table 3.3.

3.6.2

Experimental Analysis of Optimal Number of Bases

In this subsection, I give an experimental analysis of the optimal number ofbases in ILRMA. Since NMF decomposition is more suitable for music thanspeech because of the stable pitch of instruments, we expect that the optimal
number of bases will be different between them. For this reason, I evaluated theChapter 3. Determined and Overdetermined Blind Source Separation Based66
on Independent Low-Rank Matrix Analysis

Table 3.4: Experimental conditions
Sampling frequency
Window length in STFT
Window function
Window shift length
Initialization
Number of iterations

16 kHz256 ms in speech signal case and
512 ms in music signal caseHamming window
128 ms in both speech and music signal cases
Wi : identity matrix
NMF variables: uniform random values [0, 1]
200

separation performance of ILRMA without a partitioning function using various
numbers of bases for each source, where this method models all the sources
with the same fixed number of bases L. The experimental conditions used areshown in Table 3.4. As the evaluation score, I used the improvement of SDR.
Figures 3.14 and 3.15 show the average SDR improvements and theirdeviations in 10 trials with different various pseudorandom seeds, where thespeech signal (Fig. 3.14) is a female speech from the dev1 dataset with 130 ms
reverberation time and 1 m microphone spacing, and the music signal (Fig. 3.15)
is song ID4 with impulse response E2A. From these results, we confirm that
ILRMA cannot achieve a good separation performance for speech signals when
the number of bases is large. This is due to the structural complexity of the
speech spectrogram. Figure 3.16 shows cumulative singular values of eachsource spectrogram in the speech and music signals. The speech sources require
more than 50 bases to represent the spectrogram while the music sources are
saturated with 25 bases. Because of the time-varying pitch, it is difficult tocapture speech spectrograms using NMF decomposition. If ILRMA fails to
capture the correct spectrogram of each speech in the optimization, the demixing
matrix will be trapped at a poor solution (local minimum). On the other hand,owing to the low rank of music spectrograms, ILRMA gives a better performancefor music separation even if the number of bases increases.

16

16

12

12

SDR improvement [dB]SDR improvement [dB]

3.6 Comparison of Speech and Music Separation Performance

8
4

0
-4

8
4

0
-4

1 2 3 4 5 6 7 8 9 10
Number of bases foreach source ( )

67

(a)

1 2 3 4 5 6 7 8 9 10
Number of bases foreach source ( )

(b)

20

20

16

16

SDR improvement [dB]SDR improvement [dB]

Figure 3.14: Average SDR improvements for female speech (dev1) with 1 m
microphone spacing and 130 ms reverberation time: (a) first speaker and (b)second speaker.

12
8

4
0

1 5 10 15 20 25 30 35 40 45 50
Number of bases foreach source ( )

(a)

12
8

4
0

1 5 10 15 20 25 30 35 40 45 50
Number of bases foreach source ( )(b)Figure 3.15: Average SDR improvements for song ID4 with impulse response
E2A: (a) guitar and (b) synth.

Chapter 3. Determined and Overdetermined Blind Source Separation Based68
on Independent Low-Rank Matrix Analysis

Figure 3.16: Cumulative singular values of each source spectrogram in dev1
female speech and song ID4 music, where all sources are truncated to be thesame signal length.

3.6.3

Comparison of Separation Performance

Experimental Conditions
I next compare the separation performance of eight methods, namely, directional
clustering (DC) [60], Laplace IVA, Ozerov’s MNMF, Ozerov’s MNMF with random
initialization, Sawada’s MNMF, ILRMA w/o partitioning function, ILRMA withpartitioning function, and Sawada’s MNMF initialized by ILRMA. DC is a simple
separation technique, which clusters all the STFT coefficients into specific sources
using both powers and phases. In this experiment, I use k-means clusteringin directional clustering, which corresponds to a double-disjoint assumption,
namely, we assume that each time-frequency slot has only one source component.
In Ozerov’s MNMF, I used the experimental conditions described in [6], asshown in Table 3.5, where the mixing matrices and the source models are
initialized by estimation using Soft-LOST [203] with the permutation solver [181].

3.6 Comparison of Speech and Music Separation Performance69

Table 3.5: Experimental conditions used in Ozerov’s MNMF
Sampling frequency
Window length in STFT
Window function
Window shift length
Number of bases
Initialization of
mixing matrices
Initialization ofsource models(NMF variables)
Annealing for
EM algorithm
Number of iterations

16 kHz128 msHamming window
64 ms
10 bases for each speech source
and 4 bases for each music source
Mixing matrices estimated by SoftLOST [203] and permutation solver [181]Pretrained bases and activations
using simple NMF based on
KL divergence with sources
estimated by Soft-LOST and [181]Annealing with noise
injection proposed in [6]500Also, Ozerov’s MNMF with random initialization has the same conditions as
Ozerov’s MNMF except for the initialization, namely, the mixing matrices andthe source models are initialized by the identity matrix and the uniform random
values [0, 1], respectively. In the other methods, the experimental conditions
shown in Table 3.4 were used. In ILRMA with partitioning function, I only set
the total number of bases, K, and the sources are flexibly modeled with the
optimal number of bases using the partitioning function Z. Sawada’s MNMF
initialized by ILRMA has the same algorithm as Sawada’s MNMF, but the initial(s)values of the spatial covariance matrix Ri,nare given by (3.76), where the steeringvector ai,n is calculated from the inverse of the demixing matrix Wi estimated by
ILRMA w/o partitioning function.
On the basis of the results in Sect. 3.6.2, I set the number of bases of eachsource to L = 2 for the speech signals and L = 30 for the music signals in ILRMA
w/o partitioning function. In ILRMA with partitioning function and Sawada’s
MNMF, I set the total number of bases to K = 2 × N for the speech signals andK = 30 × N for the music signals. The number of bases used in Ozerov’s MNMF

Chapter 3. Determined and Overdetermined Blind Source Separation Based70
on Independent Low-Rank Matrix Analysis
is shown in Table 3.5.

Results
Figures 3.17 and 3.18 respectively show examples of results for speech signals
given by the average SDR improvements and their deviations in 10 trials withdifferent pseudorandom seeds. Also, Figs. 3.19 and 3.20 show examples ofresults for music signals. The total average scores are shown in Tables 3.6 and 3.7.
From these results, we confirm that DC cannot separate the sources becauseof the imperfect double-disjoint assumption and the deviation of the DOAs
in reverberant environments. Also, Laplace IVA cannot achieve satisfactory
separation because the source model in Laplace IVA is not flexible as describedin Sect. 3.5. Ozerov’s MNMF outperforms Laplace IVA for the music signals,
but the separation performance for speech signals is inferior to that of Laplace
IVA. In addition, Ozerov’s MNMF with random initialization cannot solve
the BSS problem. This method must be initialized by other methods to find agood solution. The results of Sawada’s MNMF have large error bars, namely,
this method is also sensitive to initial values. However, for the music signals,
Sawada’s MNMF gives better performance than Laplace IVA and Ozerov’s
MNMF. ILRMA-based methods achieve a high and stable performance. For the
speech signals, ILRMA w/o partitioning function is preferable to ILRMA withpartitioning function. This might be due to the sensitivity of the performance tothe number of bases, as discussed in Sect. 3.6.2. In contrast, for the music signals,
ILRMA with partitioning function exhibits slightly higher performance than
ILRMA w/o partitioning function. This improvement is achieved by modelingthe sources with the optimal number of bases using the partitioning function znk .
Figure 3.21 shows an example of the convergence of the partitioning function z1kfrom k = 1 to k = K in the music signal case. These values indicate whether thekth basis contributes to only source one (z1k = 1) or only source two (z1k = 0).
We can confirm that almost partitioning functions converge to one or zero, butseveral ones converge to intermediate values. This is because similar or the same
spectral patterns appear in both two sources. Thanks to the partitioning function,
all the sources can effectively be modeled with the optimal number of bases.

3.6 Comparison of Speech and Music Separation Performance

SDR improvement [dB]SDR improvement [dB]

Source 1

16
14
12
10
8
6
4
2
0
-2
-4

16
14
12
10
8
6
4
2
0
-2
-4

DC

Laplace
IVA

71

Source 2

Ozerov’s
Ozerov’s Sawada’s
ILRMA
Sawada’s
ILRMA
MNMF MNMF with MNMF
w/o
with
MNMF
randompartitioning partitioning initialized byinitialization
functionfunction
ILRMA

(a)

DC

Laplace
IVA

Ozerov’s
Ozerov’s Sawada’s
ILRMA
Sawada’s
ILRMA
MNMF MNMF with MNMF
w/o
with
MNMF
randompartitioning partitioning initialized byinitialization
functionfunction
ILRMA

(b)

Figure 3.17: Average SDR improvements for female speech (dev1) with 1 m
microphone spacing, where reverberation time is (a) 130 ms and (b) 250 ms.

Chapter 3. Determined and Overdetermined Blind Source Separation Based72
on Independent Low-Rank Matrix Analysis

SDR improvement [dB]SDR improvement [dB]

Source 1

14
12
10
8
6
4
2
0
-2
-4

14
12
10
8
6
4
2
0
-2
-4

DC

Laplace
IVA

Source 2

Ozerov’s
Ozerov’s Sawada’s
ILRMA
Sawada’s
ILRMA
MNMF MNMF with MNMF
w/o
with
MNMF
randompartitioning partitioning initialized byinitialization
functionfunction
ILRMA

(a)

DC

Laplace
IVA

Ozerov’s
Ozerov’s Sawada’s
ILRMA
Sawada’s
ILRMA
MNMF MNMF with MNMF
w/o
with
MNMF
randompartitioning partitioning initialized byinitialization
functionfunction
ILRMA

(b)

Figure 3.18: Average SDR improvements for male speech (dev1) with 1 m
microphone spacing, where reverberation time is (a) 130 ms and (b) 250 ms.

3.6 Comparison of Speech and Music Separation Performance

Source 1

73

Source 2

SDR improvement [dB]

16
12

8
4
0
-4
-8

DC

Laplace
IVA

(a)

16

SDR improvement [dB]

Ozerov’s
Ozerov’s Sawada’s
ILRMA
Sawada’s
ILRMA
MNMF MNMF with MNMF
w/o
with
MNMF
randompartitioning partitioning initialized byinitialization
functionfunction
ILRMA

12

8
4
0
-4
-8

DC

Laplace
IVA

Ozerov’s
Ozerov’s Sawada’s
ILRMA
Sawada’s
ILRMA
MNMF MNMF with MNMF
w/o
with
MNMF
randompartitioning partitioning initialized byinitialization
functionfunction
ILRMA

(b)

Figure 3.19: Average SDR improvements for music signal song ID3 with impulse
response (a) E2A and (b) JR2.

Chapter 3. Determined and Overdetermined Blind Source Separation Based74
on Independent Low-Rank Matrix Analysis

Source 1

Source 2

SDR improvement [dB]

20
15
10

5
0
-5

DC

Laplace
IVA

(a)

20

SDR improvement [dB]

Ozerov’s
Ozerov’s Sawada’s
ILRMA
Sawada’s
ILRMA
MNMF MNMF with MNMF
w/o
with
MNMF
randompartitioning partitioning initialized byinitialization
functionfunction
ILRMA

15
10

5
0
-5

DC

Laplace
IVA

Ozerov’s
Ozerov’s Sawada’s
ILRMA
Sawada’s
ILRMA
MNMF MNMF with MNMF
w/o
with
MNMF
randompartitioning partitioning initialized byinitialization
functionfunction
ILRMA

(b)

Figure 3.20: Average SDR improvements for music signal song ID4 with impulse
response (a) E2A and (b) JR2.

3.6 Comparison of Speech and Music Separation Performance75

Table 3.6: Averaged SDR improvements over various speech signals and sources
with same recording conditions in two-source case
Conditions(rev. time andmic. spacing)
130 ms & 1 m
130 ms & 5 cm
250 ms & 1 m
250 ms & 5 cm

Laplace Ozerov’s
IVA
MNMF

DC
2.59
-1.51
0.14
-1.56

2.98
2.86
2.03
2.43

1.35
2.13
0.49
0.91

Ozerov’s MNMFILRMA w/o
Sawada’s
with random
partitioning
MNMF
initialization
function
-2.11
0.68
11.91
-0.22
1.13
8.97
-2.02
0.48
7.34
-1.06
0.47
6.43

ILRMA withpartitioning
function
4.88
3.48
2.09
1.91

Sawada’s MNMF
initialized
by ILRMA
6.36
5.60
4.19
3.95

Table 3.7: Averaged SDR improvements over various music signals and sources
with same impulse response in two-source case
Impulseresponse

DC

E2A
JR2

-0.73
-1.18

Laplace Ozerov’s
IVA
MNMF
5.72
1.77

5.73
2.37

Ozerov’s MNMF
Sawada’s
with random
MNMF
initialization
-2.70
10.32
0.75
6.11

ILRMA w/opartitioning
function
12.29
6.62

ILRMA withpartitioning
function
12.29
7.40

Sawada’s MNMF
initialized
by ILRMA
14.41
9.06

1.0

Value of

0.8

0.6
0.4
0.2
0.0
0

100

200

Iteration step

300

400

Figure 3.21: Convergence of z1k from k = 1 to k = K in music signal case.

The deviations of the ILRMA-based methods are smaller than those ofOzerov’s and Sawada’s MNMFs, which is particularly evident in ILRMA w/opartitioning function. This is because the optimization of the demixing matrix
using the IVA update rules results in a stable separation performance. In fact, I
experimentally confirmed that the initialization using Soft-LOST [203] andChapter 3. Determined and Overdetermined Blind Source Separation Based76
on Independent Low-Rank Matrix Analysis
the permutation solver [181], which was employed in Ozerov’s MNMF, did
not improve the separation performance of ILRMA w/o partitioning function.
This fact means that ILRMA is robust against the initial values. For musicsignals with impulse response JR2 (Figs. 3.19 (b) and 3.20 (b)), the SDRs of the
ILRMA-based methods are markedly degraded compared with those with
impulse response E2A because the reverberation time is longer than impulse
response E2A and is close to the length of the window function in the STFT.
Even if Sawada’s MNMF has the potential to model such a mixing system by
employing a full-rank spatial model, it is a very difficult problem to find the(s)optimal Ri,n. However, Sawada’s MNMF initialized by ILRMA can achieve highand very stable separation performance even with impulse response JR2. Thismeans that the demixing matrix estimated by ILRMA can be a good initial value(s)of the spatial model Ri,nin order to find the full-rank spatial covariance.
Figure 3.22 shows an example of the SDR convergence for each methodin music signal case. Both Laplace IVA and the proposed methods showmuch faster convergence than Sawada’s MNMF. Also, the numbers of required
iterations in Sawada’s MNMF is greatly reduced by the initialization of therank-1 spatial covariance. This result shows the difficulty of optimizing the(s)full-rank spatial covariance Ri,n.
Figure 3.23 shows a result of a subjective evaluation, where I presented 48
pairs of separated speech and 48 pairs of separated music signals in random
order to 14 examinees, who selected which signal they preferred from theviewpoint of the total quality of the separated sounds. Also, Fig. 3.24 shows aprobability of selection regarding a difference between two subjective scores. For
example, the difference of subjective scores of Laplace IVA and Ozerov’s MNMF
for speech signals is around 0.9, and it means that Laplace IVA is preferably
selected with a 81% probability when it is compared with Ozerov’s MNMF. We
can confirm that Laplace IVA is better than MNMF methods for the speech
signals. In contrast, Sawada’s MNMF achieves a better result for music signals
owing to the suitable representation using NMF. ILRMA is the most preferable
method for the high-quality separation of both speech and music signals.
Similarly to FDICA and Laplace IVA, ILRMA employs the demixing matrix Wi
for the separation, which is essentially equivalent to the spatial linear filter [47]

3.6 Comparison of Speech and Music Separation Performance
Ozerov’s MNMF

Laplace IVA

SDR improvement [dB]

Ozerov’s MNMF withrandom initialization
ILRMA w/opartitioning function
Sawada’s MNMF
initialized by ILRMA

20
18
16
14
12
10
8
6
4
2
0
-2

0

50

77

Sawada’s MNMFILRMA withpartitioning function

100
Iteration step

150

200

100
Iteration step

150

200

(a)

14

SDR improvement [dB]

12
10
8
6
4

2
0
-2
-4

0

50

(b)

Figure 3.22: SDR convergence for music signal song ID4 with impulse response
E2A: (a) guitar and (b) synth.

Subjective score

Chapter 3. Determined and Overdetermined Blind Source Separation Based78
on Independent Low-Rank Matrix Analysis
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
-1.0
-1.2

Speech signals
Music signals

Laplace
IVA

Ozerov’s
MNMF

Sawda’s
MNMFILRMA w/opartitioning
function

Figure 3.23: Results of subjective scores obtained by Thurstone pairwise
comparison method, where 48 pairs of separated speech and 48 pairs of
separated music signals are presented in random order to 14 examinees, who
selected which signal they preferred from the viewpoint of total quality ofseparated sound. Scores show relative tendency of selection.

in beamforming techniques [204, 205], and it is more difficult for such linear
filtering to generate artificial noise than for time-frequency mask separation
techniques including MNMF with MWF. Thus, the quality of separated sources
via ILRMA from the viewpoint of human perception might be better than that
via MNMF.

3.6.4

Experiments on Three-Source Case with Music Signals

I also conducted an experiment involving three sources and three microphones(M = N = 3) with music signals. Similarly to the music dataset described in
Sect. 3.6.1, I produced the observed signals using the same songs and the threeProbability of selection

3.6 Comparison of Speech and Music Separation Performance

79

1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
-2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0
Difference of subjective scores

Figure 3.24: Probability of selection regarding difference between two subjectivescores.

Table 3.8: Music sources for three-source case
ID
1
2
3
4

Song namebearlin-roads
another_dreamer-the_ones_we_love
fort_minor-remember_the_nameultimate_nz_tour

Source (1/2/3)
acoustic_guit_main/bass/vocals
drums/guitar/vocals
drums/violins_synth/vocals
guitar/synth/vocals

instruments shown in Table 3.8 with the impulse responses shown in Fig. 3.25.
The experimental conditions are those in Table 3.4, where I here omit the resultsof DC, Ozerov’s MNMF, and Ozerov’s MNMF with random initialization.
Figure 3.26 shows examples of results, and Table 3.9 shows the total averagescores in the three-source case. Similarly to the previous results, the proposed
method achieves better and more stable performance than Sawada’s MNMF,
and the spatial model estimated by ILRMA provides an efficient initializationfor Sawada’s MNMF. Table 3.10 shows the actual computational time foreach method in the three-source case, where the calculations were performed
using MATLAB 8.3 (64-bit) with an Intel Core i7-4790 (3.60 GHz) CPU. The
computational times of ILRMA-based methods are less than twice that ofChapter 3. Determined and Overdetermined Blind Source Separation Based80
on Independent Low-Rank Matrix Analysis
Impulse response E2A(reverberation time: 300 ms)
Source 2
Source 3
Source 1

Impulse response JR2(reverberation time: 470 ms)
Source 2
Source 1
Source 3

2m

2m

20

20
50

70

60

2.83 cm 2.83 cm

(a)

60

2.83 cm 2.83 cm(b)

Figure 3.25: Recording conditions of impulse responses (a) E2A and (b) JR2 forthree-source case.

Table 3.9: Averaged SDR improvements over various music signals and sources
with same impulse response in three-source case

Impulse Laplace
response
IVA
E2A
JR2

3.86
2.81

Sawada’s
MNMF
7.77
4.44

ILRMA w/o ILRMA withpartitioning partitioning
functionfunction
8.03
6.18
5.03
4.11

Sawada’s MNMF
initialized
by ILRMA
9.44
7.00

Laplace IVA. Sawada’s MNMF requires a longer computational time because theeigenvalue decomposition of a 2M × 2M matrix is required for each update(s)iteration of Ri,n. From these results, ILRMA is advantageous in terms of theconvergence speed and computational cost while maintaining comparable
separation performance with Sawada’s MNMF.

3.6.5

Experimental Analysis of Optimal Window Length

Since ILRMA estimates low-rank source model using NMF decomposition, arank of an observed spectrogram directly affects the separation performance. As
already shown in Sect. 3.6.2, the inherent time-frequency structure of each source3.6 Comparison of Speech and Music Separation Performance

SDR improvement [dB]SDR improvement [dB]

Source 1

16
14
12
10
8
6
4
2
0
-2
-4

16
14
12
10
8
6
4
2
0
-2
-4

Laplace
IVA

Sawada’s
MNMF

Source 2

Source 3

ILRMA w/opartitioning
function

ILRMA withpartitioning
function

Sawada’s
MNMF
initialized by
ILRMA

ILRMA withpartitioning
function

Sawada’s
MNMF
initialized by
ILRMA

(a)

Laplace
IVA

Sawada’s
MNMF

ILRMA w/opartitioning
function(b)

81

Figure 3.26: Average SDR improvements for music signal song ID4 in three-sourcecase with impulse response (a) E2A and (b) JR2.

Chapter 3. Determined and Overdetermined Blind Source Separation Based82
on Independent Low-Rank Matrix Analysis

Table 3.10: Computational times (s) for separation of song ID1 with impulse
response E2A in three-source case
Laplace
IVA
91.6

Sawada’s
MNMF
4498.4

ILRMA w/opartitioning function
121.0

ILRMA withpartitioning function
173.4

Table 3.11: Experimental conditions used in analysis of optimal window length
Sampling frequency
Window length in STFT
Window function
Window shift length
Initialization
Number of bases
Number of iterations

16 kHz32/64/128/256/512/768/1024/1280/1536 ms
Hamming window
1/4 of window length
Wi : identity matrix
NMF variables: uniform random values [0, 1]
L = 5/10/20/30/40/50 and K = 2L
200

(e.g., speech or music) must be suitable for ILRMA to achieve a good separation,
but the rank of the spectrogram also depends on a length of analysis windowused in STFT. In this subsection, I compare separation performance of ILRMA
with various window length and experimentally analyze its optimal condition.

Conditions
Similar to Sect. 3.6.1, I used the music dataset shown in Table 3.3, and the musicsources were convolved with the impulse responses JR2 (Fig. 3.13 (b)). The otherconditions are shown in Table 3.11. I compared nine lengths of analysis window,
and its shift length was always set to a quarter of the window length. Also, thenumber of bases (L for ILRMA w/o partitioning functin and K for ILRMA withpartitioning function) was set to six patterns. In this experiment, only Laplace
IVA, ILRMA w/o partitioning function, and ILRMA with partitioning function
were compared.

3.6 Comparison of Speech and Music Separation Performance

83

Results

Figures 3.27–3.30 show the separation results with various window lengths andnumber of NMF bases, where the SDR improvements are averaged for twosources and 10 trials using various pseudorandom seeds. From these results, we
can confirm that the separation performance strongly depends on the window
length in STFT rather than the number of NMF bases L or K. In particular,
in Fig. 3.27, the long analysis window that exceeds 1.2 s provides the highest
performance even though the improvements of Laplace IVA drop when thewindow length exceeds 1.0 s. In conventional FDICA, the separation fails if
we use a too long window in STFT. This is because the sequential signals ineach frequency bin close to a sinusoidal wave when the window length is too
long, and their independence assumption between sources collapses in eachfrequency band, meaning that there is a fundamental limitation for FDICA [206].
Therefore, the separation performance has a trade-off based on the length of theanalysis window in terms of the assumptions of linear time-invariant mixing andthe independence of sources. In ILRMA, this trade-off might be solved owing totake the source model ri j,n into account for the estimation of Wi if the source
model ri j,n could accurately capture the time-frequency structure. However, inthe other results (Figs. 3.28–3.30), the optimal window length is around 0.5 s,
and longer windows do not give a better separation.
Figure. 3.31 shows a number of bases in the power spectrogram of eachsource when its cumulative singular value reaches 80% or 90%, namely, anapproximative rank of each source spectrogram. The songs ID1 and ID4 relatively
have a low-rank power spectrograms. Indeed, the separation performance of
these observations is better than those of the others (see Figs. 3.27 and 3.30).
Thus, when we apply ILRMA to the observed signals that do not have a low-ranktime-frequency structure (e.g., vocal or speech signal), the number of NMF
bases should be set to a smaller value as shown in Figs. 3.15 and 3.28 (a) and apartitioning function should be omitted. For the signals that have low-rank
spectrograms, we can increase the number of NMF bases to achieve moreaccurate separation.

Chapter 3. Determined and Overdetermined Blind Source Separation Based84
on Independent Low-Rank Matrix Analysis

16

16

14

14

14

12

12

12

10

8
6
4
2
0

SDR improvement [dB]

16

10

8
6
4
2
0

10

8
6
4
2
0

-2

-2

-2

-4
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-40.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-40.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(a)

(b)

(c)

16

16

14

14

14

12

12

12

10

8
6
4
2
0

SDR improvement [dB]16

SDR improvement [dB]SDR improvement [dB]

ILRMA with partitioning function

ILRMA w/o partitioning function

SDR improvement [dB]SDR improvement [dB]

Laplace IVA

10

8
6
4
2
0

10

8
6
4
2
0

-2

-2

-2

-4
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-40.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-40.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(d)

(e)

(f)

Figure 3.27: Averaged SDR improvements averaged for music signal song ID1
with impulse response JR2 (reverberation time is 470 ms): (a) L = 5 and K = 10,(b) L = 10 and K = 20, (c) L = 20 and K = 40, (d) L = 30 and K = 60, (e) L = 40
and K = 80, and (f) L = 50 and K = 100. Scores are averaged for two sources and
10 trials with different pseudorandom seeds.

3.6 Comparison of Speech and Music Separation Performance

10

10

8

8

8

6
4
2
0

SDR improvement [dB]

10

6
4
2
0

6
4
2
0

-2

-2

-2

-4
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-40.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-40.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(a)

(b)

(c)

10

10

8

8

8

6
4
2
0

SDR improvement [dB]10

SDR improvement [dB]SDR improvement [dB]

ILRMA with partitioning function

ILRMA w/o partitioning function

SDR improvement [dB]SDR improvement [dB]

Laplace IVA

85

6
4
2
0

6
4
2
0

-2

-2

-2

-4
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-40.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-40.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(d)

(e)

(f)

Figure 3.28: Averaged SDR improvements averaged for music signal song ID2
with impulse response JR2 (reverberation time is 470 ms): (a) L = 5 and K = 10,(b) L = 10 and K = 20, (c) L = 20 and K = 40, (d) L = 30 and K = 60, (e) L = 40
and K = 80, and (f) L = 50 and K = 100. Scores are averaged for two sources and
10 trials with different pseudorandom seeds.

Chapter 3. Determined and Overdetermined Blind Source Separation Based86
on Independent Low-Rank Matrix Analysis

8

8

6

6

6

4
2
0

SDR improvement [dB]8-20.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

4
2
0

-2
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(a)

4
2
0
-2
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(b)

(c)

8

8

6

6

6

4
2
0

SDR improvement [dB]8

SDR improvement [dB]SDR improvement [dB]

ILRMA with partitioning function

ILRMA w/o partitioning function

SDR improvement [dB]SDR improvement [dB]

Laplace IVA

4
2
0

4
2
0

-2
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-20.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]-20.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(d)

(e)

(f)

Figure 3.29: Averaged SDR improvements averaged for music signal song ID3
with impulse response JR2 (reverberation time is 470 ms): (a) L = 5 and K = 10,(b) L = 10 and K = 20, (c) L = 20 and K = 40, (d) L = 30 and K = 60, (e) L = 40
and K = 80, and (f) L = 50 and K = 100. Scores are averaged for two sources and
10 trials with different pseudorandom seeds.

3.6 Comparison of Speech and Music Separation Performance

12

12

10

10

10

8
6

4
2

SDR improvement [dB]120
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

8
6

4
2

0
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(a)

8
6

4
2
0
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(b)

(c)

12

12

10

10

10

8
6

4
2

0
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s](d)SDR improvement [dB]12

SDR improvement [dB]SDR improvement [dB]

ILRMA with partitioning function

ILRMA w/o partitioning function

SDR improvement [dB]SDR improvement [dB]

Laplace IVA

8
6

4
2

0
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s]

(e)

87

8
6

4
2
0
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6Window length in STFT [s](f)Figure 3.30: Averaged SDR improvements averaged for music signal song ID4
with impulse response JR2 (reverberation time is 470 ms): (a) L = 5 and K = 10,(b) L = 10 and K = 20, (c) L = 20 and K = 40, (d) L = 30 and K = 60, (e) L = 40
and K = 80, and (f) L = 50 and K = 100. Scores are averaged for two sources and
10 trials with different pseudorandom seeds.

Chapter 3. Determined and Overdetermined Blind Source Separation Based88
on Independent Low-Rank Matrix Analysis

Source 1 (cumulative singular value = 80%)
Source 2 (cumulative singular value = 80%)

Source 1 (cumulative singular value = 90%)
Source 2 (cumulative singular value = 90%)

Figure 3.31: Number of bases in power spectrogram of each source when its
cumulative singular value reaches 80% or 90%: (a) song ID1, guitar (source 1)and vocals (source 2), (b) song ID2, guitar (source 1) and vocals (source 2), (c)song ID3, violins synth. (source 1) and vocals (source 2), and (d) song ID4, guitar(source 1) and synth. (source 2).

3.7 Extension of ILRMA for Overdetermined and Reverberant Recording 89

3.7

Extension of ILRMA for Overdetermined and Reverberant Recording

In the discussions so far, the linear instantaneous mixture in frequency domain,(2.7), is always assumed to be valid. Figure 3.32 (a) shows the mixing systemcorresponding to (2.7), which is also called linear time-invariant mixing. In
this mixing system, all the time frames are independent of other time frames,
meaning that they do not affect each other. However, for the case of reverberant
recording, reverberant components can leak from the previous frame as shownin Fig. 3.32 (b), and the mixed signal xi j cannot be represented using only Ai .
Therefore, the assumption of linear time-invariant mixing holds only when thelengths of all impulse responses between the sources and microphones are
sufficiently shorter than the length of the analysis window in STFT.
For this reason, in this section, I propose an extended algorithm of ILRMA
particularly for overdetermined (M > N) and reverberant recordings. Since
conventional ICA-based methods and ILRMA exploit the assumption of
linear time-invariant mixing, the separation performance degrades when thisassumption does not hold. The proposed new approach for overdetermined BSS
enables us to achieve good separation performance even for reverberant signals.
The algorithm utilizes extra observations (channels) to estimate the reverberant
components of each source [207].

3.7.1

PCA for Overdetermined BSS

When M > N, in a typical separation method using FDICA or IVA, PCA isapplied in advance and the dimension of xi j is reduced so that M = N. This
preprocessing is performed with the expectation that the reverberant components
in the observed signal are eliminated by the dimensionality reduction. Therefore,
PCA is applied to make the assumption of linear time-invariant mixing (2.7) valid
even in a reverberant environment. However, if the purpose of source separation
is to obtain each source image including the reverberation, PCA degrades theseparation performance by removing the reverberation components. Moreover,
if the source powers in mixtures are unbalanced (e.g., music signals), PCA canChapter 3. Determined and Overdetermined Blind Source Separation Based90
on Independent Low-Rank Matrix Analysis
Source signal
Observed signal
MixingsystemFrequency

Observed spectrogramTime

Frequency

Observed spectrogram(a)Source signal
Observed signal
MixingsystemLeakedcomponents

Time

(b)

Figure 3.32: Mixing system of each spectrogram slot when N = M = 2; (a) has alinear time-invariant mixing system and there is no reverberation; (b) has someleaked components from the previous frame because of reverberation.
even remove direct components of weak sources, which leads to a greater risk ofpoor separation.

3.7.2

Relaxation of Rank-1 Spatial Model in ILRMA

To relax the constraint of the rank-1 spatial model in FDICA, IVA, and ILRMA,
I propose the utilization of extra observations for modeling the reverberant
components. In this method, we consider that the number of observations
M is Q times the number of sources N, namely, M = Q × N. In conventional

3.7 Extension of ILRMA for Overdetermined and Reverberant Recording 91
overdetermined BSS, PCA is applied before the separation so that M equals N as
shown in Fig. 3.33 (a). In the proposed algorithm, we estimate M separated
signals y̌ as shown in Fig. 3.33 (b). In this approach, the leaked component
from previous frames (ni( j−1) in Fig. 3.32 (b)) of each source is modeled as anadditional new source, namely, each original source is represented with rank-Q
spatial model. To obtain an estimate of the source including both direct andreverberant components, the separated signals must be clustered using somecriteria, which is a kind of permutation problem. The clustered separated signaly̌ is represented as follows:y̌i j = y̌i j,11 · · · y̌i j,1Q y̌i j,21 · · · y̌i j,2Q · · · y̌i j,NQT
≡ y̌i j,1 · · · y̌i j,M ,Í
yi j,n = q y̌i j,nq,

T(3.95)(3.96)(3.97)

where y̌i j,n1, . . . , y̌i j,nQ correspond to the direct and reverberant components of
one source n. Finally, each estimated source yi j,n is reconstructed by summing of
the clustered components as represented by (3.97).

3.7.3

Clustering with Spectral Correlations

In Sect. 3.7.2, the complex-valued spectrograms of the sources are estimatedby assuming the independence between them. However, we can expect thatthe power spectrograms of the direct and the reverberant components for thesame source have a correlation. Based on this assumption, I propose to use
cross-correlation between the power spectrograms p̌i j,m = | y̌i j,m | 2 to determine
which separated signal y̌i j,nq corresponds to the direct or reverberant componentof which source:cor( P̌m k P̌m 0 ) = max Í

i, j p̌i j,m p̌i( j+τ),m 0 | τ = 0, 1, · · · , τmax

,

(3.98)

where Pm (∈ R≥I×J
0 ) is the power spectrograms whose element is p̌i j,m and τ isan index of the delay in the time frame. For clustering, I first calculate (3.98)
between all separated signals y̌i j,m . Then, the signals are merged in descendingChapter 3. Determined and Overdetermined Blind Source Separation Based92
on Independent Low-Rank Matrix Analysis
Observed
signals

Original
sources
MixingDimensionreduced signals
PCA

Estimated
sources

BSS(a)
Separated
signals

Observed
signals

Original
sources

Mixing

BSS

Clustered signals
Source 1

Clustering

Estimated
sources

Source 2

Reconstruction(b)
Figure 3.33: Algorithms of (a) conventional and (b) proposed methods (N = 2,
M = 4, and Q = 2), where subscripts for i and j are omitted.
order of cor until the number of clusters becomes N, with all the clusters (signalsets) required to have the same number of signals (see Fig. 3.34).

3.7.4

Auto-Clustering with Basis-Shared ILRMAFor ILRMA, we can consider another approach for clustering the signals y̌i j,m into
N sources. Since the reverberation consists of a sum of time-delayed versions of
the direct component, it is represented by the convolution. Even in the power
spectrogram domain, this model is approximately valid [208]. If we assume thatthe impulse response in the power spectrogram domain is identical over all
frequency bins, the direct and reverberant components of the same source can bemodeled by the same bases Tn (spectral patterns) and different activations Vnq(time-varying gains) as follows:P̌n1 ' Tn Vn1, P̌n2 ' Tn Vn2, · · · , P̌nQ ' Tn VnQ,

(3.99)

3.7 Extension of ILRMA for Overdetermined and Reverberant Recording 93

Calculate correlations for all
combinations of spectrograms.

Sort correlations indescending order.
Cluster the signals .
Merge andas a set .
already has two components.
(do not merge into )already has two components.
(do not merge into )Merge and as a set .

Correlation

Strong 1.

3.

4.

…

…

Weak

2.

Figure 3.34: Hierarchical clustering using correlation cor (N = 2, M = 4, andQ = 2), where all sets must have the same number of signals and subscripts for iand j are omitted..

I×L
where P̌nq (∈ R≥I×J
0 ) is the power spectrogram of signal y̌i j,nq , Tn (∈ R≥ 0 ) is ashared basis matrix whose elements are ti1,n, · · · , tiL,n , and Vnq (∈ R≥L×J
0 ) is anactivation matrix whose elements are v1 j,nq, · · · , v L j,nq . This basis sharing
leads to the separated signals y̌i j,n1, . . . , y̌i j,nQ representing the direct andreverberant components of one source n. The cost function of basis-shared
ILRMA (BSILRMA) can be defined as [207]

LBSILRMA = const. − 2JÕilog | det Wi | +Õi, j,n,q

log

Õ
l

!
|yi j,n | 2
til,n vl j,nq + Í.
l til,n vl j,nq
(3.100)Chapter 3. Determined and Overdetermined Blind Source Separation Based94
on Independent Low-Rank Matrix Analysis
Impulse response JR2(reverberation time: 470 ms)
Source 1
Source 2
2m

80

60

Microphone interval: 2.83 cm

Figure 3.35: Recording condition of impulse response used in experiment ofreverberant signals.

The update rules of Wi for minimizing (3.100) are the same as (3.50)–(3.52) if weÍ
consider N ← M = NQ and ri j,n = l til,n vl j,nq , and the update rules of the NMF
variables are obtained as follows:
v
u
 −2
tÍ
Í
2
j,q |yi j,nq | vl j,nq
l 0 til 0,n vl 0 j,nq
til,n ← til,n,(3.101) −1ÍÍ
0,n vl 0 j,nq
v
t
0
l
j,nq
il
j,q
l
v
u
tÍ
 −2
Í
2
i |yi j,nq | til,nl 0 til 0,n vl 0 j,nq
vl j,nq ← vl j,nq.
(3.102) −1ÍÍ
i til,nl 0 til 0,n vl 0 j,nq
However, the clustering result fluctuates depending on the initial values ofthe variables. To avoid this problem, I used IVA and the clustering method
described in Sect. 3.7.3 to obtain initial value of demixing matrix Wi .

3.7.5

Experiments and Results

Conditions
To confirm the efficacy of the proposed algorithm, similar to Sect. 3.6.3, I
conducted an evaluation experiment using professional music signals. In thisexperiment, I produced observed signals with M = 4 channels and N = 2 sources

3.7 Extension of ILRMA for Overdetermined and Reverberant Recording 95

Table 3.12: Music sources used in experiment of reverberant signals
ID
1
2
3
4

Song nameSource (1/2)
bearlin-roads
acoustic_guit_main/pianotamy-que_pena_tanto_faz
guitar/vocals
fort_minor-remember_the_namedrums/vocals
ultimate_nz_tour
guitar/vocals

Table 3.13: Characteristics of each method used in experiment of reverberant
signals
MethodNumber of filters per source
PCA+IVA
1
PCA+ILRMA
1
Sawada’s MNMF w/o MWF
1Sawada’s MNMF
1
Ideal linear filter
1
Proposed IVA
2
Proposed BSILRMA
2

Postfilter
None
None
None
MWF
None
None
None

by convoluting the impulse response JR2 (see Fig. 3.35) [202] with each source.
Table 3.12 shows the songs and sources used in this experiment, which wereobtained from SiSEC2011 [122]. I compared IVA with PCA (PCA+IVA) and
ILRMA with PCA (PCA+ILRMA), which both assume the linear time-invariant
mixing (rank-1 spatial model). In addition, two types of Sawada’s MNMFs [84]were also evaluated: Sawada’s MNMF w/o MWF and Sawada’s MNMF. In Sawada’s
MNMF w/o MWF, the maximum SNR BF [22], which is calculated from the(s)estimated spatial covariance Ri,n, was used for separation. Sawada’s MNMF is
the same method proposed in [84], which utilizes MWF [197] to enhance theestimated sources. As the proposed methods, spatial-model-relaxed IVA withthe clustering method in Sect. 3.7.3 (Proposed IVA) and spatial-model-relaxed
BSILRMA (Proposed BSILRMA) were evaluated, where the pretrained andclustered demixing matrix was used for the initial value in Proposed BSILRMA.
Moreover, I evaluated the limit separation performance of linear filtering (IdealChapter 3. Determined and Overdetermined Blind Source Separation Based96
on Independent Low-Rank Matrix Analysis

Table 3.14: Experimental conditions used in experiment of reverberant signals
Sampling frequency
Downsampled from 44.1 kHz to 16 kHz
Window length in STFT
128 msWindow function
Hamming window
Window shift length
64 ms
Number of bases
L = 15 (K = 30)
Maximum delay in time frameτmax = 2
Number of iterations
200

linear filter) as a reference performance, which is the maximum SNR BF calculatedusing the ideal (oracle) spatial covariances of each source. It is necessary toapply the back-projection technique (3.14), except for in Sawada’s MNMF,
to the estimated sources. The characteristics of each method are shown in
Table 3.13 and the other conditions are described in Table 3.14. Note that I
used a 128-ms-long window in the STFT for the signals that have 470-ms-longreverberation, which means that the rank-1 spatial model collapses. As theevaluation scores, I used the SDR improvement.
Results
Figures 3.36–3.39 shows the average scores and their deviations in 10 trials withdifferent pseudorandom seeds. The methods using PCA cannot achieve good
separation because they require the assumption of rank-1 spatial model. The
scores of Sawada’s MNMF w/o MWF indicate poor separation accuracy andstrong dependence on the initial values. However, MWF with NMF variables(the scores of Sawada’s MNMF) can greatly enhance the estimated sources.
Proposed BSILRMA separates the sources with high accuracy. In particular, thismethod outperforms the limit performance of linear filtering (Ideal linear filter)as shown in Figs. 3.38 and 3.39. This is because ground truth sources include
reverberations, which can span more than two dimensional space, and the
proposed algorithm can effectively relax the constraint in rank-1 spatial model.
However, in Fig. 3.37, the proposed methods cannot separate sources because
guitar and vocals are split into one and three components, and the clustering3.7 Extension of ILRMA for Overdetermined and Reverberant Recording 97

12
SDR improvement [dB]10

Source 1
Source 2

8
6
4

2
0

-2
-4

PCA+IVA

PCA+
ILRMA

Sawada’s Sawada’s Ideal
Proposed Proposed
MNMF
MNMF linear filter
IVA
BSILRMA
w/o MWF

SDR improvement [dB]

Figure 3.36: Average SDR improvements for song ID1 used in experiment ofreverberant signals.

14
12
10
8
6
4
2
0
-2
-4
-6
-8

Source 1
Source 2

PCA+IVA

PCA+
ILRMA

Sawada’s Sawada’s Ideal
Proposed Proposed
MNMF
MNMF linear filter
IVA
BSILRMA
w/o MWF

Figure 3.37: Average SDR improvements for song ID2 used in experiment ofreverberant signals.

Chapter 3. Determined and Overdetermined Blind Source Separation Based98
on Independent Low-Rank Matrix Analysis

12
SDR improvement [dB]10

Source 1
Source 2

8

6
4

2
0

-2
-4

PCA+IVA

PCA+
ILRMA

Sawada’s Sawada’s Ideal
Proposed Proposed
MNMF
MNMF linear filter
IVA
BSILRMA
w/o MWF

Figure 3.38: Average SDR improvements for song ID3 used in experiment ofreverberant signals.

16
SDR improvement [dB]14

Source 1
Source 2

12

10
8

6
4

2
0

PCA+IVA

PCA+
ILRMA

Sawada’s Sawada’s Ideal
Proposed Proposed
MNMF
MNMF linear filter
IVA
BSILRMA
w/o MWF

Figure 3.39: Average SDR improvements for song ID4 used in experiment ofreverberant signals.

3.8 Summary

99

Table 3.15: Computational times for separation of song ID3 used in experimentof reverberant signals (s)PCA+IVA

PCA+ILRMA

Sawada’s MNMF

23.4

29.4

3611.8Proposed
IVA
60.1

Proposed
BSILRMA
143.9

cannot divide these components as the individual sources.
Table 3.15 shows actual computational times for the separation of song ID3,
where the calculations were performed using the same environment as theresults in Table 3.10. The computational time of Proposed BSILRMA includes theinitialization time for Wi , which is the same as that of Proposed IVA. We confirm
that Proposed BSILRMA can maintain efficient optimization and achieve good
separation performance.

3.8

Summary

In this chapter, I proposed a new efficient determined BSS technique, ILRMA,
that extends a source model in IVA from a vector to a low-rank matrix using
the NMF representation. Also, the relationship between conventional MNMF
and IVA was revealed: ILRMA is equivalent to MNMF with a rank-1 spatialmodel, and time-varying Gaussian IVA can be thought of as a special case of
ILRMA, namely, ILRMA can be thought of as IVA with increased flexibility of themodel. ILRMA can be optimized using fast update rules based on the auxiliary
function technique. The experimental results show that ILRMA achieves
faster convergence and better results than the conventional BSS techniques. In
addition, an extension of ILRMA for overdetermined BSS with reverberant
observation was proposed. This algorithm utilizes extra observations for
simultaneously modeling both direct and reverberant components in eachsource. This method can be considered as an effective relaxation of the constraint
in rank-1 spatial model. Owing to the source modeling using extra observations,the proposed method can exceed a limit performance of linear separation filter

Chapter 3. Determined and Overdetermined Blind Source Separation Based100
on Independent Low-Rank Matrix Analysis
for the reverberant signals in some cases.

101

4

Single-Channel Semi-Supervised
Source Separation Based on
Discriminative Nonnegative Matrix
Factorization

4.1

Introduction

In this chapter, I address the single-channel supervised source separation, andpropose a new algorithm for discriminative basis training in a semi-supervised
situation. First, I review some existing methods for single-channel source
separation and explain a problem in supervised NMF methods, which is related
to discriminancy of supervised NMF bases. Next, conventional approached
for discriminative basis training in a full-supervised situation are reviewed,

102

Chapter 4. Single-Channel Semi-Supervised Source Separation Based on
Discriminative Nonnegative Matrix Factorization

then the motivation and strategy for discriminative training in semi-supervised
situation are clarified. After giving an explanation of the proposed algorithm, its
performance of music source separation is experimentally confirmed. Finally,
the whole contents in this chapter are summarized.

4.2

Existing NMF-Based Single-Channel Source Separation

A single-channel source separation task is one the most important problem
in acoustic signal processing field because it can be applied to a front-end
system for almost all acoustic applications. In contrast to the source separation
for multichannel signals, in this problem, spatial information cannot be used,
and a clue for solving the separation is only the difference of spectral featuresin each source, e.g., spectral patterns, spectral envelopes (linear prediction
cepstral coefficients [209, 210, 211]), and mel-frequency cepstrum coefficients(MFCC) [212, 213]. The problem can be formulated as a clustering or classificationof these extracted features.
In the past decade, NMF becomes the most popular approach for singlechannel source separation. Many techniques based on NMF have been proposed
and investigated so far, and it is still a growing research topic. In [214, 68, 70],
some regularizations including sparseness, temporal continuity (smoothness),
lower complexity, and better predictability, were introduced to the cost function
in NMF. These regularized cost function can be derived from a maximum
a posteriori estimation with a prior distribution of parameters, and it was
generalized as NMF with Bayesian estimation in [72]. We may solve theseparation problem only when such prior models fit to the inherent natureof the sources. As another approach, in some literatures, the structures ofsources were modeled with NMF decomposition. In [67, 75], shifted NMF wasproposed. They assumed that the timbre of a musical note produced by the same
instrumental source is constant for the entire range of pitch, and the sourcewise
NMF basis can be shifted to represent all the notes of the same source. Similar
idea was introduced in [124], but they utilized average spectral patterns for the4.3 Conventional Supervised NMF and Discriminative Training ofSupervised Bases

103

clustering criteria of NMF bases. Also, in [73, 128, 76], a source filter model or
MFCC and its estimation were exploited for the clustering. In [69], the NMF
bases were extended to have two dimensions (frequency bins × very short timeframes) as spectral fragments. Since such two-dimensional bases should be
convolved to approximately represent the observed matrix, this method iscalled nonnegative matrix factor deconvolution. In [71, 74, 126, 127], a Markov
chain model was introduced to NMF bases. These methods mainly aim tomodel the temporal (continuous) fluctuation or variation. For example, a pianonote is accurately characterized by a succession of several spectral patterns
corresponding to “attack,” “sustain,” “decay,” and “release” segments. Also, amusical vibrato can simply be represented by the Markov chain model.
To learn these sourcewise structures, patterns, or natures, a supervised
approach is very effective and has a big potential to achieve better separation
performance. The simplest way of supervised method is to prepare thesourcewise basis matrix, which can be obtained by independently applying
simple NMF to the training signal for each source. This approach is calledsupervised NMF [7, 131, 98], and for music signals, sample sequential notes(tones) with wide range of pitch (e.g., two or more octaves) can be a goodtraining signal for the instrumental sources. In particular, supervised NMF
using training signals for all the sources is called FSNMF, and the other approach(e.g., preparing the training signals for only the target source) is called SSNMF.
The detailed algorithms of these methods are described in the following section.

4.3

Conventional Supervised NMF and Discriminative Training of Supervised Bases

4.3.1

Conventional Supervised NMF

The algorithms in FSNMF and SSNMF are depicted in Fig. 4.1. In FSNMF,
we prepare the supervised basis matrix for all the sources in the observed
mixture. Let ∆ ∈ RΦ×Ψ
be a power or an amplitude spectrogram that includes two≥0
Φ×Ψ
sources, S1 ∈ R≥ 0 and S2 ∈ RΦ×Ψ
≥ 0 . Here, note that two mixture spectrograms ∆104

Chapter 4. Single-Channel Semi-Supervised Source Separation Based on
Discriminative Nonnegative Matrix Factorization
Training stage,
Separation stage

Fixed

Fixed

(a)
Training stage

Separation stage

Fixed

(b)
Figure 4.1: Training and separation stages in (a) FSNMF and (b) SSNMF.

and S1 + S2 are generally not identical because they are the amplitude or power
spectrograms, and the addition S1 + S2 do not take the phase cancellation into
account. ∆ is the amplitude or the power spectrogram of the waveform that is
an addition of two time-domain waveforms of S1 and S2 . Now, we prepare12
the training signals for both sources as S1(train) ∈ RΦ×Ψand S2(train) ∈ RΦ×Ψ
. In the≥0≥0
1
training stage, we estimate the sourcewise supervised basis matrices F1 ∈ RΦ×K
≥02
and F2 ∈ RΦ×K
(dictionaries of spectral patterns for each source) by performing≥04.3 Conventional Supervised NMF and Discriminative Training ofSupervised Bases105

the following NMFs:(F1, C1 ) = arg min D β S1(train) kF1 C1 ,

(4.1)



(F2, C2 ) = arg min D β S2(train) kF2 C2 ,(4.2)

F1, C1F2, C2where these minimization are carried out under a nonnegative constraint for
all the variables (hereafter, all the minimizations in this chapter include thenonnegative constraint). The activation matrices C1 ∈ RK≥ 01×Ψ1 and C2 ∈ RK≥ 02×Ψ2 canbe discarded after the training stage. In the separation stage, we decompose themixture ∆ using fixed F1 and F2 as(G1, G2 ) = arg min D β (∆kF1 G1 + F2 G2 ).

(4.3)

G1, G2

Therefore, we expect that the sources in ∆ will be separated by the estimatedactivation matrices G1 ∈ RK≥ 01×Ψ and G2 ∈ RK≥ 02×Ψ as S1 ≈ F1 G1 and S2 ≈ F2 G2 .
Similarly to (2.24), the update rules for these activations can be derived from theminimization of (4.3) as(
G1 ← G1 ◦G2 ← G2 ◦ ) .ϕ(β)
F1T ∆ ◦ (F1 G1 + F2 G2 ).−2

F1T (F1 G1 + F2 G2 ).−1( ) .ϕ(β)
F2T ∆ ◦ (F1 G1 + F2 G2 ).−2
F2T (F1 G1 + F2 G2 ).−1

,

(4.4)

.

(4.5)

In SSNMF, we only focus on the extraction of the target source from theTmixture. Similarly to FSNMF, the supervised basis matrix F ∈ RΦ×K
is obtained≥00using a training signal of the target source S (train) ∈ RΦ×Ψ
as
≥0


(F, C) = arg min D β S (train) kFC .
F, C

(4.6)

106

Chapter 4. Single-Channel Semi-Supervised Source Separation Based on
Discriminative Nonnegative Matrix Factorization

In the separation stage, ∆ is decomposed using fixed F as
(G, H, U) = arg min D β (∆kF G + HU),

(4.7)

G, H, UN
and U ∈ RK≥ 0N×Ψ are the basis and activation matrices for thewhere H ∈ RΦ×K
≥0
non-target sources. Thus, the target source and non-target sources are ideallyseparated into F G and HU, respectively. The update rules for G, H, and U are
obtained as follows:( ) .ϕ(β)
F T ∆ ◦ (F G + HU).−2,
F T (F G + HU).−1() .ϕ(β)∆ ◦ (F G + HU).−2 U T
H←H◦,(F G + HU).−1 U T( ) .ϕ(β)
H T ∆ ◦ (F G + HU).−2
U ←U◦.
H T (F G + HU).−1
G←G◦

(4.8)

(4.9)

(4.10)

General NMF-based source separation including FSNMF and SSNMF
does not ensure the physically accurate signal decomposition because theadditivity of the amplitude or the power spectrograms is not valid and the
NMF decomposition is an approximation. Also, the phase information of eachsource cannot be obtained, and the observed phase (noisy phase spectrogram) is
often utilized to perform an inverse STFT. However, even though there are suchdrawbacks, FSNMF and SSNMF are still a powerful method and are used formany situations.
As another approach for obtaining the supervised basis matrices, exemplarbased NMF was proposed [215, 216, 217]. In this method, every supervised basis
corresponds to an observation in the training data, namely, the training stage isjust a sampling of several spectra from different time frames for each source. Thisapproach becomes popular for a large-scale training of audio signals because it
does not require a large computational cost in the training stage. However, forthe music signals, it is hard to prepare such large-scale training dataset for all thepossible instruments and vocals. Thus, the separation technique that requires

4.3 Conventional Supervised NMF and Discriminative Training ofSupervised Bases107

only the small dataset is desired for supervised music source separation.

4.3.2

Drawback in Supervised NMF and Motivation for Discriminative Basis Training

These supervised methods have the potential to achieve high separation
performance. However, in both FSNMF and SSNMF, the cost function (4.3) or(4.7) in the separation stage represents how well the NMF model approximates∆, which does not include any criteria regarding the degree of separation, andthe pretrained supervised bases may represent not only the relevant source but
also a part of the other sources. For example, in SSNMF, F may represent a partof the non-target source spectra that should originally be absorbed in H, and thisphenomenon markedly degrades the separation quality. The dominant causeof this problem is that the supervised bases are independently trained using
isolated training signals of each source as (4.1) and (4.1), or (4.6), and there
may be some spectral overlaps between these training signals even if they are
inherently different sources.
In the context of FSNMF, several methods have been proposed to solve thisproblem [134, 135, 8, 9, 10]. In [134], the cross-coherence of the bases was added
to the cost function. In [135, 8, 9, 133], a sample mixture signal obtained bymixing the sample sounds of each source was utilized in the training stage
to estimate more discriminative sourcewise basis matrices. Also, the authorsin [10] proposed a joint optimization of NMF and a classification problem,
where the NMF variables are optimized so that each basis is classified into one
source during the training. Since these algorithms aim to estimate or train adiscriminative supervised bases, they are called discriminative NMF. However,the conventional approaches are only applicable to the full-supervised situation
because it is difficult to train such discriminative bases in semi-supervised
situation. In SSNMF, a penalized SSNMF that forces the non-target bases H
to be different from the target bases F was proposed [218, 98]1, but the target
bases F may still represent non-target source components. For this reason,
1After [218] was submitted, a similar penalty was independently proposed in [134] for
FSNMF, but [218, 98] also include another type of penalty.

108

Chapter 4. Single-Channel Semi-Supervised Source Separation Based on
Discriminative Nonnegative Matrix Factorizationin this dissertation, I address the discriminative training of supervised basisthat is applicable not only for FSNMF but also for SSNMF, and propose a newalgorithm for achieving the same objective in the conventional discriminative
NMF.

4.3.3

Algorithm of Discriminative Basis Training for FSNMF

We here review the discriminative NMF in the full-supervised situation. As
discriminative basis training for FSNMF, the following bilevel optimization [219]has been proposed [8, 9] (in which a two-source case was considered):C1∗, C2∗ = arg min D β
C1, C2(train)Smix
kF1 C1 + F2 C2
F1∗, F2∗ = arg min D β
F1, F2S1(train) kF1 C1∗







+ Reg(C1, C2 ),+ D β S2(train) kF2 C2∗,(4.11)(4.12)(train)where Smix
≈ S1(train) + S2(train) (addition in the time domain) and Reg(C1, C2 ) is aregularization term for the activations corresponding to, for example, sparsenesscriteria. Note that the cost (4.12) depends on the minimizers C1∗ and C2∗ of (4.11);
thus, they are functions of F1 and F2 . Hence, (4.12) is a bilevel optimization
problem since the basis matrices appear in both levels. This optimization finds
the basis matrices by taking into account the reconstruction of each source,
S1(train) and S2(train) . Therefore, the obtained bases tend to be discriminative.

The authors in [9] also mentioned that the basis matrices F1 and F2 in (4.11)
and (4.12) do not have to be the same, making the method in [9] a generalized
discriminative NMF. However, it is more challenging to simultaneously optimize
the different bases in (4.11) and (4.12). Thus, they simply used the independently
trained F1 and F2 in (4.11) and obtained different F1 and F2 in (4.12).

4.4 New Algorithm for Discriminative Basis Training

4.4

109

New Algorithm for Discriminative Basis Training

4.4.1

Strategy

In the proposed method, I only focus on SSNMF. With a similar motivation to [9],I propose the training of two types of supervised basis matrix for one source
without bilevel optimization because it is difficult to find supervised bases thathave reconstructive and discriminative spectra simultaneously. To maximize thediscrimination from other sources, the supervised bases should consist of uniquespectral components of the target source, which ideally do not overlap withthose of the other sources. For example, inharmonic components are significant
cues in distinguishing piano spectra from other instrumental sounds. Suchdiscriminative bases (hereafter, referred to as discriminative bases F 0) should only
be used to estimate the activations G of the target source in the separation stage.
After the separation, the target source can be reconstructed with G and thereconstructive bases (hereafter, referred to as reconstructive bases F).
Figure 4.2 depicts the conceptual difference between conventional andproposed algorithms of separation. In this figure, the target source (colored inblack) is subjected to interference by the non-target source (colored in gray),where the fundamental frequency components overlap. When the discriminativebasis F 0 consists of unique components of the target source as shown in Fig. 4.2(b), the discrimination (finding the correct activation G of the target source)
becomes easier than in the case when the supervised basis consists of all thecomponents of the target source. The fundamental components in the target
source, which are missing in F 0, are represented by another non-target basis
H2 in the NMF decomposition. Since the activation vectors G are correctly
estimated in the proposed algorithm, we can reconstruct the target source using
reconstructive basis F as F G.
In conventional SSNMF, the supervised bases F for the target source S
are trained using a sample sound S (train) of the target source. However, totrain the discriminative bases F 0, we here prepare a simulative mixture as(train)Smix≈ S (train) + N (train) , where N (train) is a sample sound of the non-target

110

Chapter 4. Single-Channel Semi-Supervised Source Separation Based on
Discriminative Nonnegative Matrix Factorization

Conventional separation

Time

Frequency

Time

Amp.

Time

Amp.

Amp. Amp.

Amp.

Amp. Amp.

Supervised bases
Separated signal

Frequency

Frequency

Frequency

Mixture signal

Frequency

Frequency

Mixture signal

Frequency

Supervised bases

TimeSpectral overlaps(a)Proposed separation with discriminative bases
Frequency

Unique component oftarget source

Time
Time
Time

Frequency

Separated signal

Amp.

Time
Amp.

Time

Reconstructive bases
Frequency

Frequency

Amp. Amp. Amp.

Amp. Amp. Amp.

Amp.

Frequency

Frequency

Frequency

Mixture signal

Mixture signal

Frequency

Discriminative bases(b)
Figure 4.2: Difference between (a) conventional and (b) proposed algorithms
of separation, where black components correspond to target source and graycomponents correspond to interfering source. Proposed method utilizes
discriminative bases (F 0) that has unique component of target source forseparation, and target source is synthesized using reconstructive bases (F) thathas complete spectral components.

sources and the addition is performed in the time domain. Although we do notknow the non-target sources in the semi-supervised scenario, we can collectpossible candidates such as different instruments for the target source. Note that(train)the mixture Smix
is utilized to train the discriminative bases F 0, namely, for theestimation of the unique component in the target source spectra, and the bases
for N (train) are never used in the separation stage.

4.4 New Algorithm for Discriminative Basis Training

4.4.2

111

Discriminative and Reconstructive Basis Training in SSNMF

Since the role of the discriminative bases F 0 is to obtain the accurate activations,
we would like to find F 0 such that the following C ∗ and C 0 become equivalent:

C = arg min D β S∗(train)

kFC ,(4.13)

F, C

C

0



(train)= arg min D β Smix
kF 0 C + T V
C, T,V

.

(4.14)

This is also a bilevel optimization problem, and it is challenging to optimize F 0
using this criterion. Instead, I here propose the following simple optimization
for training both the discriminative and reconstructive bases F 0 and F:(F, C ∗ ) = arg min D β S (train) kFC ,(4.15)F, C(train)(F 0, T, V ) = arg min D β Smix
kF 0 C ∗ + T V ,F 0, T, V(4.16)

where T and V are new NMF matrices representing the non-target sources(train)included in Smix
. The reconstructive bases F and their activations C ∗ are
obtained by (4.15). Then, we initialize the discriminative bases as F 0 ← F andcalculate the optimization (4.16) with fixed C ∗ . Since C ∗ is the true activation of(train)the target source in Smix
, F 0 is trained such that the activations of the target
source obtained by NMF with F 0 applied to the mixture are as close to C ∗ as
possible. In the separation stage, the mixture ∆ is decomposed using fixed F 0 as
(G, H, U) = arg min D β (∆kF 0 G + HU).

(4.17)

G, H, UThe update rules for (4.15)–(4.17) are the same as those described in Sects. 2.5
and 4.3.1 with the replacement of the corresponding variables. After the
separation stage, the estimated target source Ŝ can be reconstructed as F G or

112

Chapter 4. Single-Channel Semi-Supervised Source Separation Based on
Discriminative Nonnegative Matrix Factorization

using Wiener filtering asŜ ≈FG
F 0 G + HU

◦ ∆.

(4.18)

In [9], the authors refined the bases used for reconstruction after first fixing thebases used in the separation. In this paper, we refine the discriminative bases
using the fixed reconstructive bases.

4.5

Experiments

4.5.1

Simple Experiment Using Piano and Flute Tones

In this subsection, I confirm how the proposed algorithm works using simple
audio data of piano and flute tones provided by the MIDI tone generator Garritan
Personal Orchestra 4. The sample sound S (train) contains only a piano tone (C5)and N (train) contains only a flute tone (C6). Figure 4.3 shows the NMF bases,
which are independently trained for the piano and flute tones using a single
basis.
After initializing F 0 as shown in Fig. 4.3 (a), I calculated (4.16) with C ∗ andrandomized T and V , where the number of bases in T was set to two. Figure 4.4
shows the estimated F 0 and the bases in T after 50 iterations of the update rules
for optimizing (4.16). From Fig. 4.4 (a), we can confirm that spectral notches
appear at the second, sixth, and eighth peaks in F 0 and do not overlap with thepeaks in the flute tone, and such lost peak components are compensated by
the other basis as shown in Fig. 4.4 (c). From these results, I consider that the
proposed algorithm can train the unique components of the target source, as
shown in Fig. 4.2, to some extent.

4.5.2

Music Source Separation

Conditions
We compare the separation performance between simple SSNMF and the
proposed method in the music separation task. We used three songs and six

4.5 Experiments

113

Power [dB]

20

-20

-60

-100
0

1

2
3
Frequency [kHz]

4

5

4

5

(a)

Power [dB]

20

-20

-60

-100
0

1

2
3
Frequency [kHz](b)
Figure 4.3: Spectral bases obtained from simple NMF: (a) C5 piano tone and (b)C6 flute tone.

mixtures obtained from SiSEC2011 [122] (see Table 4.1). Each song consists of
two sources, and 4-fold cross-validation was applied to each song to obtain a
training sound of the target source S (train) and a test mixture ∆ ≈ S (test) + N (test) ,where addition was performed in the time domain. More precisely, the training
sound of the target source S (train) was obtained from three-quarters of a source ina song, and the remaining quarter S (test) was used to obtain a mixture ∆. The
sample sound of the non-target source N (train) was obtained from a different
song as shown in Table 4.2. For example, in the ID3 data, the mixture ∆ waspart of the song “Que pena tanto faz” and include the classic guitar sound

114

Chapter 4. Single-Channel Semi-Supervised Source Separation Based on
Discriminative Nonnegative Matrix Factorization

Power [dB]

20

-20

-60

-100
0

1

2
3
Frequency [kHz]

4

5

4

5

4

5

(a)

Power [dB]

20

-20

-60

-100
0

1

2
3
Frequency [kHz](b)Power [dB]

20

-20

-60

-100
0

1

2
3
Frequency [kHz](c)Figure 4.4: Spectral bases obtained from (4.16), where (a) is discriminative basis
F 0 of piano tone and (b) and (c) are the other bases in T .

4.5 Experiments

115

Table 4.1: Mixture ∆ with target and non-target sources
ID
1
2
3
4
5
6

Song nameof mixture ∆
Roads
Roads
Que pena tanto faz
Que pena tanto faz
Ultimate NZ tour
Ultimate NZ tour

Target source

S (test) and S (train)
Acoustic guitar
Drums
Classic guitar
Female vocals
Electric guitar
Synthesizer

Non-target source
N (test)Drums
Acoustic guitar
Female vocals
Classic guitar
Synthesizer
Electric guitar(train)Table 4.2: Sample sounds of non-target source N (train) for preparing Smix

ID
Song name1
The ones we love2
The ones we love
3 Remember the name4
Ultimate NZ tour
5 Remember the name
6
Roads

Sample sound N (train)
Drums
Acoustic guitar
Male vocals
Electric guitar
Synthetic violins
Acoustic guitar

S (test) and female vocals N (test) . The sample sound S (train) was the same classic
guitar sound obtained from a different segment of S (test) in the same song, andthe sample sound N (train) was obtained from the different song “Remember
the name.” In addition, since N (train) is a male vocal sound, FSNMF does notwork well in this situation. However, this experimental setting is convenient forevaluating the proposed method because we assumed that a similar type (butnot exactly the same) of source to N was available as the sample sound N (train) .
(train)In the proposed algorithm, preparing N (train) or Smix
is a very important issue,
and this experiment simulates the case that we know only the type of non-target
source in the observed mixture ∆.
The spectrograms were computed by an STFT with a 92-ms-long Hamming
window and half-size shifting. In this experiment, all NMF decompositionis performed with amplitude spectrograms obtained via STFT, and as the

116

Chapter 4. Single-Channel Semi-Supervised Source Separation Based on
Discriminative Nonnegative Matrix Factorization

divergence criterion in NMF, we set β to 1, namely, NMF based on KL divergence(KLNMF). The numbers of bases in F, F 0, T , and H were set to 35, and thenumbers of iterations in the training ((4.6) or (4.15)) and separation ((4.7) or (4.17))
stages were set to 1000. As the number of iterations in (4.16), we investigatednumbers from 0 to 50, where 0 corresponds to simple SSNMF because thediscriminative bases F 0 are equal to the reconstructive bases F. As the separation
performance, we used the improvement of SDR.
Results
Figure 4.5 shows the average SDR improvements of the target source. The zero
point of the horizontal axis corresponds to simple SSNMF. As shown in Fig. 4.5,
the proposed algorithm converges after about 20 iterations and the separation
performance was improved at the point of convergence in all cases except ID6.
Moreover, most of the improvement was obtained in the first four iterations.
This indicates that better discriminative bases F 0 exist than F but they are notobtained at the point of convergence of the proposed method. We consider that
this may be caused by the fact that we do not solve the bilevel optimization
described in the beginning of Sect. 4.4.2. In a future work, we will investigate
how to obtain better discriminative bases as a result of optimization.

4.6

Summary

In this chapter, I proposed a new basis training method for SSNMF. The proposed
algorithm estimates both types of supervised basis matrix, namely, discriminativeand reconstructive bases, for the target source and can be considered as an
approximation to solving bilevel optimization to find discriminative bases.
The efficacy of the proposed method was confirmed by performing a music
separation.

4.6 Summary

117

Results of ID1
Results of ID4

Results of ID2
Results of ID5

Results of ID3
Results of ID6

18
16

SDR improvement [dB]

14
12
10
8
6
4
2
0

0

5

10

15 20 25 30 35 40
Number of iterations in (4.16)

45

50

Figure 4.5: Average SDR improvement for each mixture and each number of
iterations in (4.16).

119

5

Effective Initialization forNonnegative Matrix Factorization
Based on Statistical Independence

5.1

Introduction

In this chapter, I deal with an initialization problem for NMF. As alreadydiscussed so far, NMF is a powerful unsupervised learning method that extracts
meaningful nonnegative features from an observed nonnegative matrix, which
is not only the audio spectrogram but also many data. Various applications
using NMF have been proposed as described in Sect. 2.5. However, the result ofsuch applications always depends on the initial values of the NMF variables
because of the existence of local minima. To solve this problem, in this chapter, I
propose new initialization methods based on statistical independence between
NMF components. First, existing conventional initializations for NMF are

120

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

introduced with the viewpoint of presence of random values. Next, after the
motivation is clarified, the efficient initialization method based on ICA. To takethe nonnegativity in NMF into account, I propose two types of algorithms: (i)applying nonnegative ICA (NICA) [220, 221, 222] to the observed data matrix; (ii)applying simple ICA with zero-mean Laplace prior to the differential of observed
data matrix using nonnegative projection in each update. The convergence speed
and the converged value of NMF cost function is compared with conventional
and proposed methods. Also, the availability of the proposed initialization forsource separation is experimentally investigated via FSNMF, BSS based on
ILRMA proposed in Chap. 3, and discriminative SSNMF proposed in Chap. 4.
Finally, the contents in this chapter are summarized.

5.2

Conventional NMF Initializations

Similar to Sect. 2.5, let ∆ ∈ RΦ×Ψ
be the observed nonnegative matrix, F ∈ RΦ×K
and
≥0
≥0
K×Ψ
G ∈ R≥ 0 are the nonnegative basis and activation matrices, Φ is the dimension
in observation, Ψ is the number of observed data samples, and K is the numberof bases. Also, δψ , f k , and g k denote the vectors in the matrices ∆, F and G,
respectively, namely, ∆ = (δ1, · · · , δψ ), F = ( f1, · · · , f K ), and G = (g1, · · · , gK )T .
The optimization method in NMF, e.g., MU rules (2.20) and (2.21), requires
initial variables F (ini) and G(ini) . Then, a neighborhood local minimum can beobtained as a solution of the cost function (2.18). Figure 5.1 shows an example
of full-supervised music source separation using FSNMF, where the NMF
variables are initialized using pseudorandom values obtained from various
pseudorandom seeds (hereafter, referred to as Rand1 to Rand10). We can confirm
that the separation performance strongly depends on the NMF initialization.
“Good” initial values for NMF are defined as follows [223]: (i) those that leadto rapid minimization of the divergence and fast convergence; (ii) those thatlead to low overall divergence at the point of convergence. In this chapter, I
concentrate on the both of these objectives.

5.2 Conventional NMF Initializations

121

10
8
6

4

Rand10

Rand9

Rand8

Rand7

Rand6

Rand5

Rand4

Rand3

0

Rand2

2

Rand1

SDR improvement [dB]12

Figure 5.1: Example of SDR improvements of music source separation, where
FSNMF initialized by random values with various pseudorandom seeds is
performed.

5.2.1

Initialization with Random Values

The simplest initialization method is randomization, namely, we prepare(F (ini), G(ini) ) by producing pseudorandom values. This method leads various
results depending on the random seed. Therefore, the best result should be
adopted via many trials with different seeds. In [224, 225], a genetic algorithm
was utilized to find good initial values. In addition, several initialization methods
based on clustering of the data matrix ∆ have been proposed [226, 227, 228]. The
methods in [226, 227, 228] utilized the result of clustering, i.e., centroid vectors,
to define F (ini) , but they required initial centroid vectors, which were usually
determined as random values.

5.2.2

Initialization without Random Values

The authors in [229] proposed the use of subtractive clustering, which does
not require initial centroids, namely, a unique result can be obtained for NMF
decomposition. However, the subtractive clustering includes two hyperparame-

122

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

ters that must be heuristically tuned by the users. As initialization methods
that do not require both random values and hyperparameter tuning, PCA andsingular value decomposition (SVD) have been utilized [230, 223]. In the former
method [230], orthogonal bases and their weights obtained by applying PCA
to ∆ are assigned to (F (ini), G(ini) ), where the negative entries are replaced bytheir absolute values. In the SVD-based method [223], nonnegative double
SVD (NNDSVD) is applied to ∆. The initial values (F (ini), G(ini) ) are set to thenonnegative left and right singular vectors obtained via NNDSVD. These two
methods can provide a unique decomposition.

5.3

Efficient NMF Initialization Based on ICA

5.3.1

Motivation and Strategy

The initialization methods using PCA or SVD are based on the orthogonality
between the bases representing the data matrix ∆. However, it has been shownthat the optimal NMF bases are along the edges of a convex polyhedral cone, which
is defined by the observed points in ∆, in an Φ-dimensional space [231, 232].
Figure 5.2 shows the various NMF bases when Φ = K = 2. The optimal bases are
satisfactory for representing all the data points, whereas the close bases cannot
represent them because of the nonnegative constraint of the activations. The
orthogonal bases are excessive for representing the data points and have a risk to
represent even a meaningless area. Therefore, PCA and SVD may not be the best
methods for the initialization in NMF.
In this chapter, I propose the utilization of bases and independent sources
estimated by ICA for F (ini) and G(ini) , respectively. ICA can estimate nonorthogonal bases a k that provide a mixing matrix A = (a1, · · · , a K ) for
the independent sources as AS, where a k is the K × 1 kth ICA basis andS = (s1, . . . , s K )T , s k is the Ψ × 1 kth source signal. Thus, ICA can estimate basesso that the sources are independent of each other, and such bases tend to bedissimilar but they are not orthogonal. In addition, the estimated sources s k tend
to be sparse if we assume a super-Gaussian distribution as a source distribution
in ICA. When the coefficients are sparse, their bases will be along the edges of5.3 Efficient NMF Initialization Based on ICA

123

Optimal bases

Orthogonal bases

Close bases

(a)

(b)

(c)

Figure 5.2: Geometry of (a) optimal, (b), orthogonal, and (c) close bases, where
black dots indicate observed data points in positive orthant, gray area indicatescone defined by data points, broken lines indicate edges of cone, f k denotes kth
NMF basis, Φ = K = 2, and Ψ = 10.

the cone as shown in Fig. 5.2 (a). Therefore, by using the independent sources
and their bases for the initial values in NMF, the optimization may avoid local
minima. In fact, an initialization method for probabilistic latent component
analysis (PLCA) [233] based on ICA has been proposed [234], where PLCA
is inherently identical to KL-divergence-based NMF. However, the method
in [234] did not use the ICA bases a k but the demixing filters w k , which are theinverse of the ICA bases, W = (w1, · · · , wK )T = (a1, · · · , a K )−1 , and provide
the estimated sources y k . Also, the authors in [234] did not discuss how totreat the nonnegative entries in w k and y k . Moreover, there was no comparison
with other initializations such as the PCA-based method and NNDSVD. To
take the nonnegativity into account, I propose the employment of ICA for the
initialization in NMF. Also, the proposed method performs PCA before ICA as apreprocess for simulating the dimensionality reduction in NMF. To take thenonnegativity in NMF into account, I here propose two types of initialization
algorithms: (i) applying NICA [220, 221, 222] to the observed data matrix ∆;(ii) applying simple ICA with zero-mean Laplace prior to the differential ofobserved data matrix, ∆Θ, and applying nonnegative projection in each updateof ICA, where Θ is a differential matrix that takes difference between the data

124

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

point and its neighbor in each dimension φ, namely, δφψ − δφ(ψ+1) , as
© 1 0 0 ···­−1 1 0 · · ·­­Θ = ­­ 0 −1 1 · · ·­ ..
.. .. . .
­ .
.
. .
­
« 0 0 0 ···

5.3.2

0ª
0®®
®
0®® .
.. ®
. ®®
1¬

(5.1)

Combination of PCA and ICA

The dimensionality reduction for arbitrary nonnegative matrix X ∈ RΦ×Ψ
using
≥0
PCA can be represented as P 1 X = ASP2X ≈ 0


,

(5.2)

where
P1
P=
P2

!
(5.3)

is the Φ × Φ transform matrix of PCA and the sizes of P 1 and P 2 are K × Φ and(Φ − K) × Φ, respectively. The row vectors in P correspond to the eigenvectorsof a variance-covariance matrix XX T , and the eigenvectors are arranged indescending order from the first row to the last row on the basis of theireigenvalues. Therefore, P 1 includes the top K eigenvectors of XX T and P 2
includes the remaining eigenvectors. In addition, 0 is the (Φ − K) × Ψ zeromatrix. Thus, we assume that the independent sources in S are mixed via themixing matrix A and are observed as the mixture P 1 X. From the NMF side, thenonnegative activations are assumed to be independent of each other, as shownin Fig. 5.3. Note that since NICA will be applied to P 1 X (after the dimensionality
reduction via PCA), the estimated ICA bases a k are not orthogonal.

5.3 Efficient NMF Initialization Based on ICA

…

…

…

125

Figure 5.3: Assumption of proposed method, where nonnegative activations are
assumed to be independent of each other.

5.3.3

Proposed Initialization using NICA

NICA can estimate the nonnegative independent components from an observed
multichannel mixture. The essence of NICA is to find a rotation matrix W for thenoncentered and whitened data so that all the estimated (separated) sources
become nonnegative [220]:
Y = W Ω,

(5.4)

Ω = WP 1 X = W AS,(5.5)

where W is a whitening matrix, which transforms P 1 X so that P 1 X(P 1 X)Tbecomes the identity matrix, and X = ∆ in this method. Note that this whitening
process does not center the data, namely, it does not remove the mean of P 1 X.
In addition, Y = (y1, . . . , yK )T is a matrix that comprises of estimated sources y k ,and W is a demixing matrix that rotates the whitened data Ω. If the sources s k
are truly nonnegative, we can obtain a global solution such that all the estimatesy k become nonnegative. However, in the proposed method, such a global
solution probably does not exist because of the dimensionality reduction via
PCA. The optimization in NICA is defined as the minimization of the total
power of the residual negative estimates [220]:
min
W

Õ
k,ψ

min(0, y kψ )2,

(5.6)

126

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

where y kψ is the entry of Y . The steepest gradient descent has been proposed for(5.6) as follows [221]:w˜k = w k − 2ηÕmin(0, ω kψ )ω kψ,

(5.7)

ψ


 −1/2
W = W̃ W̃ T
W̃,

(5.8)

where w k and w̃ k are the column vectors of W and W̃ , respectively, η is thestepsize parameter, ω kψ is the entry of Ω, and W̃ is the matrix with w k as its
columns. Whereas optimization without a hyperparameter such as η has also
been proposed as “fast NICA” [222], I use (5.7) and (5.8) in this dissertation.
The estimated sources Y can be used for the initial values of the activation
matrix G. Also, the basis matrix F can be calculated from the estimated demixing
matrix W . If we approximately assume X = F G, S = Y , and A = (W W)−1 , thefollowing equation can be obtained from (5.2):"
PF G ≈(W W)−1

#

0

G.

(5.9)

Then, the basis matrix F can be obtained as
"

F≈P

5.3.4

−1
−1 (W W)0#.

(5.10)

Proposed Initialization using ICA and Differential of Data
MatrixWhen the source S and the observed data X have both positive and negativevalues, the regular ICA algorithm can be used for the estimation of W . Thus, I
also propose a utilization of ICA with differentiated data matrix ∆Θ. Whereas
we assumed X = ∆ and estimate S = G in Sect. 5.3.3, in this method, X = ∆Θ isassumed to estimate S = GΘ. I here apply ICA with Laplace distribution as thesuper-Gaussian source distribution because the ICA cost function with Laplace
distribution becomes convex with respect to W , and the unique solution can be5.3 Efficient NMF Initialization Based on ICA

127

obtained via optimization. In addition, the fast and stable optimization based on
auxiliary function technique has been proposed [176]. After the estimation of
W , the initial activation and basis matrices can be calculated as G = W ∆ (notG = W X) and
"

#
−1
W
F ≈ P −1.
0(5.11)In this method, there is no guarantee that the basis matrix F is a nonnegative
matrix. To ensure the nonnegativity, in each iteration of ICA optimization, I
propose to calculate F using (5.11), update as F ← max(F, 0) (projected to thenonnegative values), and recalculate W from the updated F.

5.3.5

Nonnegativization

Since we apply PCA for dimensionality reduction, there is no guarantee that
all the entries of the obtained activation matrix G become nonnegative. In
particular, the proposed method using ICA does not ensure the nonnegativity ofthe basis matrix F. For these reasons, I apply nonnegativization to the obtained F
and G by the proposed methods. I here perform any of the following threenonnegativizations:Nonnegativization 1 : F (ini) = |F |, G(ini) = |G|,T

Nonnegativization 2 : F (ini) = |F |, G(ini) = αG F (ini) ∆,T

Nonnegativization 3 : G(ini) = |G|, F (ini) = αF ∆G(ini) ,where αF and αG are coefficients for fitting the scale of F (ini) G(ini) to ∆. The
values of these coefficients depend on the following NMF after the proposed
initialization and can easily be calculated fromT
αF = arg min D β ∆kα∆G(ini) G(ini) ,α(ini) (ini) T
αG = arg min D β ∆kαF F∆ .
α(5.12)(5.13)

128

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

Here, I describe the solutions of (5.12) and (5.13) for the cases of NMF based on
EU distance (EUNMF), KLNMF, and ISNMF as follows:
For EUNMF : αF =(ini) (ini)ψ 0,k δφψ 0 gkψ 0 gkψ 0Í Í(ini) (ini) 2φ,ψ
ψ 0,k δφψ 0 gkψ 0 gkψ 0

Í

For KLNMF : αF = Í
1For ISNMF : αF = ΦΨφ,ψ δφψÍÍφ,ψ(ini) (ini) , αG = Íψ 0,k δφψ 0 gkψ 0 gkψ 0

ÍÍφ,ψ δφψ, αG =

δφψ
φ,ψ Í 0 δ 0 g (ini) g (ini) ,ψ ,k φψ kψ 0 kψ 0(ini) (ini)
φ 0,k fφ 0 k fφ 0 k δφ 0 ψ2Í Í(ini) (ini)φ,ψ
φ 0,k fφ 0 k fφ 0 k δφ 0 ψÍφ,ψ δφψÍφ,ψ δφψ(ini) (ini)
φ 0,k fφ 0 k fφ 0 k δφ 0 ψ

Í

φ,ψ

Í

1
αG = ΦΨÍ

φ,ψ Í 0,,δφψ(ini) (ini)

φ ,k fφ 0 k fφ 0 k δφ 0 ψ

,

(ini)
(ini)
are the entries of F (ini) and G(ini) , respectively.
and gkψwhere fφk

5.4

Experimental Comparisons

5.4.1

Performance as Initial Value for NMF

Conditions
To evaluate the performance of the proposed method, I compare the convergence
speed of the cost function in NMF among five initialization methods, namely,
uniform random values in the range between ε and 1, PCA-based initialization [230], NNDSVD [223], the proposed method using NICA, and the proposed
method using ICA with differentiated data matrix. As the observed data ∆, I
used a power spectrogram of the music signal “Actions - One Minute Smile”
obtained from the MSD100 dataset, which was published by SiSEC2015 [235].
MSD100 consists of four parts for each song, namely, vocals, bass, drums, andother. I here chose only the vocals and other sources. In addition, the section ofthese source signals from 40 s to 100 s was extracted to obtain 60-s-long vocals
and other signals, and the observed signal is a mixture of them. The power
spectrogram of the observed signal was computed by an STFT with a 92-ms-long
Hamming window and half-size shifting. The size of matrix ∆ was Φ = 2049 andΨ = 1290, and the sampling frequency was 44.1 kHz. After the initialization of
F and G using the conventional or proposed method, EUNMF, KLNMF, or
ISNMF was preformed, where the MU update rules (2.20) and (2.21) were used
to minimize (2.18), and the number of bases was set to K = 60. For the random5.4 Experimental Comparisons

129

1

1000

9
8

7
6

Cost function of ICA

Cost function in NICA

6
5
4
3
2

0.1
6
5
4
3

5
4

3

2

2

0.01

0

500

1000

Iteration

(a)

1500

2000

100

0

5

10

15

20

Iteration

25

30

(b)

Figure 5.4: Convergence of cost function in (a) NICA and (b) ICA.

initialization, I performed 10 trials with various pseudorandom seeds (Rand1 toRand10). Also, for the proposed methods, I compared six methods, namely, twoalgorithms (NICA and ICA) and three nonnegativizations (hereafter, referred toas NICA1 to NICA3 and ICA1 to ICA3, where the number corresponds to thetype of nonnegativization described in Sect. 5.3.5). The number of iterations for
optimization was set to 2000 in NICA1–NICA3 and 30 in ICA1–ICA3.

Results
Figure 5.4 shows the convergence of the cost function (5.6) in NICA and ICA.
Since I used the steepest gradient descent (5.7) and (5.8), more than 1000
iterations were required for NICA. This may be reduced by using fast NICA [222].
The convergence of ICA based on auxiliary function technique (Fig. 5.6 (b)) ismuch faster than that of NICA. However, since this algorithm includes thenonnegative projection in each iteration, the optimization does not ensure
monotonic decrease. In addition, the computational complexity of ICA is large

130

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

(a)

(b)
Figure 5.5: Convergences of cost function in EUNMF, where NICA1–NICA3 aredepicted in (a), ICA1–ICA3 are depicted in (b), and conventional methods are
depicted in both.

5.4 Experimental Comparisons

131

(a)

(b)
Figure 5.6: Convergences of cost function in KLNMF, where NICA1–NICA3 aredepicted in (a), ICA1–ICA3 are depicted in (b), and conventional methods are
depicted in both.

132

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

(a)

(b)
Figure 5.7: Convergences of cost function in ISNMF, where NICA1–NICA3 aredepicted in (a), ICA1–ICA3 are depicted in (b), and conventional methods are
depicted in both.

5.4 Experimental Comparisons

133

Table 5.1: Examples of computational time in each process (s)Process

Algorithm
NICA1–NICA3 (2000 iterations)
ICA1–ICA3 (30 iterations)
Initialization
PCA-based initialization
NNDSVDEUNMF (1000 iterations)NMF
KLNMF (1000 iterations)
ISNMF (1000 iterations)

Processing time
4.36
10.89
0.98
2.40
12.78
48.07
214.26

because it includes inverse calculation of K × K matrix W . Figures 5.5–5.7 show
the convergences of the cost function (2.18) in NMF, where NICA1–NICA3 aredepicted in (a), ICA1–ICA3 are depicted in (b), and the conventional methods
are depicted in both. From these results, we can confirm that the random
initialization has a slow convergence speed and a poor local minimum at the
point of convergence. PCA-based initialization provides faster and deeper
minimization than random initialization, especially for EUNMF and ISNMF.
NNDSVD has the best convergence except for ISNMF. The methods basedon NICA outperform these conventional methods in all the cases of NMF. In
particular, NICA2 provides the most stable convergence. The methods basedon ICA also provide comparable performance with PCA-based initialization
and NNDSVD. However, the best nonnegativization depends on the criteria in
NMF cost function. Examples of the computational time for each initialization
method and NMF are given in Table 5.1. Although the proposed methods havelarger computational costs than the other methods, the increase is not criticalcompared with the case of NMF iterations.

5.4.2

Full-Supervised Audio Source Separation

Conditions
I compare the performance of audio source separation using FSNMF. In thisexperiment, I chose the top 15 songs in alphabetical order of a test dataset in134

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

MSD100 and only the vocals and other parts as the source signals. Similarly
to in Sect. 5.4.1, 60-s-long vocals and other signals were used. The power
spectrograms of the vocals and other signals were calculated via STFT as ∆Vand ∆O , respectively. In addition, sixfold cross-validation was applied to them,
namely, the training sounds of each source (∆(train)and ∆(train)) were obtainedV
O
from five-sixths of the frames in ∆V and ∆O , and the remaining one-sixth of the(test)
frames, ∆(test)and ∆(test), were used to obtain a mixture ∆(test)+ ∆(test)ofmix ≈ ∆V
V
O
O
vocals and other source signals, where the mixing was performed in the time
domain. The supervised sourcewise basis matrices FV and FO were trained by
the following NMF:,

(5.14)



,(5.15)FV = arg min D β ∆(train)
kF G
V
F, G
FO = arg min D β ∆(train)
kF G
O
F, G

where F and G are initialized by F (ini) and G(ini) , respectively, and the nonnegative
constraint is assumed in this minimization. In the separation stage, the supervised
NMF was performed with fixed bases FV and FO under the nonnegative constraint
as follows:
min D β ∆(test)
kF
G
+
F
G
V V
O O ,
mix

GV, GO(5.16)

where the initial values of GV and GO were set to GV = αG FVT ∆(test)
mix and(test)T
GO = αG FO ∆mix , respectively. Therefore, the separation performance only
depends on the initial values F (ini) and G(ini) used in (5.14) and (5.15). I used
IS-NMF for both the training and separation stages. The number of iterations inboth stages was set to 200, and the numbers of bases in FV and FO were set to 50.
As a performance measure of the source separation, I used the improvement ofSDR. The other experimental settings were the same as those in Sect. 5.4.1.
Results
Figure 5.8 shows the average SDR improvements of the 15 songs. The proposed
methods achieve better separation performance than random initialization, as5.4 Experimental Comparisons

135

10
8
6

4

Rand2

Rand3

Rand4

Rand5

Rand6

Rand7

Rand8

Rand9

Rand10

Rand2

Rand3

Rand4

Rand5

Rand6

Rand7

Rand8

Rand9

Rand10

Rand1

SVD

PCA

ICA3

ICA2

ICA1

NICA3

0

NICA2

2

NICA1

SDR improvement [dB]

12

(a)

4

3
2

Rand1

SVD

PCA

ICA3

ICA2

ICA1

NICA3

0

NICA2

1

NICA1

SDR improvement [dB]5(b)Figure 5.8: SDR improvement of supervised NMF for (a) vocals and (b) other.

particularly clearly shown in Fig. 5.8 (b). In supervised NMF, it is important totrain the appropriate bases that represent only the corresponding source anddo not represent interfering components. This result suggests that the ICA
bases are preferable to the orthogonal bases, which can cover a wider convex
polyhedral cone as shown in Fig. 5.2 (b).

136

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence
Observed signal

IVA
Power spectrogramof separated signal

NMF
initialization
Initial NMF variables
ISNMF
Initial source model
ILRMA

Separated signal

Figure 5.9: Process flow of BSS based on ILRMA with NMF initialization method.

5.5

Application to ILRMA and Discriminative SSNMF

The separation performance of NMF-based source separation always depends
on the initial values of basis and activation matrices. Although the deep
minimization of NMF cost function does not guarantee good separation
performance, faster and stable optimization is preferable for many NMF-basedalgorithms. In this section, I investigate the availability of the proposed
initializations for BSS based on ILRMA proposed in Chap. 3 and discriminative
SSNMF proposed in Chap. 4.

5.5.1

BSS Based on ILRMA with Various Initialization for NMF

Figure 5.9 shows a process flow of initialization in ILRMA. Since the task is ablind situation, I first apply IVA to an observed multichannel signal xi j to obtain

5.5 Application to ILRMA and Discriminative SSNMF

137

Table 5.2: Averaged SDR improvements over various speech signals and sources
with same recording conditions, where these scores are obtained by ILRMA w/opartitioning function with various NMF initializations
Conditions(rev. time andmic. spacing)
130 ms & 1 m
130 ms & 5 cm
250 ms & 1 m
250 ms & 5 cm

NICA1 NICA2 NICA3

ICA1 ICA2 ICA3

5.13
5.52
2.71
5.89

12.40
6.17
4.81
5.74

4.13
5.67
2.06
5.40

5.18
5.69
3.24
5.61

7.07
6.08
3.37
5.85

12.63
4.67
5.76
4.73

PCA-based
initialization

NNDSVD

Randominitialization

10.27
6.54
3.10
5.75

10.21
5.79
2.26
5.04

11.91
8.97
7.34
6.43

a tentative separated signal yi j and its power spectrogram Pn . It is used for
determining the initial basis and activation matrices Tn(ini) and Vn(ini) for eachsource. Then, the initial source model can be obtained by simple ISNMF. When I
apply ILRMA, the demixing matrix Wi is reset to the identity matrix to avoid thepoor local solution.
I conducted the same experiment in Sect. 3.6.3. Tables 5.2 and 5.3 show
the result of average SDR improvement, where only ILRMA w/o partitioning
function is performed and compared with various NMF initialization methods.
The result of ILRMA using random initialization is the same as those in
Tables 3.6 and 3.7, namely, those are the average scores of 10 trials with different
pseudorandom seeds, and the other scores are the results of only the single trial.
From these results, we can confirm that the initialization of source model in
ILRMA does not lead better separation performance than the random source
model for both speech and music signals. The reason may be that the estimatesof IVA is one of the local minimum solution for ILRMA, and the pretrained
source model does not drastically change from its initial values in some signals.

5.5.2

Discriminative SSNMF with Various Initialization for NMF

I here conducted the same experiment in Sect. 4.5.2 with proposed discriminative
SSNMF, where the various NMF initialization methods are applied to (4.15) inthe training stage. The other NMF variables T , V , H, and U used in (4.16) and(4.17) are initialized by random values, but the same random values are used forChapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

138

Table 5.3: Averaged SDR improvements over various music signals and sources
with same impulse response, where these scores are obtained by ILRMA w/opartitioning function with various NMF initializations
ImpulseNICA1 NICA2
response
E2A
JR2

8.28
3.52

8.66
3.87

NICA3

ICA1 ICA2

ICA3

8.07
3.57

9.12
4.04

8.68
4.49

8.92
4.07

PCA-based
RandomNNDSVD
initialization
initialization
9.02
4.13

8.46
3.98

14.41
9.06

all the methods.
Figures 5.10–5.15 show the average scores of each data shown in Table 4.2,
where the I showed only the best nonnegativization in NICA1–NICA3 and
ICA1–ICA3 in each result for readability, but the other results also achieve
comparable improvements with those. From these results, we can confirm thatthe proposed initialization clearly improves the separation performance of
SSNMF, and these facts fit to the results in Fig. 5.8. Thus, we can guess that thegood initialization for NMF may be effective particularly for the supervised
source separation using NMF, such as FSNMF and SSNMF.

5.6

Summary

In this chapter, I addressed an efficient initialization method for NMF andproposed the utilization of ICA bases and estimated independent sources as
the initial values of the basis and activation matrices, respectively. From anexperimental comparison, some of the proposed method provides faster anddeeper convergence of the NMF cost function than the conventional methods.
Also, in supervised audio source separation, the proposed method achieves
better performance than that obtained by random initialization.

5.6 Summary

139

Figure 5.10: Average SDR improvement of ID1 for each number of iterations in(4.16) with various NMF initializations.

Figure 5.11: Average SDR improvement of ID2 for each number of iterations in(4.16) with various NMF initializations.

140

Chapter 5. Effective Initialization for Nonnegative Matrix Factorization
Based on Statistical Independence

Figure 5.12: Average SDR improvement of ID3 for each number of iterations in(4.16) with various NMF initializations.

Figure 5.13: Average SDR improvement of ID4 for each number of iterations in(4.16) with various NMF initializations.

5.6 Summary

141

Figure 5.14: Average SDR improvement of ID5 for each number of iterations in(4.16) with various NMF initializations.

Figure 5.15: Average SDR improvement of ID6 for each number of iterations in(4.16) with various NMF initializations.

143

6

Conclusion

6.1

Summary of Dissertation

In this dissertation, I addressed a problem of music source separation, which canbe applied to many valuable systems. This problem generally includes many
situations, and I mainly dealt with following two main topics:• determined (and overdetermined) BSS• single-channel semi-supervised source separation
For both of them, I proposed new effective optimization algorithms based on
NMF, which simultaneously achieve satisfactory separation performance andpractical efficiency.
In Chap. 3, I proposed a new determined BSS algorithm called ILRMA. Thismethod can be considered as a natural extension of traditional BSS algorithms,
FDICA and IVA. In FDICA, the source model was a scalar random variable,

144

Chapter 6. Conclusion

which obeys a non-Gaussian distribution. This scalar source model is extended toa multivariate vector variable as IVA, which enables a permutation-problem-freefrequency domain separation. As a further extension of the source model,
ILRMA employs NMF decomposition for capturing the low-rank time-frequency
structure of the sources. Owing to the low-rankness of the music spectrogram,
ILRMA can effectively model its structure and accurately estimate spatial
demixing filters. Also, I revealed the intriguing relationships between IVA,
MNMF, and ILRMA; MNMF with rank-1 spatial model is identical to ILRMA, andILRMA with the single NMF basis for each source is essentially the same modelas IVA. The optimization of ILRMA is based on the auxiliary function technique,resulting in a faster and more stable convergence than the conventional MNMF.
The advantage of its separation performance was experimentally confirmed inboth speech and music signals.
In Chap. 4, I proposed a new algorithm for discriminative training of
NMF bases. This algorithm employs two types of supervised bases for one
target source, which are called reconstructive and discriminative bases. The
reconstructive bases include all the frequency components of the target source,
and the discriminative bases consist of only the unique components to maximize
the ability of discrimination from the other non-target sources. Whereas thediscriminative bases training comes down to a bilevel optimization problem, I
proposed a simple optimization algorithm for obtaining both reconstructive
and discriminative bases instead. The efficacy of the proposed discriminative
SSNMF was validated via semi-supervised music source separation task.
In Chap. 5, a general problem in NMF optimization was dealt. Since NMF
is not a convex optimization problem, all the performance of NMF-based
application always depend on the initial values of basis and activation matrices.
As a remedy of this problem, I proposed a new efficient initialization method
based on statistical independence. In this method, the estimates of ICA are
utilized for the initial values of NMF variables, which are non-orthogonalbases and their coefficients. The proposed initialization achieved faster anddeeper minimization than the conventional orthogonality-based initializations.
In addition, the availability of the proposed initialization for determined
BSS, single-channel FSNMF, and discriminative SSNMF was experimentally6.2 Future Works

145

investigated. The result showed that the proposed method can lead better
separation for a supervised NMF approach.

6.2

Future Works

The following points still remain to be investigated or clarified.
• Since ILRMA has an NMF source model, we can easily employ some priorknowledges of each source for the optimization. The simple extension
is a supervised ILRMA, which utilizes pretrained NMF bases of eachsource. As further supervision, external information such as music scoreor an annotation by users can be a good candidate for improving source
model in ILRMA. However, to utilize such external information, the scaleambiguity among frequency bins in ILRMA must be solved in advance.
This is because the scale ambiguity has a risk to distort the pretrained
NMF bases or the given structure by the users. Some ICA optimizations
without the scale ambiguity were proposed, for example, ICA withminimal distortion principle [236, 237], single-input and multiple-output
ICA [238, 239, 240, 241], and multiple-input and single-output ICA [242].
These algorithms may be unified with ILRMA.
• Whereas a spectrogram of instrumental source has a low-rank timefrequency structure, a vocal spectrogram may not have the low-rank
property as shown in Fig. 2.7 owing to its variety of pitches and phonemes.
In the experiments presented in Sect. 3.6, ILRMA can achieve accurate
source separation even for the mixture with a vocal source. This might be acollateral effect of an accurate modeling of the other instrumental source,
namely, the better vocal separation might be achieved as a side effect of theaccurate separation of remaining instrumental sources. Indeed, the speech
separation based on ILRMA with a large number of bases could not givethe highest performance of separation (see Fig. 3.14). For vocal and speech
signals, another property may be useful to capture their time-frequency
structures. In the literature [243, 244], robust PCA [245, 246] is used forextracting a vocal source from music signals, where robust PCA can146

Chapter 6. Conclusion
decompose an observed matrix into a low-rank structure and a sparse
component, and the vocal source tends to be estimated as the sparse
component. If we introduce such sparse source model to ILRMA for
representing vocals, the music source separation would be improved.

• The cost function in ILRMA (3.46) is based on IS divergence, and the
MU rules (3.54) and (3.55) are the same as those of simple ISNMF. The
other popular criteria, KL divergence and EU distance, may also be used
for the cost function in ILRMA. The source models corresponding thesecriteria (Poisson distribution for KL divergence and Gaussian distribution
for EU distance) do not have a reproductive property. Thus, unlike
ISNMF, the additivity of power spectrograms in expectation sense does
not hold in KLNMF and EUNMF. As another source model with thereproductive property, NMF based on Cauchy distribution was proposed as
Cauchy NMF [247], and it is reported that it gives slightly better separation
performance than ISNMF. This source model can be used for an extensionof ILRMA.
• In Chap. 4, I have dealt with only SSNMF with a new discriminative
training method. However, the main idea in the proposed method canalso be used for full-supervised situation. The comparison with theconventional discriminative NMF and the proposed algorithm in the
full-supervised situation need to be investigated.
• The proposed algorithm of discriminative bases training is an approximation of the solution obtained by original bilevel optimization. This factcauses the decrease of SDR improvement at the converged point of algorithm as shown in Fig. 4.5. To avoid this problem, another approximative
optimization or a stopping criteria must be considered for a practical use.
• The proposed algorithm of discriminative SSNMF can be combined with apenalized SSNMF [98], which forces the non-target bases to be different
from the target bases.

147

Bibliography[1] P. Comon, “Independent component analysis, a new concept?” Signal
Processing, vol. 36, no. 3, pp. 287–314, 1994.
[2] P. Smaragdis, “Blind separation of convolved mixtures in the frequency
domain,” Neurocomputing, vol. 22, no. 1, pp. 21–34, 1998.
[3] A. Hiroe, “Solution of permutation problem in frequency domain ICA using
multivariate probability density functions,” in Proceedings of International
Conference on Independent Component Analysis and Blind Signal Separation(ICA), 2006, pp. 601–608.
[4] T. Kim, T. Eltoft, and T.-W. Lee, “Independent vector analysis: an extension
of ICA to multivariate components,” in Proceedings of International Conferenceon Independent Component Analysis and Blind Signal Separation (ICA), 2006,
pp. 165–172.
[5] D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative
matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791, 1999.
[6] A. Ozerov and C. Févotte, “Multichannel nonnegative matrix factorizationin convolutive mixtures for audio source separation,” IEEE Transactions on
Audio, Speech, and Language Processing, vol. 18, no. 3, pp. 550–563, 2010.
[7] P. Smaragdis, B. Raj, and M. Shashanka, “Supervised and semi-supervised
separation of sounds from single-channel mixtures,” in Proceedings ofIEEE International Conference on Independent Component Analysis and Signal
Separation (ICA), 2007, pp. 414–421.

148

Bibliography[8] P. Sprechmann, A. M. Bronstein, and G. Sapiro, “Supervised nonEuclidean sparse NMF via bilevel optimization with applications tospeech enhancement,” in Proceedings of Joint Workshop on Hands-free SpeechCommunication and Microphone Arrays (HSCMA), 2014, pp. 11–15.
[9] F. Weninger, J. L. Roux, J. R. Hershey, and S. Watanabe, “Discriminative
NMF and its application to single-channel source separation,” in Proceedingsof Interspeech, 2014, pp. 865–869.
[10] H. Chung, E. Plourde, and B. Champagne, “Discriminative training of
NMF model based on class probabilities for speech enhancement,” IEEE
Signal Processing Letters, vol. 23, no. 4, pp. 502–506, 2016.
[11] P. Paatero and U. Tapper, “Positive matrix factorization: A non-negative
factor model with optimal utilization of error estimates of data values,”
Environmetrics, vol. 5, no. 2, pp. 111–126, 1994.
[12] D. D. Lee and H. S. Seung, “Algorithms for non-negative matrix factorization,” in Proceedings of Advances in Neural Information Processing Systems(NIPS), 2000, pp. 556–562.
[13] A. Cichocki, R. Zdunek, A. H. Phan, and S. Amari, Nonnegative Matrix andTensor Factorizations: Applications to Exploratory Multi-way Data Analysis andBlind Source Separation. Wiley, 2009.
[14] H. Kameoka, “Non-negative matrix factorization and its variants for audio
signal processing,” in Applied Matrix and Tensor Variate Data Analysis,
T. Sakata, Ed. Springer Japan, 2016, ch. 2, pp. 21–50.
[15] J.-F. Cardoso, “Multidimensional independent component analysis,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), vol. 4, 1998, pp. 1941–1944.
[16] E. Vincent, “Musical source separation using time-frequency source
priors,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 14,
no. 1, pp. 91–98, 2006.

Bibliography149[17] C. Marro, Y. Mahieux, and K. U. Simmer, “Analysis of noise reduction anddereverberation techniques based on microphone arrays with postfiltering,”
IEEE Transactions on Speech and Audio Processing, vol. 6, no. 3, pp. 240–259,
1998.
[18] K. Lebart, J. M. Boucher, and P. N. Denbigh, “A new method based on
spectral subtraction for speech dereverberation,” Acta Acustica united with
Acustica, vol. 87, no. 3, pp. 359–366, 2001.
[19] P. Naylor and N. D. Gaubitch, Eds., Speech Dereverberation. Springer-Verlag
London, 2000.
[20] J. B. Allen and L. R. Rabiner, “A unified approach to short-time Fourier
analysis and synthesis,” Proceedings of the IEEE, vol. 65, no. 11, pp.
1558–1564, 1977.
[21] J. B. Allen, “Short term spectral analysis, synthesis, and modification
by discrete Fourier transform,” IEEE Transactions on Acoustics Speech andSignal Processing, vol. 25, no. 3, pp. 235–238, 1977.
[22] H. L. V. Trees, Optimum Array Processing: Part IV of Detection, Estimationand Modulation Theory. Wiley-Interscience, 2002.
[23] R. Roy and T. Kailath, “ESPRIT-estimation of signal parameters via
rotational invariance techniques,” IEEE Transactions on Acoustics, Speech,
and Signal Processing, vol. 37, no. 7, pp. 984–995, 1989.
[24] D. H. Johnson and D. E. Dudgeon, Array Signal Processing: Concepts andTechniques. Prentice Hall, 1993.
[25] A. V. Oppenheim and R. W. Schafer, Discrete-Time Signal Processing.
Prentice Hall, 1989.
[26] D. W. Griffin and J. S. Lim, “Signal estimation from modified short-time
Fourier transform,” IEEE Transactions on Acoustics, Speech, and Signal
Processing, vol. 32, no. 2, pp. 236–243, 1984.

150

Bibliography[27] B. Yang, “A study of inverse short-time Fourier transform,” in Proceedingsof IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2008, pp. 3541–3544.
[28] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,”
SIAM Review, vol. 51, no. 3, pp. 455–500, 2009.
[29] J.-F. Cardoso, “Infomax and maximum likelihood for blind source
separation,” IEEE Signal Processing Letters, vol. 4, no. 4, pp. 112–114, 1997.
[30] T.-W. Lee, Independent Component Analysis: Theory and Applications. Springer
US, 1998.
[31] S. Haykin, Ed., Unsupervised Adaptive Filtering (Volume I: Blind Source
Separation). Wiley-Interscience, 2000.
[32] A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis.
Wiley-Interscience, 2001.
[33] A. Cichocki and S. Amari, Adaptive Blind Signal and Image Processing:
Learning Algorithms and Applications. Wiley, 2002.
[34] H. Sawada, R. Mukai, S. Araki and S. Makino, “Frequency-domain blindsource separation,” in Speech Enhancement. Springer Berlin Heidelberg,
2005, ch. 13, pp. 299–327.
[35] S. Makino, H. Sawada and S. Araki, “Frequency-domain blind source
separation,” in Blind Speech Separation. Springer Netherlands, 2007, ch. 2,
pp. 47–78.
[36] T.-W. Lee, J. B. Anthony, and R. H. Lambert, “Blind separation of delayedand convolved sources,” in Proceedings of Advances in Neural Information
Processing Systems (NIPS), 1997, pp. 758–764.
[37] K. Torkkola, “Blind separation for audio signals – are we there yet?” inProceedings of International Workshop on Independent Component Analysis andBlind Signal Separation (ICA), 1999, pp. 11–15.

Bibliography151[38] K. Torkkola, “Blind separation of delayed and convolved sources,” in
Unsupervised Adaptive Filtering (Volume I: Blind Source Separation), S. Haykin,
Ed. Wiley-Interscience, 2000, ch. 8, pp. 321–375.
[39] R. H. Lambert and C. L. Nikias, “Blind deconvolution of multipath
mixtures,” in Unsupervised Adaptive Filtering (Volume I: Blind Source
Separation), S. Haykin, Ed. Wiley-Interscience, 2000, ch. 9, pp. 377–436.
[40] S. Haykin, Ed., Unsupervised Adaptive Filtering (Volume II: Blind Deconvolution). Wiley-Interscience, 2000.
[41] H. Sawada, R. Mukai, S. Araki, and S. Makino, “Convolutive blind source
separation for more than two sources in the frequency domain,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2004, pp. III–885–III–888.
[42] H. Buchner, R. Aichner, and W. Kellerman, “A generalization of blindsource separation algorithms for convolutive mixtures based on second
order statistics,” IEEE Transactions on Speech and Audio Processing, vol. 13,
no. 1, pp. 120–134, 2005.
[43] H. Saruwatari, T. Kawamura, T. Nishikawa, A. Lee, and K. Shikano, “Blindsource separation based on a fast-convergence algorithm combining ICA
and beamforming,” IEEE Transactions on Audio, Speech, and Language
Processing, vol. 14, no. 2, pp. 666–678, 2006.
[44] O. L. Frost, “An algorithm for linearly constrained adaptive array processing,” Proceedings of the IEEE, vol. 60, no. 8, pp. 926–935, 1972.
[45] L. J. Griffiths and C. W. Jim, “An alternative approach to linearly constrained
adaptive beamforming,” IEEE Transactions on Antennas and Propagation,
vol. 30, no. 1, pp. 27–34, 1982.
[46] T. J. Shan and T. Kailath, “Adaptive beamforming for coherent signals andinterference,” IEEE Transactions on Acoustics, Speech, and Signal Processing,vol. 33, no. 3, pp. 527–536, 1985.

152

Bibliography[47] S. Araki, S. Makino, Y. Hinamoto, R. Mukai, T. Nishikawa, and
H. Saruwatari, “Equivalence between frequency-domain blind source
separation and frequency-domain adaptive beamforming for convolutive
mixtures,” EURASIP Journal on Advances in Signal Processing, vol. 2003,
no. 11, pp. 1–10, 2003.
[48] H. Cox, R. Zeskind, and M. Owen, “Robust adaptive beamforming,” IEEE
Transactions on Acoustics, Speech, and Signal Processing, vol. 35, no. 10, pp.
1365–1376, 1987.
[49] O. Hoshuyama and A. Sugiyama, “Robust adaptive beamforming,” inMicrophone Arrays: Signal Processing Techniques and Applications, M. Brandstein and D. Ward, Eds. Springer-Verlag Berlin Heidelberg, 2001, ch. 5,
pp. 87–109.
[50] S. A. Vorobyov, A. B. Gershman, and Z.-Q. Luo, “Robust adaptive
beamforming using worst-case performance optimization: A solution
to the signal mismatch problem,” IEEE Transactions on Signal Processing,vol. 51, no. 2, pp. 313–324, 2003.
[51] S. A. Vorobyov, A. B. Gershman, Z.-Q. Luo, and N. Ma, “Adaptive
beamforming with joint robustness against mismatched signal steeringvector and interference nonstationarity,” IEEE Signal Processing Letters,
vol. 11, no. 2, pp. 108–111, 2004.
[52] J. Li and P. Stoica, Robust adaptive beamforming.
vol. 88.

John Wiley & Sons, 2005,[53] T.-W. Lee, M. S. Lewicki, M. Girolami, and T. J. Sejnowski, “Blindsource separation of more sources than mixtures using overcomplete
representations,” IEEE Signal Processing Letters, vol. 6, no. 4, pp. 87–90,
1999.
[54] B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete
basis set: A strategy employed by V1?” Vision Research, vol. 37, no. 23, pp.
3311–3325, 1997.

Bibliography153[55] P. Bofill and M. Zibulevsky, “Underdetermined blind source separation
using sparse representations,” Signal Processing, vol. 81, no. 11, pp.
2353–2362, 2001.
[56] T. Virtanen, “Sound source separation using sparse coding with temporalcontinuity objective,” in Proceedings of International Computer Music
Conference (ICMC), 2003, pp. 231–234.
[57] F. J. Theis, E. W. Lang, and C. G. Puntonet, “A geometric algorithm forovercomplete linear ICA,” Neurocomputing, vol. 56, pp. 381–398, 2004.
Ü Yılmaz, “Blind separation of disjoint[58] A. Jourjine, S. Rickard, and O.
orthogonal signals: demixing N sources from 2 mixtures,” in Proceedingsof IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), vol. 5, 2000, pp. 2985–2988.
Ü Yılmaz and S. Rickard, “Blind separation of speech mixtures via[59] O.
time-frequency masking,” IEEE Transactions on Signal Processing, vol. 52,
no. 7, pp. 1830–1847, 2004.
[60] S. Araki, H. Sawada, R. Mukai, and S. Makino, “Underdetermined blindsparse source separation for arbitrarily arranged multiple sensors,” Signal
Processing, vol. 87, no. 8, pp. 1833–1847, 2007.
[61] T. Yoshioka, N. Ito, M. Delcroix, A. Ogawa, K. Kinoshita, M. Fujimoto,C. Yu, W. J. Fabian, M. Espi, T. Higuchi, S. Araki, and T. Nakatani, “TheNTT CHiME-3 system: Advances in speech enhancement and recognition
for mobile multi-microphone devices,” in Proceedings of IEEE Workshop on
Automatic Speech Recognition and Understanding (ASRU), 2015, pp. 436–443.
[62] T. Higuchi, N. Ito, T. Yoshioka, and T. Nakatani, “Robust MVDR beamforming using time-frequency masks for online/offline ASR in noise,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2016, pp. 5210–5214.
[63] M. Aoki, M. Okamoto, S. Aoki, H. Matsui, T. Sakurai, and Y. Kaneda,“Sound source segregation based on estimating incident angle of each

154

Bibliographyfrequency component of input signals acquired by multiple microphones,”
Acoustical Science and Technology, vol. 22, no. 2, pp. 149–157, 2001.

[64] N. Roman, D.L. Wang, and G. J. Brown, “Speech segregation based on
sound localization,” The Journal of the Acoustical Society of America, vol. 114,
no. 4, pp. 2236–2252, 2003.
[65] Y. Izumi, N. Ono, and S. Sagayama, “Sparseness-based 2ch BSS using the
EM algorithm in reverberant environment,” in Proceedings of IEEE Workshop
on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2007,
pp. 147–150.
[66] H. Sawada, S. Araki, and S. Makino, “Underdetermined convolutive blindsource separation via frequency bin-wise clustering and permutationalignment,” IEEE Transactions on Audio, Speech, and Language Processing,vol. 19, no. 3, pp. 516–527, 2011.
[67] D. FitzGerald, M. Cranitch, and E. Coyle, “Shifted non-negative matrix
factorisation for sound source separation,” in Proceedings of IEEE Workshop
on Statistical Signal Processing (SSP), 2005, pp. 1132–1137.
[68] A. Cichocki, R. Zdunek, and S. Amari, “New algorithms for non-negative
matrix factorization in applications to blind source separation,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2006, pp. 5479–5482.
[69] M. N. Schmidt and M. Mørup, “Nonnegative matrix factor 2-D deconvolution for blind single channel source separation,” in Proceedings ofInternational Conference on Independent Component Analysis and Signal
Separation (ICA), 2006, pp. 700–707.
[70] T. Virtanen, “Monaural sound source separation by nonnegative matrix
factorization with temporal continuity and sparseness criteria,” IEEE
Transactions on Audio, Speech, and Language Processing, vol. 15, no. 3, pp.
1066–1074, 2007.

Bibliography155[71] A. Ozerov, C. Févotte, and M. Charbit, “Factorial scaled hidden Markov
model for polyphonic audio representation and source separation,” inProceedings of IEEE Workshop on Applications of Signal Processing to Audio andAcoustics (WASPAA), 2009, pp. 121–124.
[72] O. Dikmen and A. T. Cemgil, “Unsupervised single-channel source
separation using Bayesian NMF,” in Proceedings of IEEE Workshop on
Applications of Signal Processing to Audio and Acoustics (WASPAA), 2009, pp.
93–96.
[73] M. Spiertz and V. Gnann, “Source-filter based clustering for monauralblind source separation,” in Proceedings of International Conference on Digital
Audio Effects (DAFx), 2009.
[74] G. J. Mysore, P. Smaragdis, and B. Raj, “Non-negative hidden Markov
modeling of audio with application to source separation,” in Proceedings ofInternational Conference on Latent Variable Analysis and Signal Separation(LVA/ICA), 2010, pp. 140–148.
[75] R. Jaiswal. D. FitzGerald, D. Barry, E. Coyle, and S. Rickard, “Clustering
NMF basis functions using shifted NMF for monaural sound source
separation,” in Proceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2011, pp. 245–248.
[76] X. Guo, S. Uhlich, and Y. Mitsufuji, “NMF-based blind source separation
using a linear predictive coding error clustering criterion,” in Proceedingsof IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2015, pp. 261–265.
[77] D. FitzGerald, M. Cranitch, and E. Coyle, “Non-negative tensor factorisation
for sound source separation,” in Proceedings of Irish Signals and Systems
Conference (ISSC), 2005, pp. 8–12.
[78] R. M. Parry and I. A. Essa, “Estimating the spatial position of spectral components in audio,” in Proceedings of International Conference on Independent
Component Analysis and Blind Signal Separation (ICA), 2006, pp. 666–673.

156

Bibliography[79] J.-L. Durrieu, A. Ozerov, C. Févotte, G. Richard, and B. David, “Maininstrument separation from stereophonic audio signals using a source/filter
model,” in Proceedings of European Signal Processing Conference (EUSIPCO),
2009, pp. 15–19.
[80] H. Kameoka, T. Yoshioka, M. Hamamura, J. L. Roux, and K. Kashino,“Statistical model of speech signals based on composite autoregressive
system with application to blind source separation,” in Proceedings ofInternational Conference on Latent Variable Analysis and Signal Separation(LVA/ICA), 2010, pp. 245–253.
[81] S. Arberet, A. Ozerov, N. Q. K. Duong, E. Vincent, R. Gribonval, F. Bimbot,
and P. Vandergheynst, “Nonnegative matrix factorization and spatial
covariance model for under-determined reverberant audio source separation,” in Proceedings of Information Sciences Signal Processing and their
Applications (ISSPA), 2010, pp. 1–4.
[82] A. Ozerov, C. Févotte, R. Blouet, and J.-L. Durrieu, “Multichannel
nonnegative tensor factorization with structured constraints for userguided audio source separation,” in Proceedings of IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011, pp.
257–260.
[83] Y. Mitsufuji and A. Roebel, “Sound source separation based on nonnegative tensor factorization incorporating spatial cue as prior knowledge,”in Proceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2013, pp. 71–75.
[84] H. Sawada, H. Kameoka, S. Araki, and N. Ueda, “Multichannel extensionsof non-negative matrix factorization with complex-valued data,” IEEE
Transactions on Audio, Speech, and Language Processing, vol. 21, no. 5, pp.
971–982, 2013.
[85] Z. Rafii and B. Pardo, “Music/voice separation using the similarity matrix,”in Proceedings of The International Society for Music Information Retrieval
Conference (ISMIR), 2012, pp. 583–588.

Bibliography157[86] A. Liutkus, Z. Rafii, R. Badeau, B. Pardo, and G. Richard, “Adaptive
filtering for music/voice separation exploiting the repeating musicalstructure,” in Proceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2012, pp. 53–56.
[87] D. FitzGerald, “Vocal separation using nearest neighbours and medianfiltering,” in IET Irish Signals and Systems Conference (ISSC), 2012, pp. 1–5.
[88] Z. Rafii and B. Pardo, “REpeating Pattern Extraction Technique (REPET):
A simple method for music/voice separation,” IEEE transactions on audio,
speech, and language processing, vol. 21, no. 1, pp. 73–84, 2013.
[89] Z. Rafii, Z. Duan, and B. Pardo, “Combining rhythm-based and pitch-based
methods for background and melody separation,” IEEE/ACM Transactions
on Audio, Speech, and Language Processing, vol. 22, no. 12, pp. 1884–1893,
2014.
[90] Z. Rafii, A. Liutkus, and B. Pardo, “A simple user interface system forrecovering patterns repeating in time and frequency in mixtures of sounds,”
in Proceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2015, pp. 271–275.
[91] A. Liutkus, Z. Rafii, B. Pardo, D. FitzGerald, and L. Daudet, “Kernel
spectrogram models for source separation,” in Proceedings of Joint Workshop
on Hands-free Speech Communication and Microphone Arrays (HSCMA), 2014,
pp. 6–10.
[92] A. Liutkus, D. FitzGerald, Z. Rafii, B. Pardo, and L. Daudet, “Kernel
additive models for source separation,” IEEE Transactions on Signal
Processing, vol. 62, no. 16, pp. 4298–4310, 2014.
[93] D. FitzGerald, A. Liukus, Z. Rafii, B. Pardo, and L. Daudet, “Harmonic/percussive separation using kernel additive modelling,” in Proceedings of IET Irish Signals & Systems Conference and China-Ireland International
Conference on Information and Communications Technologies (ISSC/CIICT),
2013, pp. 35–40.

158

Bibliography[94] A. Liutkus, D. FitzGerald, and Z. Rafii, “Scalable audio separation withlight kernel additive modelling,” in Proceedings of IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp.
76–80.
[95] P. Smaragdis, “Convolutive speech bases and their application to supervised
speech separation,” IEEE Transactions on Audio, Speech, and Language
Processing, vol. 15, no. 1, pp. 1–12, 2007.
[96] P. D. O’grady and B. A. Pearlmutter, “Discovering convolutive speechphones using sparseness and non-negativity,” in Proceedings of International
Conference on Independent Component Analysis and Signal Separation (ICA),
2007, pp. 520–527.
[97] N. Mohammadiha, P. Smaragdis, and A. Leĳon, “Supervised and unsupervised speech enhancement using nonnegative matrix factorization,” IEEE
Transactions on Audio, Speech, and Language Processing, vol. 21, no. 10, pp.
2140–2151, 2013.
[98] D. Kitamura, H. Saruwatari, K. Yagi, K. Shikano, Y. Takahashi, andK. Kondo, “Music signal separation based on supervised nonnegative
matrix factorization with orthogonality and maximum-divergence penalties,” IEICE Transactions on Fundamentals of Electronics, Communications andComputer Sciences, vol. E97-A, no. 5, pp. 1113–1118, 2014.
[99] D. Kitamura, H. Saruwatari, H. Kameoka, Y. Takahashi, K. Kondo, and
S. Nakamura, “Multichannel signal separation combining directional clustering and nonnegative matrix factorization with spectrogram restoration,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23,
no. 4, pp. 654–669, 2015.
[100] R. Hennequin, B. David, and R. Badeau, “Score informed audio source
separation using a parametric model of non-negative spectrogram,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2011, pp. 45–48.

Bibliography159[101] S. Ewert and M. Müller, “Using score-informed constraints for NMF-basedsource separation,” in Proceedings of IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP), 2012, pp. 129–132.
[102] J. Fritsch and M. D. Plumbley, “Score informed audio source separation
using constrained nonnegative matrix factorization and score synthesis,”in Proceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2013, pp. 888–891.
[103] N. Q. K. Duong, A. Ozerov, and L. Chevallier, “Temporal annotation-based
audio source separation using weighted nonnegative matrix factorization,”in Proceedings of IEEE International Conference on Consumer Electronics Berlin(ICCE-Berlin), 2014, pp. 220–224.
[104] T. Ono, N. Ono, and S. Sagayama, “User-guided independent vector
analysis with source activity tuning,” in Proceedings of IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012, pp.
2417–2420.
[105] M. Fakhry and F. Nesta, “Underdetermined source detection and separation
using a normalized multichannel spatial dictionary,” in Proceedings ofInternational Workshop on Acoustic Signal Enhancement (IWAENC), 2012.
[106] F. Nesta and M. Fakhry, “Unsupervised spatial dictionary learning forsparse underdetermined multichannel source separation,” in Proceedingsof IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2013, pp. 86–90.
[107] Y. Wang and D. Wang, “Towards scaling up classification-based speech
separation,” IEEE Transactions on Audio, Speech, and Language Processing,vol. 21, no. 7, pp. 1381–1390, 2013.
[108] A. Narayanan and D. Wang, “Ideal ratio mask estimation using deepneural networks for robust speech recognition,” in Proceedings of IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP),
2013, pp. 7092–7096.

160

Bibliography[109] F. Weninger, J. R. Hershey, J. L. Roux, and B. Schuller, “Discriminatively
trained recurrent neural networks for single-channel speech separation,”in Proceedings of IEEE Global Conference on Signal and Information Processing(GlobalSIP), 2014, pp. 577–581.
[110] Y. Tu, J. Du, Y. Xu, L. Dai, and C.-H. Lee, “Speech separation based on
improved deep neural networks with dual outputs of speech features
for both target and interfering speakers,” in Proceedings of International
Symposium on Chinese Spoken Language Processing (ISCSLP), 2014, pp.
250–254.
[111] P.-S. Huang, M. Kim, M. H.-Johnson, and P. Smaragdis, “Singing-voice separation from monaural recordings using deep recurrent neural networks,”
in Proceedings of The International Society for Music Information Retrieval
Conference (ISMIR), 2014, pp. 477–482.
[112] P.-S. Huang, M. Kim, M. H.-Johnson, and P. Smaragdis, “Deep learning formonaural speech separation,” in Proceedings of IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 1562–1566.
[113] P.-S. Huang, M. Kim, M. H.-Johnson, and P. Smaragdis, “Joint optimizationof masks and deep recurrent neural networks for monaural source
separation,” IEEE/ACM Transactions on Audio, Speech, and Language
Processing, vol. 23, no. 12, pp. 2136–2147, 2015.
[114] S. Araki, T. Hayashi, M. Delcroix, M. Fujimoto, K. Takeda, and T. Nakatani,“Exploring multi-channel features for denoising-autoencoder-based speechenhancement,” in Proceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2015, pp. 116–120.
[115] S. Uhlich, F. Giron, and Y. Mitsufuji, “Deep neural network based
instrument extraction from music,” in Proceedings of IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp.
2135–2139.
[116] A. Narayanan and D. Wang, “Improving robustness of deep neuralnetwork acoustic models via speech separation and joint adaptive training,”

Bibliography

161

IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23,
no. 1, pp. 92–101, 2015.
[117] Y. Wang and D. Wang, “A deep neural network for time-domain signal
reconstruction,” in Proceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2015, pp. 4390–4394.
[118] A. A. Nugraha, A. Liutkus, and E. Vincent, “Multichannel audio source
separation with deep neural networks,” IEEE/ACM Transactions on Audio,
Speech, and Language Processing, vol. 24, no. 9, pp. 1652–1664, 2016.
[119] Y. Liu, P. Zhang, and T. Hain, “Using neural network front-ends on far
field multiple microphones based speech recognition,” in Proceedingsof IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2014, pp. 5542–5546.
[120] S. Renals and P. Swietojanski, “Neural networks for distant speech recognition,” in Proceedings of Joint Workshop on Hands-Free Speech Communicationand Microphone Arrays (HSCMA), 2014, pp. 172–176.
[121] H. Sawada, R. Mukai, S. Araki, and S. Makino, “Polar coordinate based
nonlinear function for frequency-domain blind source separation,” IEICE
Transactions on Fundamentals of Electronics, Communications and Computer
Sciences, vol. 86, no. 3, pp. 590–596, 2003.
[122] S. Araki, F. Nesta, E. Vincent, Z. Koldovský, G. Nolte, A. Ziehe, and A. Benichoux, “The 2011 signal separation evaluation campaign (SiSEC2011):audio source separation,” in Proceedings of International Conference on Latent
Variable Analysis and Signal Separation (LVA/ICA), 2012, pp. 414–422.
[123] P. Smaragdis and J. C. Brown, “Non-negative matrix factorization for
polyphonic music transcription,” in Proceedings of IEEE Workshop on
Applications of Signal Processing to Audio and Acoustics (WASPAA), 2003, pp.
177–180.
[124] Z. Duan, Y. Zhang, C. Zhang, and Z. Shi, “Unsupervised single-channel
music source separation by average harmonic structure modeling,” IEEE

162

Bibliography
Transactions on Audio, Speech, and Language Processing, vol. 16, no. 4, pp.
766–778, 2008.

[125] C. Févotte, N. Bertin, and J.-L. Durrieu, “Nonnegative matrix factorization
with the itakura–saito divergence. With application to music analysis,”
Neural Computation, vol. 21, no. 3, pp. 793–830, 2009.
[126] M. Nakano, J. L. Roux, H. Kameoka, Y. Kitano, N. Ono, and S. Sagayama,“Nonnegative matrix factorization with Markov-chained bases for modelingtime-varying patterns in music spectrograms,” in Proceedings of International
Conference on Latent Variable Analysis and Signal Separation (LVA/ICA), 2010,
pp. 149–156.
[127] M. Nakano, J. L. Roux, H. Kameoka, T. Nakamura, N. Ono, S. Sagayama,“Bayesian nonparametric spectrogram modeling based on infinite factorial
infinite hidden Markov model,” in Proceedings of IEEE Workshop on
Applications of Signal Processing to Audio and Acoustics (WASPAA), 2011, pp.
325–328.
[128] H. Kameoka, M. Nakano, K. Ochiai, Y. Imoto, K. Kashino, and S. Sagayama,“Constrained and regularized variants of non-negative matrix factorization
incorporating music-specific constraints,” in Proceedings of IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012, pp.
5365–5368.
[129] K. Ochiai, H. Kameoka, and S. Sagayama, “Explicit beat structure modelingfor non-negative matrix factorization-based multipitch analysis,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2012, pp. 133–136.
[130] D. Kitamura, H. Saruwatari, Y. Iwao, K. Shikano, K. Kondo, and Y. Takahashi, “Superresolution-based stereo signal separation via supervised
nonnegative matrix factorization,” in Proceedings of IEEE International
Conference on Digital Signal Processing (DSP), 2013.
[131] D. Kitamura, H. Saruwatari, K. Shikano, K. Kondo, and Y. Takahashi,“Music signal separation by supervised nonnegative matrix factorization

Bibliography163

with basis deformation,” in Proceedings of IEEE International Conference on
Digital Signal Processing (DSP), 2013.
[132] D. Kitamura, H. Saruwatari, S. Nakamura, Y. Takahashi, K. Kondo, and
H. Kameoka, “Divergence optimization in nonnegative matrix factorization
with spectrogram restoration for multichannel signal separation,” inProceedings of Joint Workshop on Hands-free Speech Communication andMicrophone Arrays (HSCMA), 2014, pp. 92–96.
[133] H. Nakajima, D. Kitamura, N. Takamune, S. Koyama, H. Saruwatari, N. Ono,
Y. Takahashi, and K. Kondo, “Music signal separation using supervised
nmf with all-pole-model-based discriminative basis deformation,” inProceedings of The European Signal Processing Conference (EUSIPCO), 2016,
pp. 1143–1147.
[134] E. M. Grais and H. Erdogan, “Discriminative nonnegative dictionary learning using cross-coherence penalties for single channel source separation,”in Proceedings of Interspeech, 2013, pp. 808–812.
[135] Z. Wang and F. Sha, “Discriminative non-negative matrix factorization for
single-channel speech separation,” in Proceedings of IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp.
3749–3753.
[136] J. L. Roux, J. R. Hershey, and F. Weninger, “Deep NMF for speech
separation,” in Proceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2015, pp. 66–70.
[137] K. Kwon, J. W. Shin, and N. S. Kim, “Discriminative training of NMF model
based on class probabilities for speech enhancement,” IEICE Transactions
on Information and Systems, vol. E98-D, no. 11, pp. 2017–2020, 2015.
[138] W. Xu, X. Liu, and Y. Gong, “Document clustering based on non-negative
matrix factorization,” in Proceedings of Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval, 2003, pp.
267–273.

164

Bibliography[139] M. W. Berry and M. Browne, “Email surveillance using non-negative
matrix factorization,” Computational & Mathematical Organization Theory,
vol. 11, no. 3, pp. 249–264, 2005.
[140] J. Sivic, B. C. Russell, A. A. Efros, A. Zisserman, and W. T. Freeman,“Discovering objects and their location in images,” in Proceedings of IEEE
International Conference on Computer Vision (ICCV), vol. 1, 2005, pp. 370–377.
[141] D. Soukup and I. Bajla, “Robust object recognition under partial occlusions
using NMF,” Computational Intelligence and Neuroscience, vol. 2008, 2008.
[142] M. M. Kalayeh, H. Idrees, and M. Shah, “NMF-KNN: Image annotation using weighted multi-view non-negative matrix factorization,” in Proceedingsof IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014,
pp. 184–191.
[143] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online learning for matrix
factorization and sparse coding,” Journal of Machine Learning Research,
vol. 11, no. Jan, pp. 19–60, 2010.
[144] R. Sandler and M. Lindenbaum, “Nonnegative matrix factorization withearth mover’s distance metric for image analysis,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 33, no. 8, pp. 1590–1602, 2011.
[145] V. Monga and M. K. Mihçak, “Robust and secure image hashing via
non-negative matrix factorizations.” IEEE Transactions on Information
Forensics and Security, vol. 2, no. 3-1, pp. 376–390, 2007.
[146] W. Lu, W. Sun, and H. Lu, “Robust watermarking based on DWT andnonnegative matrix factorization,” Computers & Electrical Engineering,
vol. 35, no. 1, pp. 183–188, 2009.
[147] Z. Chen, A. Cichocki, and T. M. Rutkowski, “Constrained non-negative
matrix factorization method for EEG analysis in early detection of
Alzheimer disease,” in Proceedings of IEEE International Conference on
Acoustics Speech and Signal Processing Proceedings, vol. 5, 2006, pp. 893–896.

Bibliography165[148] H. Lee, A. Cichocki, and S. Choi, “Kernel nonnegative matrix factorization
for spectral EEG feature extraction,” Neurocomputing, vol. 72, no. 13, pp.
3182–3190, 2009.
[149] C. Damon, A. Liutkus, A. Gramfort, and S. Essid, “Non-negative matrix
factorization for single-channel EEG artifact rejection,” in Proceedingsof IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2013, pp. 1177–1181.
[150] C. Damon, A. Liutkus, A. Gramfort, and S. Essid, “Non-negative tensor
factorization for single-channel EEG artifact rejection,” in Proceedingsof IEEE International Workshop on Machine Learning for Signal Processing(MLSP), 2013, pp. 1–6.
[151] J.-P. Brunet, P. Tamayo, T. R. Golub, and J. P. Mesirov, “Metagenes andmolecular pattern discovery using matrix factorization,” Proceedings of the
National Academy of Sciences, vol. 101, no. 12, pp. 4164–4169, 2004.
[152] Y. Gao and G. Church, “Improving molecular cancer class discovery
through sparse non-negative matrix factorization,” Bioinformatics, vol. 21,
no. 21, pp. 3970–3975, 2005.
[153] D. Greene, G. Cagney, N. Krogan, and P. Cunningham, “Ensemble
non-negative matrix factorization methods for clustering protein–protein
interactions,” Bioinformatics, vol. 24, no. 15, pp. 1722–1728, 2008.
[154] M. N. Schmidt, J. Larsen, and F.-T. Hsiao, “Wind noise reduction using
non-negative sparse coding,” in Proceedings of IEEE Workshop on Machine
Learning for Signal Processing (MLSP), 2007, pp. 431–436.
[155] K. W. Wilson, B. Raj, and P. Smaragdis, “Regularized non-negative matrix
factorization with temporal dependencies for speech denoising,” inProceedings of Interspeech, 2008, pp. 411–414.
[156] J. L. Roux, H. Kameoka, N. Ono, A. D. Cheveigne, and S. Sagayama,“Computational auditory induction as a missing-data model-fitting

166

Bibliography
problem with Bregman divergence,” Speech Communication, vol. 53, no. 5,
pp. 658–676, 2011.

[157] K. Y. Yılmaz, A. T. Cemgil, and U. Şimşekli, “Generalised coupled tensor
factorisation,” in Proceedings of Advances in Neural Information Processing
Systems (NIPS), 2011, pp. 2151–2159.
[158] J. Nikunen, T. Virtanen, and M. Vilermo, “Multichannel audio upmixing
based on non-negative tensor factorization representation,” in Proceedingsof IEEE Workshop on Applications of Signal Processing to Audio and Acoustics(WASPAA), 2011, pp. 33–36.
[159] S. A. Abdallah and M. D. Plumbley, “Polyphonic music transcription
by non-negative sparse coding of power spectra,” in Proceedings of The
International Society for Music Information Retrieval Conference (ISMIR), 2004,
pp. 318–325.
[160] E. Vincent, N. Bertin, and R. Badeau, “Two nonnegative matrix factorization
methods for polyphonic pitch transcription,” in Proceedings of Music
Information Retrieval Evaluation eXchange (MIREX), 2007.
[161] E. Vincent, N. Bertin, and R. Badeau, “Harmonic and inharmonic
nonnegative matrix factorization for polyphonic pitch transcription,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2008, pp. 109–112.
[162] N. Bertin, R. Badeau, and E. Vincent, “Enforcing harmonicity andsmoothness in Bayesian non-negative matrix factorization applied topolyphonic music transcription,” IEEE Transactions on Audio, Speech, andLanguage Processing, vol. 18, no. 3, pp. 538–549, 2010.
[163] E. Vincent, N. Bertin, and R. Badeau, “Adaptive harmonic spectral
decomposition for multiple pitch estimation,” IEEE Transactions on Audio,
Speech, and Language Processing, vol. 18, no. 3, pp. 528–537, 2010.
[164] I. Csiszár, “I-divergence geometry of probability distributions and minimization problems,” The Annals of Probability, pp. 146–158, 1975.

Bibliography167[165] F. Itakura and S. Saito, “Analysis synthesis telephony based on the
maximum likelihood method,” in Proceedings of International Congress on
Acoustics (ICA), 1968, pp. C–17–C–20.
[166] S. Eguchi and Y. Kano, “Robustifying maximum likelihood estimation,”
Institute of Statistical Mathematics, Tech. Rep., 2001.
[167] M. Nakano, H. Kameoka, J. L. Roux, Y. Kitano, N. Ono, and S. Sagayama,“Convergence-guaranteed multiplicative algorithms for nonnegative matrix
factorization with beta-divergence,” in Proceedings of IEEE International
Workshop on Machine Learning for Signal Processing (MLSP), 2010, pp.
283–288.
[168] C. Févotte and J. Idier, “Algorithms for nonnegative matrix factorization
with the β-divergence,” Neural Computation, vol. 23, no. 9, pp. 2421–2456,
2011.
[169] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood fromincomplete data via the EM algorithm,” Journal of the royal statistical society.
Series B (methodological), pp. 1–38, 1977.
[170] D. R. Hunter and K. Lange, “Quantile regression via an MM algorithm,”
Journal of Computational and Graphical Statistics, vol. 9, no. 1, pp. 60–77, 2000.
[171] D. R. Hunter and K. Lange, “A tutorial on MM algorithms,” The American
Statistician, vol. 58, no. 1, pp. 30–37, 2004.
[172] Y. Sun, P. Babu, and D. P. Palomar, “Majorization-minimization algorithms
in signal processing, communications, and machine learning,” IEEE
Transactions on Signal Processing, vol. 65, no. 3, pp. 794–816, 2017.
[173] T. Kim, H. T. Attias, S.-Y. Lee, and T.-W. Lee, “Blind source separation
exploiting higher-order frequency dependencies,” IEEE Transactions on
Audio, Speech, and Language Processing, vol. 15, no. 1, pp. 70–79, 2007.
[174] A. J. Bell and T. J. Sejnowski, “An information-maximization approach toblind separation and blind deconvolution,” Neural Computation, vol. 7,
no. 6, pp. 1129–1159, 1995.

168

Bibliography[175] S. Amari, A. Cichocki, and H. H. Yang, “A new learning algorithm forblind signal separation,” in Proceedings of Advances in Neural Information
Processing Systems (NIPS), vol. 8, 1996, pp. 757–763.
[176] N. Ono and S. Miyabe, “Auxiliary-function-based independent component
analysis for super-gaussian sources,” in Proceedings of International Conference on Latent Variable Analysis and Signal Separation (LVA/ICA), 2010, pp.
165–172.
[177] T. Adali, H. Ki, and J.-F. Cardoso, “Complex ICA using nonlinear functions,”
IEEE Transactions on Signal Processing, vol. 56, no. 9, pp. 4536–4544, 2008.
[178] S. Kurita, H. Saruwatari, S. Kajita, K. Takeda, and F. Itakura, “Evaluation ofblind signal separation method using directivity pattern under reverberant
conditions,” in Proceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2000, pp. 3140–3143.
[179] N. Murata, S. Ikeda, and A. Ziehe, “An approach to blind source separation
based on temporal structure of speech signals,” Neurocomputing, vol. 41,
no. 1–4, pp. 1–24, 2001.
[180] H. Sawada, R. Mukai, S. Araki, and S. Makino,, “A robust and precise
method for solving the permutation problem of frequency-domain blindsource separation,” IEEE Transactions on Speech and Audio Processing, vol. 12,
no. 5, pp. 530–538, 2004.
[181] H. Sawada, S. Araki, and S. Makino, “Measuring dependence of bin-wise
separated signals for permutation alignment in frequency-domain BSS,” inProceedings of IEEE International Symposium on Circuits and Systems (ISCAS),
2007, pp. 3247–3250.
[182] S. Kotz, T. J. Kozubowski, and K. Podgórski, “Symmetric multivariate
laplace distribution,” in The Laplace Distribution and Generalizations: A
Revisit with Applications to Communications, Economics, Engineering, andFinance. Birkhäuser Basel, 2001, ch. 5, pp. 231–238.

Bibliography169[183] T. Eltoft, T. Kim, and T.-W. Lee, “On the multivariate Laplace distribution,”
IEEE Signal Processing Letters, vol. 13, no. 5, pp. 300–303, 2006.
[184] N. Ono, “Stable and fast update rules for independent vector analysis
based on auxiliary function technique,” in Proceedings of IEEE Workshop on
Applications of Signal Processing to Audio and Acoustics (WASPAA), 2011, pp.
189–192.
[185] N. Ono, “Fast stereo independent vector analysis and its implementation
on mobile phone,” in Proceedings of International Workshop on Acoustic Signal
Enhancement (IWAENC), 2012.
[186] N. Ono, “Auxiliary-function-based independent vector analysis with powerof vector-norm type weighting functions,” in Proceedings of Asia-PacificSignal and Information Processing Association Annual Summit and Conference(APSIPA ASC), 2012.
[187] F. D. Neeser and J. L. Massey, “Proper complex random processes withapplications to information theory,” IEEE Transactions on Information Theory,
vol. 39, no. 4, pp. 1293–1302, 1993.
[188] A. T. Cemgil, P. Peeling, O. Dikmen, and S. Godsill, “Prior structures fortime-frequency energy distributions,” in Proceedings of IEEE Workshop on
Applications of Signal Processing to Audio and Acoustics (WASPAA), 2007, pp.
151–154.
[189] C. Févotte and J.-F. Cardoso, “Maximum likelihood approach for blindaudio source separation using time-frequency gaussian source models,” inProceedings of IEEE Workshop on Applications of Signal Processing to Audio andAcoustics (WASPAA), 2005, pp. 78–81.
[190] K. Hild, H. T. Attias, and S. Nagarajan, “An expectation-maximization
method for spatio-temporal blind source separation using an AR-MOG
source model,” IEEE Transactions on Neural Networks, vol. 19, no. 3, pp.
508–519, 2008.

170

Bibliography[191] T. Nakatani, B.-H. Juang, T. Yoshioka, K. Kinoshita, M. Delcroix, and
M. Miyoshi, “An expectation-maximization method for spatio-temporalblind source separation using an AR-MOG source model,” IEEE Transactions
on Audio, Speech, and Language Processing, vol. 16, no. 8, pp. 1512–1527, 2008.
[192] A. R. López, N. Ono, U. Remes, K. Palomäki, and M. Kurimo, “Designing
multichannel source separation based on single-channel source separation,”in Proceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2015, pp. 469–473.
[193] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari,“Efficient multichannel nonnegative matrix factorization exploiting rank-1
spatial model,” in Proceedings of IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), 2015, pp. 276–280.
[194] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari,“Determined blind source separation unifying independent vector analysis
and nonnegative matrix factorization,” IEEE/ACM Transactions on Audio,
Speech, and Language Processing, vol. 24, no. 9, pp. 1626–1641, 2016.
[195] N. Q. K. Duong, E. Vincent, and R. Gribonval, “Spatial covariance models
for under-determined reverberant audio source separation,” in Proceedingsof IEEE Workshop on Applications of Signal Processing to Audio and Acoustics(WASPAA), 2009, pp. 129–132.
[196] N. Q. K. Duong, E. Vincent, and R. Gribonval, “Under-determined
reverberant audio source separation using a full-rank spatial covariancemodel,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18,
no. 7, pp. 1830–1840, 2010.
[197] K. U. Simmer, J. Bitzer, and C. Marro, “Post-filtering techniques,” in Microphone Arrays: Signal Processing Techniques and Applications, M. Brandsteinand D. Ward, Eds. Springer-Verlag Berlin Heidelberg, 2001, ch. 3, pp.
39–60.

Bibliography171[198] W. James and C. Stein, “Estimation with quadratic loss,” in Proceedings ofBerkeley Symposium on Mathematical Statistics and Probability, vol. 1, 1961, pp.
361–379.
[199] B. Kulis, M. Sustik, and I. Dhillon, “Learning low-rank kernel matrices,” inProceedings of International Conference on Machine Learning (ICML), 2006, pp.
505–512.
[200] Y. Takahashi, H. Saruwatari, K. Shikano, and K. Kondo, “Musicalnoise analysis in methods of integrating microphone array and spectral
subtraction based on higher-order statistics,” EURASIP Journal on Advancesin Signal Processing, vol. 2010, 2010, article ID 431347.
[201] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement inblind audio source separation,” IEEE Transactions on Audio, Speech, andLanguage Processing, vol. 14, no. 4, pp. 1462–1469, 2006.
[202] S. Nakamura, K. Hiyane, F. Asano, T. Nishiura, and T. Yamada, “Acousticalsound database in real environments for sound scene understanding andhands-free speech recognition,” in Proceedings of International Conference on
Language Resources and Evaluation (LREC), 2000, pp. 965–968.
[203] P. D. O’Grady and B. A. Pearlmutter, “Soft-LOST: EM on a mixture oforiented lines,” in Proceedings of Independent Component Analysis and Blind
Signal Separation (ICA), 2004, pp. 430–436.
[204] J.-F. Cardoso and A. Souloumiac, “Blind beamforming for non-gaussian
signals,” IEE Proceedings F - Radar and Signal Processing, vol. 140, no. 6, pp.
362–370, 1993.
[205] D. B. Ward, R. A. Kennedy, and R. C. Williamson, “Constant directivity
beamforming,” in Microphone Arrays: Signal Processing Techniques andApplications, M. Brandstein and D. Ward, Eds. Springer-Verlag Berlin
Heidelberg, 2001, ch. 1, pp. 3–17.
[206] S. Araki, R. Mukai, S. Makino, T. Nishikawa, and H. Saruwatari, “Thefundamental limitation of frequency domain blind source separation for

172

Bibliography
convolutive mixtures of speech,” IEEE Transactions on Speech and Audio
Processing, vol. 11, no. 2, pp. 109–116, 2003.

[207] D. Kitamura, N. Ono, H. Sawada, H. Kameoka, and H. Saruwatari,“Relaxation of rank-1 spatial constraint in overdetermined blind source
separation,” in Proceedings of The European Signal Processing Conference(EUSIPCO), 2015, pp. 1271–1275.
[208] H. Kameoka, T. Nakatani, and T. Yoshioka, “Robust speech dereverberation
based on non-negativity and sparse nature of speech spectrograms,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2009, pp. 45–48.
[209] B. P. Bogert, M. J. R. Healy, and J. W. Tukey, “The quefrency analysis oftime series for echoes: Cepstrum, pseudo-autocovariance, cross-cepstrum,and saphe-cracking,” in Proceedings of the Symposium on Time Series Analysis,
M. Rosenblatt, Ed. Wiley, New York, 1963, ch. 15, pp. 209–243.
[210] D. G. Childers, D. P. Skinner, and R. C. Kemerait, “The cepstrum: A guideto processing,” Proceedings of the IEEE, vol. 65, no. 10, pp. 1428–1443, 1977.
[211] S. Furui, “Cepstral analysis technique for automatic speaker verification,”
IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 29, no. 2,
pp. 254–272, 1981.
[212] P. Mermelstein, “Distance measures for speech recognition, psychologicaland instrumental,” Pattern Recognition and Artificial Intelligence, vol. 116, pp.
374–388, 1976.
[213] S. Davis and P. Mermelstein, “Comparison of parametric representations
for monosyllabic word recognition in continuously spoken sentences,”
IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 28, no. 4,
pp. 357–366, 1980.
[214] P. O. Hoyer, “Non-negative matrix factorization with sparseness constraints,” Journal of Machine Learning Research, vol. 5, no. Nov, pp. 1457–1469,
2004.

Bibliography173[215] P. Smaragdis, M. Shashanka, and B. Raj, “A sparse non-parametric approachfor single channel separation of known sounds,” in Advances in Neural
Information Processing Systems (NIPS), 2009, pp. 1705–1713.
[216] B. King and L. Atlas, “Single-channel source separation using simplifiedtraining complex matrix factorization,” in Proceedings of IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), 2010, pp.
4206–4209.
[217] B. Raj, T. Virtanen, S. Chaudhuri, and R. Singh, “Non-negative matrix factorization based compensation of music for automatic speech recognition,”in Proceedings of Interspeech, 2010, pp. 717–720.
[218] D. Kitamura, H. Saruwatari, K. Yagi, K. Shikano, Y. Takahashi, andK. Kondo, “Robust music signal separation based on supervised nonnegative matrix factorization with prevention of basis sharing,” in Proceedings ofIEEE International Symposium on Signal Processing and Information Technology(ISSPIT), 2013, pp. 392–397.
[219] B. Colson, P. Marcotte, and G. Savard, “An overview of bilevel optimization,”
Annals of Operations Research, vol. 153, no. 1, pp. 235–256, 2007.
[220] M. D. Plumbley, “Algorithms for nonnegative independent component
analysis,” IEEE Transactions on Neural Networks, vol. 14, no. 3, pp. 534–543,
2003.
[221] E. Oja and M. D. Plumbley, “Blind separation of positive sources by
globally convergent gradient search,” Neural Computation, vol. 16, no. 9, pp.
1811–1825, 2004.
[222] Z. Yuan and E. Oja, “A FastICA algorithm for non-negative independent
component analysis,” in Proceedings of International Conference on Independent
Component Analysis and Signal Separation (ICA), 2004, pp. 1–8.
[223] C. Boutsidis and E. Gallopoulos, “SVD based initialization: a head start for
nonnegative matrix factorization,” Pattern Recognition, vol. 41, no. 4, pp.
1350–1362, 2008.

174

Bibliography[224] K. Stadlthanner, F. J. Theis, E. W. Lang, A. M. Tomé, C. G. Puntonet,P. G. Vilda, T. Langmann, and G. Schmitz, “Sparse nonnegative matrix
factorization applied to microarray data sets,” in Proceedings of International
Conference on Independent Component Analysis and Signal Separation (ICA),
2006, pp. 254–2618.
[225] A. Janecek and Y. Tan, “Using population based algorithms for initializing
nonnegative matrix factorization,” in Proceedings of International Conference
in Swarm Intelligence (ICSI), 2011, pp. 307–316.
[226] Y. Xue, C. S. Tong, Y. Chen, and W. S. Chen, “Clustering-based initialization
for non-negative matrix factorization,” Applied Mathematics and Computation,
vol. 205, no. 2, pp. 525–536, 2008.
[227] Z. Zheng, J. Yang, and Y. Zhu, “Initialization enhancer for non-negative
matrix factorization,” Engineering Applications of Artificial Intelligence,
vol. 20, no. 1, pp. 101–110, 2007.
[228] M. Rezaei, R. Boostani, and M. Rezaei, “An efficient initialization method
for nonnegative matrix factorization,” Journal of Applied Sciences, vol. 11,
no. 2, pp. 354–359, 2011.
[229] G. Casalino, N. D. Buono, and C. Mencar, “Subtractive clustering forseeding non-negative matrix factorizations,” Journal of Applied Sciences, vol.
257, pp. 369–387, 2014.
[230] L. Zhao, G. Zhuang, and X. Xu, “Facial expression recognition based on
PCA and NMF,” in Proceedings of World Congress on Intelligent Control andAutomation (WCICA), 2008, pp. 6826–6829.
[231] D. Donoho and V. Stodden, “When does non-negative matrix factorization
give a correct decomposition into parts?” in Proceedings of Advances in
Neural Information Processing Systems (NIPS), 2003.
[232] C. Bauckhage, “A purely geometric approach to non-negative matrix
factorization,” in Proceedings of LWA, 2014, pp. 125–136.

Bibliography

175[233] P. Smaragdis, B. Raj, and M. Shashanka, “A probabilistic latent variable
model for acoustic modeling,” Advances in Models for Acoustic Processing,
NIPS, vol. 148, 2006.
[234] Y. Kawaguchi, M. Togami, H. Nagano, Y. Hashimoto, M. Sugiyama, and
Y. Takada, “Ica-based acceleration of probabilistic latent component
analysis for mass spectrometry-based explosives detection,” in Proceedingsof IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2013, pp. 2795–2799.
[235] N. Ono, Z. Rafii, D. Kitamura, N. Ito, and A. Liutkus, “The 2015 signal
separation evaluation campaign,” in Proceedings of International Conferenceon Latent Variable Analysis and Signal Separation (LVA/ICA), 2015, pp.
387–395.
[236] K. Matsuoka and S. Nakashima, “Minimal distortion principle for blindsource separation,” in Proceedings of International Symposium on ICA and
BSS, 2001, pp. 722–727.
[237] K. Matsuoka, “Minimal distortion principle for blind source separation,”in Proceedings of the 41st SICE Annual Conference SICE 2002, vol. 4, 2002, pp.
2138–2143.
[238] T. Takatani, T. Nishikawa, H. Saruwatari, and K. Shikano, “High-fidelity
blind source separation of acoustic signals using SIMO-model-based
ICA with information-geometric learning,” in Proceedings of International
Workshop on Acoustic Echo and Noise Control (IWAENC), 2003, pp. 251–254.
[239] T. Takatani, T. Nishikawa, H. Saruwatari, and K. Shikano, “Blind separationof binaural sound mixtures using SIMO-model-based independent
component analysis,” in Proceedings of IEEE International Conference on
Acoustics, Speech, and Signal Processing (ICASSP), vol. IV, 2004, pp. 113–116.
[240] T. Takatani, T. Nishikawa, H. Saruwatari, and K. Shikano, “High-fidelity
blind separation of acoustic signals using SIMO-model-based independent
component analysis,” IEICE Transactions on Fundamentals of Electronics,

176

Bibliography
Communications and Computer Sciences, vol. E87-A, no. 8, pp. 2063–2072,
2004.

[241] T. Takatani, S. Ukai, T. Nishikawa, H. Saruwatari, and K. Shikano, “A
selfgenerator method for initial filters of SIMO-ICA applied to blindseparation of binaural sound mixtures,” IEICE Transactions on Fundamentals
of Electronics, Communications and Computer Sciences, vol. E88-A, no. 7, pp.
1673–1682, 2005.
[242] K. Matsuoka and S. Nakashima, “Overdetermined blind separation of
acoustic signals based on MISO-constrained frequency-domain ICA,” inProceedings of International Congress on Acoustics (ICA), 2004, pp. IV–3143–
IV–3146.
[243] P. S. Huang, S. D. Chen, P. Smaragdis, and M. H.-Johnson, “Singing-voice
separation from monaural recordings using robust principal component
analysis,” in Proceedings of IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP), 2012, pp. 57–60.
[244] Y. Ikemiya, K. Yoshii, and K. Itoyama, “Singing voice analysis and editing
based on mutually dependent F0 estimation and source separation,” inProceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2015, pp. 574–578.
[245] J. Wright, A. Ganesh, S. Rao, Y. Peng, and Y. Ma, “Robust principal
component analysis: Exact recovery of corrupted low-rank matrices via
convex optimization,” in Proceedings of Advances in Neural Information
Processing Systems (NIPS), 2009, pp. 2080–2088.
[246] H. Xu, C. Caramanis, and S. Sanghavi, “Robust PCA via outlier pursuit,”
in Proceedings of Advances in Neural Information Processing Systems (NIPS),
2010, pp. 2496–2504.
[247] A. Liutkus, D. FitzGerald, and R. Badeau, “Cauchy nonnegative matrix
factorization,” in Proceedings of IEEE Workshop on Applications of Signal
Processing to Audio and Acoustics (WASPAA), 2015.

177

A
Derivation of Shape Parameter for
Artificial Random Spectrogram with
Constant Kurtosis

To produce an artificial random spectrogram F G with constant kurtosis, wederive the optimal shape parameter κ for each value of R. Hereafter, we denoteÍan I × J matrix whose elements are fir gr j as Fr Gr , namely, F G = r Fr Gr . Also,
we denote a pth-order moment and pth-order cumulant of Fr Gr as µ pr and cpr
and those of F G as µ0p and c0p , respectively. When R increases beyond one, theÍRFr Gr . Therefore, thematrix F G becomes a linear combination expressed as r=1
kurtosis of F G can be derived via the moment-cumulant transform [200]. Since
fir and gr j are generated from i.i.d. gamma distributions, µ pr is equal to the178

Chapter A. Derivation of Shape Parameter for Artificial Random
Spectrogram with Constant Kurtosis

product of pth-order moments of Fr and Gr as follows:µ pr = θ

2p

p−1
Ö

(κ + q)2 .

(A.1)

q=0

By the moment-cumulant transform, cpr from p = 1 to p = 4 can be representedas
c1r = µ1r ,(A.2)

c2r = µ2r − µ21r ,(A.3)

c3r = µ3r − 3µ1r µ2r + 2µ31r ,(A.4)

c4r = µ4r − 4µ1r µ3r − 3µ22r + 12µ21r µ2r − 6µ41r .

(A.5)

Since a cumulant satisfies additivity for the variables, the cumulant of F G iseasily derived as follows:
c0p =

R
Õ

cpr = Rcpr .

(A.6)

r=1

The moments of F G for p = 2 and p = 4 are given by the moment-cumulant
transform asµ01 = c10 ,(A.7)µ02 = c20 + c021,(A.8)µ03 = c30 + 3c10 c20 + c10 3,(A.9)µ04 = c40 + 3c022 + 4c10 c30 + 6c021 c20 + c041 .

(A.10)

Finally, the kurtosis of F G can be derived as
kurtosis(F G) =

µ04

=
2

µ02

ζ(κ, R).
ξ(κ, R)(A.11)

Therefore, by solving (3.92), we can obtain the shape parameter κ so that the
kurtosis of F G has the same value (kurt) for any value of R.

179

Publication List

Book Chapters[B.1] Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka,
and Hiroshi Saruwatari, “Determined blind source separation withindependent low-rank matrix analysis,” Audio Source Separation, Shoji
Makino, Eds. Springer, 31 pages (to appear).
[B.2] Daichi Kitamura, “Q11 ビームフォーミングって何ですか？,” Acousticpedia for Beginners, Acoustical Society of Japan, Eds. Corona Publishing,pp. 44–47, March 2017 (in Japanese).

Journal Papers[J.1] Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka,
and Hiroshi Saruwatari, “Determined blind source separation unifying independent vector analysis and nonnegative matrix factorization,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24,
no. 9, pp. 1626–1641, September, 2016.
[J.2] Yoshiaki Bando, Hiroshi Saruwatari, Nobutaka Ono, Shoji Makino,
Katsutoshi Itoyama, Daichi Kitamura, Masaru Ishimura, Moe Takakusaki,
Narumi Mae, Kouei Yamaoka, Yutaro Matsui, Yuichi Ambe, Masashi
Konyo, Satoshi Tadokoro, Kazuyoshi Yoshii, and Hiroshi G. Okuno, “Lowlatency and high-quality two-stage human-voice-enhancement system for180

Publication Lista hose-shaped rescue robot,” Journal of Robotics and Mechatronics, vol. 27,
no. 1, pp.198–212, February 2017.

Peer-Reviewed International Conference Proceedings[C.1] Daichi Kitamura, Nobutaka Ono, and Hiroshi Saruwatari, “Experimental
analysis of optimal window length for independent low-rank matrix
analysis,” in The 2017 European Signal Processing Conference (EUSIPCO), Kos,
Greece, September 2017 (invited special session, submitted).
[C.2] Daichi Kitamura and Nobutaka Ono, “Efficient initialization for nonnegative matrix factorization based on nonnegative independent component
analysis,” in Proceedings of The 15th International Workshop on Acoustic Signal
Enhancement (IWAENC), Xi’an, China, September 2016.
[C.3] Daichi Kitamura, Nobutaka Ono, Hiroshi Saruwatari, Yu Takahashi, andKazunobu Kondo, “Discriminative and reconstructive basis training
for audio source separation with semi-supervised nonnegative matrix
factorization,” in Proceedings of The 15th International Workshop on Acoustic
Signal Enhancement (IWAENC), Xi’an, China, September 2016.
[C.4] Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka,
and Hiroshi Saruwatari, “Relaxation of rank-1 spatial constraint in
overdetermined blind source separation,” in Proceedings of The 2015
European Signal Processing Conference (EUSIPCO), pp. 1271–1275, Nice,
France, September 2015 (invited special session).
[C.5] Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka,
and Hiroshi Saruwatari, “Efficient multichannel nonnegative matrix
factorization exploiting rank-1 spatial model,” in Proceedings of IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP),
pp. 276–280, Brisbane, Australia, April 2015.
[C.6] Yoshiki Mitsui, Daichi Kitamura, Shinnosuke Takamichi, Nobutaka Ono,and Hiroshi Saruwatari, “Blind source separation based on independentPublication List

181

low-rank matrix analysis with sparse regularization for time-series activity,”in Proceedings of IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), pp. 21–25, New Orleans, U.S.A., March, 2017.
[C.7] Antoine Liutkus, Fabian-Robert Stöter, Zafar Rafii, Daichi Kitamura,
Bertrand Rivet, Nobutaka Ito, Nobutaka Ono, and Julie Fontecave, “The2016 signal separation evaluation campaign,” in Proceedings of 13th
International Conference on Latent Variable Analysis and Signal Separation(LVA/ICA), pp. 323–332, Grenoble, France, February, 2017.
[C.8] Narumi Mae, Masaru Ishimura, Daichi Kitamura, Nobutaka Ono, Takeshi
Yamada, Shoji Makino, and Hiroshi Saruwatari, “Ego noise reduction for
hose-shape rescue robot combining independent low-rank matrix analysis
and multichannel noise cancellation,” in Proceedings of 13th International
Conference on Latent Variable Analysis and Signal Separation (LVA/ICA),
pp. 141–151, Grenoble, France, February, 2017.
[C.9] Narumi Mae, Daichi Kitamura, Masaru Ishimura, Takeshi Yamada, andShoji Makino, “Ego noise reduction for hose-shaped rescue robot combining
independent low-rank matrix analysis and noise cancellation,” in Proceedingsof Asia-Pacific Signal and Information Process Processing Association Annual
Summit and Conference (APSIPA ASC), Jeju, Korea, December 2016
[C.10] Hiroaki Nakajima, Daichi Kitamura, Norihiro Takamune, Shoichi Koyama,
Hiroshi Saruwatari, Yu Takahashi, and Kazunobu Kondo, “Audio signal
separation using supervised NMF with time-variant all-pole-model-based
basis deformation,” in Proceedings of Asia-Pacific Signal and Information
Process Processing Association Annual Summit and Conference (APSIPA ASC),
Jeju, Korea, December 2016
[C.11] Moe Takakusaki, Daichi Kitamura, Nobutaka Ono, Takeshi Yamada, Shoji
Makino, and Hiroshi Saruwatari, “Ego-noise reduction for a hose-shaped
rescue robot using determined rank-1 multichannel nonnegative matrix
factorization,” in Proceedings of The 15th International Workshop on Acoustic
Signal Enhancement (IWAENC), Xi’an, China, September 2016.

182

Publication List

[C.12] Hiroaki Nakajima, Daichi Kitamura, Norihiro Takamune, Shoichi Koyama,
Hiroshi Saruwatari, Nobutaka Ono, Yu Takahashi, and Kazunobu Kondo,“Music signal separation using supervised NMF with all-pole-model-based
discriminative basis deformation,” in Proceedings of The 2016 European
Signal Processing Conference (EUSIPCO), pp.1143–1147, Budapest, Hungary,August 2016.
[C.13] Nobutaka Ono, Zafar Rafii, Daichi Kitamura, Nobutaka Ito, and Antoine
Liutkus, “The 2015 signal separation evaluation campaign,” in Proceedings of
12th International Conference on Latent Variable Analysis and Signal Separation(LVA/ICA), Liberec, Czech, August 2015.

Non-Reviewed Domestic Workshop Proceedings andTechnical Reports[D.1] Daichi Kitamura, Nobutaka Ono, Hiroshi Saruwatari, Yu Takahashi, andKazunobu Kondo, “Effective basis learning for sound source separation bysemi-supervised nonnegative matrix factorization,” IEICE Technical Report,
EA2015-130, vol. 115, no. 521, pp. 355–360, Oita, March 2016 (in Japanese).
[D.2] Daichi Kitamura and Nobutaka Ono, “Statistical-independence-based
effective initialization for nonnegative matrix factorization,” Proceedings of2016 Spring Meeting of Acoustical Society of Japan (ASJ), 3-3-5, pp. 619–622,
Kanagawa, March 2016 (in Japanese).
[D.3] Daichi Kitamura, Hiroshi Saruwatari, Nobutaka Ono, Hiroshi Sawada,and Hirokazu Kameoka, “Study on source and spatial models for BSS
with rank-1 spatial approximation,” Proceedings of 2015 Autumn Meeting of
Acoustical Society of Japan (ASJ), 3-6-10, pp. 583–586, Fukushima, September
2015 (in Japanese).
[D.4] Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka,
and Hiroshi Saruwatari, “Relaxation of rank-1 spatial model in overdeter-Publication List

183

mined BSS,” Proceedings of 2015 Spring Meeting of Acoustical Society of Japan(ASJ), 3-10-11, pp. 629–632, Tokyo, March 2015 (in Japanese).
[D.5] Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka,
and Hiroshi Saruwatari, “Efficient multichannel nonnegative matrix
factorization with rank-1 spatial model,” Proceedings of 2014 Autumn
Meeting of Acoustical Society of Japan (ASJ), 2-1-11, pp. 579–582, Hokkaido,
September 2014 (in Japanese).
[D.6] Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka,
and Hiroshi Saruwatari, “Performance evaluation of source separation in
multichannel nonnegative matrix factorization with rank-1 spatial model,”Proceedings of 107th IPSJ Special Interest Group on Music and Computer(IPSJ-SIGMUS), vol. 2015-MUS-107, no. 31, Tokyo, May 2015 (in Japanese).
[D.7] Yoshiki Mitsui, Satoshi Mizoguchi, Hiroshi Saruwatari, Daichi Kitamura,
Nobutaka Ono, Masaru Ishimura, Narumi Mae, Moe Takakusaki, Yutaro
Matsui, Kouei Yamaoka, and Shoji Makino, “Development of blind source
separation system for flexible hose-shaped robot using independent
low-rank matrix analysis and statistical speech enhancement,” Proceedingsof 2017 Spring Meeting of Acoustical Society of Japan (ASJ), 1-P-3, pp. 517–518,
Kanagawa, March 2017 (in Japanese).
[D.8] Yoshiki Mitsui, Daichi Kitamura, Shinnosuke Takamichi, Nobutaka Ono,and Hiroshi Saruwatari, “Study on efficient solver for independent low-rankmatrix analysis with sparse time-series-activity regularization,” Proceedingsof 2016 Autumn Meeting of Acoustical Society of Japan (ASJ), 1-7-3, pp. 325–328,
Toyama, September 2016 (in Japanese).
[D.9] Yoshiki Mitsui, Daichi Kitamura, Shinnosuke Takamichi, and Hiroshi
Saruwatari, “Blind source separation using independent low-rank matrix
analysis with sparse regularization for time-series activations,” IEICE
Technical Report, EA2016-72, vol. 116, no. 420, pp. 25–30, Kyoto, January
2017 (in Japanese).

184

Publication List

[D.10] Moe Takakusaki, Daichi Kitamura, Nobutaka Ono, Takeshi Yamada, Shoji
Makino, and Hiroshi Saruwatari, “Noise reduction for a hose-shaped
rescue robot using rank-1 multichannel NMF,” Proceedings of The 2016
JSME Conference on Robotics and Mechatronics (ROBOMECH 2016), no. 16-2,
1A2-10a3, Kanagawa, June 2016 (in Japanese).
[D.11] Moe Takakusaki, Daichi Kitamura, Nobutaka Ono, Takeshi Yamada,
Shoji Makino, and Hiroshi Saruwatari, “Noise reduction using rank-1
multichannel NMF for a hose-shaped rescue robot,” Proceedings of 2016
IEICE General Conference, A-5-1, Fukuoka, March 2016 (in Japanese).
[D.12] Kazuma Takata, Daichi Kitamura, Hiroaki Nakajima, Shoichi Koyama,
Hiroshi Saruwatari, Nobutaka Ono, and Shoji Makino, “Source separation
with supervised multichannel NMF and statistical speech enhancement
method for hose-shaped rescue robot,” Proceedings of 2016 Spring Meeting of
Acoustical Society of Japan (ASJ), 3-3-2, pp. 609-612, Kanagawa, March 2016
(in Japanese).
[D.13] Hiroaki Nakajima, Daichi Kitamura, Norihiro Takamune, Shoichi Koyama,
Hiroshi Saruwatari, Nobutaka Ono, Yu Takahashi, and Kazunobu Kondo,“Music signal separation using supervised NMF with time-valiant all-polemodel-based basis deformation,” Proceedings of 2016 Spring Meeting of
Acoustical Society of Japan (ASJ), 3-3-11, pp. 635-638, Kanagawa, March 2016
(in Japanese).
[D.14] Hiroaki Nakajima, Daichi Kitamura, Norihiro Takamune, Shoich Koyama,
Hiroshi Saruwatari, Nobutaka Ono, Yu Takahashi, and Kazunobu Kondo,“Analysis on degree of freedom for supervised NMF with all-pole-modelbased basis deformation,” IEICE Technical Report, EA2015-42, vol. 115,
no. 359, pp. 13–18, Kanazawa, December 2015 (in Japanese).

185Awards Received

1. The 7th Ikushi Prize from Japan Society for the Promotion of Science (JSPS),March 2017.
2. The Telecom System Technology Student Award from The Telecommunication Advancement Foundation (TAF), March 2016.
3. Student Conference Paper Award from IEEE Signal Processing Society(SPS) Japan Chapter, November 2015.
4. 2015 The 1st Best Student Award from National Institute of Informatics(NII), September 2015.
5. The 37th Awaya Prize Young Researcher Award from The AcousticalSociety of Japan (ASJ), March 2015.
6. 2013 Technical Group on Signal Processing Young Researcher’s Award fromThe Institute of Electronics, Information and Communication Engineers(IEICE), November 2014.