Instantaneous and frequency-warped signal processing techniques for auditory source separation

This thesis summarizes several contributions to the areas of signal processing and auditory source separation. The philosophy of Frequency-Warped Signal Processing is introduced as a means for separating the AM and FM contributions to the bandwidth of a complex-valued, frequency-varying sinusoid p[n], transforming it into a signal with slowly-varying parameters. This transformation facilitates the removal of p[n] from an additive mixture while minimizing the amount of damage done to other signal components. The average winding rate of a complex-valued phasor is explored as an estimate of the instantaneous frequency. Theorems are provided showing the robustness of this measure. To implement frequency tracking, a Frequency-Locked Loop algorithm is introduced which uses the complex winding error to update its frequency estimate. The input signal is dynamically demodulated and ltered to extract the envelope. This envelope may then be remodulated to reconstruct the target partial, which may be subtracted from the original signal mixture to yield a new, quicklyadapting form of notch ltering. Enhancements to the basic tracker are made which, under certain conditions, attain the Cram er-Rao bound for the instantaneous frequency estimate. To improve tracking, the novel idea of Harmonic-Locked Loop tracking, using N harmonically constrained trackers, is introduced for tracking signals, such as voices and certain musical instruments. The estimated fundamental frequency is computed from a maximum-likelihood weighting of the N tracking estimates, making it highly robust. The result is that harmonic signals, such as voices, can be isolated from complex mixtures in the presence of other spectrally overlapping signals. Additionally, since phase information is preserved, the resynthesized harmonic signals may be removed from the original mixtures with relatively little damage to the residual signal. Finally, a new methodology is given for designing linear-phase FIR lters which require a small fraction of the computational power of conventional FIR implementations. This design strategy is based on truncated and stabilized IIR lters. These signal-processing methods have been applied to the problem of auditory source separation, resulting in voice separation from complex music that is signi cantly better than previous results at far lower computational cost. v Acknowledgments There are many people who were involved in providing an environment in which this thesis and the ideas contained within could be realized. I would like to extend my profound gratitude to my advisor, Julius O. Smith, III, for in nite patience, kindness, encouragement, friendship, enthusiasm, and encyclopedic knowledge of signal processing. Despite being in high demand, he always made the time to advise and share his broadband wisdom in matters beyond just signal processing. His excellent advising during the two years I have been at CCRMA were among the salient factors motivating the most creative and productive months of my research career to date. My appreciation goes to the faculty, sta , and students at the Center for Computer Research in Music and Acoustics, John Chowning, Chris Chafe, and Heidi Kugler, and many others. The interdisciplinary blend of musicians and engineers from all over the world created an open, friendly, and relaxed environment in which to work. I would especially like to thank Fernando Pablo LopezLezcano, our jack-of-all-trades, resident composer, and Unix sysadmin, who helped greatly and showed great patience and trust as I continually performed minor surgeries on our NeXT network. I am grateful for the CCRMA DSP Seminar, organized by Julius, and its participants. It was in this relaxed, friendly, and well-informed forum that I was able to beta-test my craziest ideas without fear of unconstructive criticism. Thanks, also, go to Malcolm Slaney for running the CCRMA Hearing Seminar. I would like to honor the memory of the late Allen M. Peterson, who was my advisor during the early, amorphous stage of my graduate student career. Professor Peterson was always exible and encouraging, allowing me to explore many di erent areas while I was trying to decide on my specialization. This freedom enabled me to masquerade as a neuroscientist for some time with no pressure to nd a direct link with Electrical Engineering, letting me explore new horizons and form new insights. Professor Peterson will be remembered for his unfailing kindness, encouragement, and humor, and for his many achievements, including his leadership of STAR Lab. I would also like to acknowledge the Stanford Neurobiology Department, and especially Bill Newsome, Ken Britten,1 who provided a great opportunity for an outsider to learn about the fascinating world of neurobiology. Ken and Bill were patient and friendly with me while I worked in their lab doing data analysis. I would like to thank the National Science Foundation for their generous graduate fellowship which allowed me three years of no-strings-attached funding, during which I was free to explore many di erent areas inside and outside of Electrical Engineering. The creativity expressed in this thesis may not have been possible without that freedom. I would also like to thank the CCRMA A liates, whose generosity, along with that of the Stanford Electrical Engineering Department, provided funding for my last 6 months of graduate work. 1Presently at UC Davis. vii I extend my gratitude to Professor Christoph von der Malsburg of the Institut f ur Neuroinformatik at the Ruhr-Universit at Bochum, in Bochum, Germany, under whom I studied from August of 1990 to September of 1992 while on a Fulbright scholarship. I would like to thank him, the German Fulbright Kommission, and its director, Reiner Rohr, for generously providing me the opportunity to learn about another culture and language, as well as giving me free reign and resources to pursue my scienti c interests. I also express my thanks to Christoph for being patient with me while I experienced culture shock and went o in directions which were orthogonal to his group's main research focus. I would also like to acknowledge my colleagues and friends there, especially Martin Lades (now at Lawrence Livermore), the Halbauslander whose openness and friendship I value highly, and Ren e Doursat, the amazing and unconventional Frenchman whose kindness, creativity, humor, irreverence and discussions I will always cherish. Thanks also to Michael Neef, the patient sysadmin who ruled over a large, heterogeneous network with heterogeneous users. My gratitude goes to the members of my orals committee: Julius Smith, Stephen Boyd, John Chowning, Constance Chang-Hasnian, and the late Allen Peterson. Also, my appreciation goes to my reading committee: Julius Smith, Stephen Boyd, and especially Gene Franklin, who kindly and personally took on the responsibility of covering for Dr. Peterson on very short notice. I would like to express my thanks to my friends: you know who you are. Special thanks go to Francisco Marquez, Robert Sage, and Jim Spencer, who were my housemates during this project. Their support, exibility, and friendship were valuable in keeping me sane. I would like to extend my great appreciation to Timothy J. Hawks, who is among the most multi-dimensionally talented people I have had the privilege to befriend. Tim kept me focussed on the immediate goals when I didn't know which of many directions to take in my research. His comments and insights were always deep and valuable. I am profoundly thankful for his quick advice, proofreading, and suggestions. I would like to express my great thanks and love to my dear Katja Andrea Rault, who is enthusiastic, positive, curious, generous, in nitely loving, and supportive. I would like to thank her for her patient understanding and exibility, especially during the time this thesis was nally being condensed from vague notions into its nal draft. My greatest thanks are to my parents, who made all this possible through their incredible support. I am proud to be able to dedicate this dissertation to them. viii Preface This dissertation is one continuing step in the evolution of my excursion through the various domains of knowledge related to my life-long interest in how the mind works. The material presented here may seem rather removed from the problems of the philosophy of mind. Let me con rm that it is indeed largely irrelevant to such musings. However, let me also try to explain my perspective that makes the research presented here a plausible, if haphazard, continuation of my prime interest. If I were not as lazy as I am, I most likely would have studied biology as an undergraduate and continued with neurosciences in graduate school. Upon arriving at Stanford as an undergraduate I had decided to study psychology, and not mathematics, since I felt that I had had enough of math in high school and wanted to pursue somethingmore directly related to my most burning interest. After sitting in on a few sessions of Psych 1 I decided that biology would suit my goals better and shortly thereafter declared biology to be my major. I took the Bio Core at Stanford and found that I was worse at photographically memorizing pages out of biology texts than at organic chemistry, despite my enthusiasm and fascination for the mechanisms of life and the mind. My lack of discipline in forcing myself to memorize nally led me to the path of least resistance, whereupon I nally changed my major to mathematics{the subject in which most things are \obvious by inspection" or \left as an exercise to the reader. " My interest in the inner workings of the mind persisted in the background, leading me to courses in the Department of Electrical Engineering. I was interested in problems in cybernetics and thought that knowledge in the elds of control theory, information theory, and signal processing could be applied to problems in biological information processing. I fel