WHAT HAS MULTIMODAL RESEARCH GAINED BY CROSSING BORDERS ?

Multimodal research represents “ Science without Borders” because it requires combining expertise from different component technologies, academic disciplines, and cultural/international perspectives. It also is rapidly erasing borders as it promotes the increased accessibility of computing for diverse and non-specialist users, and for field and mobile usage environments. This paper reviews two studies that highlight recent advances within the field. It also draws parallels between the multimodal areas of speech/pen and speech/lip movement research. Finally, it indicates new research challenges that will require additional bold “border crossings” in the near future. In the medical community, there is an international group called Physicians without Borders that many of you undoubtedly are familiar with (URL: http://www.dwb.org/ ). Physicians without Borders//Medecins sans Frontiers is an organization of volunteer medical personnel who respond to medical needs and emergencies around the world, and who frequently “cross borders” to get where they’re going. Personally, I have always viewed them as an inspiration, in part because I like the idea of crossing borders to get where you need to go. Maybe it’s just the intrigue or element of risk, but I’d like to entertain the possibility that there’s something more substantial about crossing borders that should appeal to us. Apart from crossing national borders, which we all have just done to arrive here in Beijing for this new millennium meeting of ICSLP’2000, there also is the challenge of striving to cross intellectual borders when necessary to achieve rapid scientific progress. To most of us, the advantages of scientific crossfertilization and teamwork are abundantly clear, when accomplished with the right spirit. Multidisciplinary research that involves combining and shifting among very different perspectives can generate a fast-paced and creative style of research, as well as providing an antidote to intellectual insularity and parochialism. One might argue that the very existence of multimodal research, along with the advantages and opportunities that it represents, has been at least a small triumph of intellectual cross-fertilization. 1. WHY IS MULTIMODAL RESEARCH A SCIENCE WITHOUT BORDERS? In the most basic sense, of course, those of us working within multimodal research have needed to combine at least two modalities, each requiring intensive knowledge of a different component technology. Most often this has meant combining speech with vision-based technologies or pen input, since multimodal speech/pen systems [20] and speech/lip movement systems ([26], [27]) currently represent the two most mature areas within the field. In both types of multimodal system, one explicit goal has been to integrate complementary modalities in a manner that yields a synergistic blend, such that each mode can be capitalized upon and used to overcome weaknesses in the other mode. This approach to system design has promoted the philosophy of using modes and component technologies to their natural advantage, and of combining them in a manner that permits mutual compensation. One advantage of achieving such a blend is that the resulting multimodal architecture can function more robustly than individual recognition-based technologies, such as spoken language systems. I will return to this point in the section on “What has multimodal research gained by crossing borders? ” Another visible type of cross-fertilization within multimodal research is its rather obvious multidisciplinarity. Advancing the state-of-the-art of multimodal systems clearly requires multidisciplinary expertise in a variety of areas beyond computer science— including speech and hearing science, perception and vision, linguistics, psychology, signal processing, pattern recognition, and statistics. The multidisciplinary nature of our research agenda has several implications. To evolve successfully as a field, it means that computer science will need to become broader and more synthetic in its worldview, and to begin encouraging and rewarding researchers who successfully reach across the borders of their narrowly defined fields. It also means that any individual research group is unlikely to be able to conduct meaningful research across the entire spectrum. As a result, collaborative research and “community building” among multimodal researchers and sites will be critically needed to forge the necessary relations among those representing key component technologies, academic disciplines, and different cultural/international perspectives. There are other respects in which multimodal research is a science without borders too. Among the major goals of multimodal research is an increase in the accessibility of computing for diverse and non-specialist users, which effectively would work toward erasing the borders between computer literate and illiterate users. Multimodal interfaces are expected to increase the accessibility of computing for users of different ages, skill levels, cognitive styles, sensory and motor impairments, native languages, and even temporary illnesses. There are large individual differences in people’s ability and preference to use different modes of communication, and a multimodal interface provides the user with interaction choices that can circumvent personal limitations. Multimodal systems also can expand the usage contexts in which computing is viable  for example, in natural field settings and while users are mobile. They permit users to alternate modes, and to switch between modalities as needed during the changing conditions of mobile use. For example, since speech and pen input are complementary along many dimensions, their combination provides broad utility across varied usage contexts. A person may use hands-free speech input for voice dialing a car cell phone, but switch to pen input to avoid speaking private information during a public transaction. In a very physical sense, multimodal interfaces that include input modes well suited for mobility free us to cross the artificial borders that represent onand off-the-desktop computing, and potentially to incorporate computing more naturally into aspects of our daily lives. In brief, multimodal research represents a science without borders because it requires the breadth of knowledge to combine: • two or more modalities representing different component technologies, • diverse multidisciplinary perspectives, and • a multicultural/international perspective on natural communication patterns. In addition, multimodal research is busily erasing borders as it promotes the increased accessibility of computing for: • diverse and non-specialist users, and • varied usage contexts. 2. THE CHALLENGE: CURRENT SHORTCOMINGS OF SPOKEN LANGUAGE TECHNOLOGY When Physicians without Borders crosses national boundaries, their accomplishments can involve performing successful operations and sometimes saving lives. What can we claim has been our parallel success and reason for “crossing borders” intellectually in multimodal research? Most of us with ties to the speech community would acknowledge that traditional unimodal speech processing still is limited by the rate of recognition errors and lack of graceful error handling. Although speech tec hnology now often performs adequately for adult native speakers of a language, for read speech, or when users are speaking in idealized laboratory conditions, current estimates typically indicate a 20-50% decrease in recognition rates when speech is delivered: • during natural spontaneous interaction, • by diverse speakers (e.g., accented, children), or • in a natural field environment. During spontaneous interaction, speakers typically are engaged in real tasks, and this generates variability in their speech signal and dialogue for a host of reasons. For example, frequent miscommunication during a difficult task can prompt a speaker to hyperarticulate during their attempts at repair, which leads to durational and other signal adaptations [19]. Interpersonal tasks or stress also can be associated with fluctuating emotional states, giving rise to pitch adaptations [2]. In addition to the difficulties presented by spontaneous speech and speaker stylistic adaptations, it is widely recognized that laboratory assessments overestimate recognition rates in natural field settings [8]. Field environments usually involve variable noise levels, social interchange, multi-tasking and interruption of tasks, increased cognitive load and human performance errors, and other sources of stress, which also have been estimated to produce 20-50% drops in speech recognition accuracy. In fact, environmental noise currently is considered one of the primary obstacles to widespread commercialization of spoken language technology ([8], [11]). During field use and mobility, there actually are two main problems that each contribute to this degradation in system accuracy: • noise itself contaminates the speech signal, making it harder to process, and • people speak differently in noisy conditions in order to make themselves understood. “Stationary” noise sources (e.g., white noise) often can be modeled and processed successfully, when they can be predicted (e.g., road noise in a moving car). However, many noises in natural field environments are “non-stationary” ones that either change abruptly, or involve variable phase-in/phase-out noise as the user moves. Natural field environments also present qualitatively different sources of noise that cannot always be anticipated and modeled. During noise, speakers also have an automatic normalization response, called the “Lombard effect,” which causes systematic speech modifications including increased volume, reduced speaking rate, and changes in articulation and pitch [11]. The Lombard effect not only occurs in human adults, but also in young children, primates, and even quail. From an interface design standpoint, it i