A Review of Verbal and Non-Verbal Human-Robot Interactive Communication

In this paper, an overview of human-robot interactive communication is presented, covering verbal as well as non-verbal aspects. Following a historical introduction,and motivation towards fluid human-robot communication, ten desiderata are proposed, which provide an organizational axis both of r ecent as well as of future research on human-robot communication. Then, the ten desiderata are examined in detail, culminatin g to a unifying discussion, and a forward-looking conclusion. I. I NTRODUCTION: HISTORICAL OVERVIEW While the first modern-day industrial robot, Unimate, began work on the General Motors assembly line in 1961, and was conceived in 1954 by George Devol [1], [2], the concept of a robot has a very long history, starting in mythology and folklore, and the first mechanical predecessors (automa ta) having been constructed in Ancient Times. For example, in Greek mythology, the God Hephaestus is reputed to have made mechanical servants from gold ([3] in p.114, and [4] verse 18.419). Furthermore, a rich tradition of designing a d building mechanical, pneumatic or hydraulic automata also exists: from the automata of Ancient Egyptian temples, to th e mechanical pigeon of the Pythagorean Archytas of Tarantum circa 400BC [5], to the accounts of earlier automata found in the Lie Zi text in China in 300BC [6], to the devices of Heron of Alexandria [7] in the 1st century. The Islamic world also plays an important role in the development of automata; AlJazari, an Arab inventor, designed and constructed numerou s automatic machines, and is even reputed to have devised the first programmable humanoid robot in 1206AD [8]. The word “robot”, a Slavic word meaning servitude, was first used in this context by the Czech author Karel Capek in 1921 [9]. However, regarding robots with natural-language conversa tional abilities, it wasnt until the 1990’s that the first pio neering systems started to appear. Despite the long history of mytho logy and automata, and the fact that even the mythological handmaidens of Hephaestus were reputed to have been given a voice [3], and despite the fact that the first general-purpo se electronic speech synthesizer was developed by Noriko Omed a in Japan in 1968 [10], it wasnt until the early 1990’s that conversational robots such as MAIA [11], RHINO [12], and AESOP [13] appeared. These robots cover a range of intended application domains; for example, MAIA was intended to carry objects and deliver them, while RHINO is a museum guide robot, and AESOP a surgical robot. In more detail, the early systems include Polly, a robotic guide that could give tours in offices [14], [15]. Polly had very simple interaction capacities; it could perceive huma n feet waving a “tour wanted” signal, and then it would just use pre-determined phrases during the tour itself. A slightly m ore advanced system was TJ [16]. TJ could verbally respond to simple commands, such as “go left”, albeit through a keyboar d. RHINO, on the other hand [12], could respond to tour-start commands, but then, again, just offered a pre-programmed to ur with fixed programmer-defined verbal descriptions. Regardi ng mobile assistant robots with conversational capabilities in the 1990s, a classic system is MAIA [11], [17], obeying simple commands, and carrying objects around places, as well as the mobile office assistant which could not only deliver parcels but guide visitors described in [18], and the similar in functio nality Japanese-language robot Jijo-2 [19], [20], [21]. Finally, an important book from the period is [22], which is characteristi c of the traditional natural-language semantics-inspired the oretical approaches to the problem of human-robot communication, and also of the great gap between the theoretical proposals and the actual implemented systems of this early decade. What is common to all the above early systems is that they share a number of limitations. First, all of them only accept a fixed and small number of simple canned commands , and they respond with a set of canned answers . Second, the only speech acts(in the sense of Searle [23]) that they can handle are requests. Third, the dialogue they support is cle arly ot flexibly mixed initiative; in most cases it is just humaninitiative. Four, they dont really support situated language , i.e. language about their physical situations and events th at are happening around them; except for a fixed number of canned location names in a few cases. Five, they are not able to handleaffective speech ; i.e. emotion-carrying prosody is either recognized nor generated. Six, their non-verbal communication[24] capabilities are almost non-existent; for example, gestures, gait, facial expressions, and head nods are neither recognized nor produced. And seventh, their dialog ue systems are usually effectively stimulus-response or stim ulusstate-response systems; i.e. no real speech planningor purposeful dialogue generation is taking place, and certainly not in conjunction with the motor planning subsystems of the robot. Last but quite importantly, no real learning, off-line or on-the-fly is taking place in these systems; verbal behavior s have to be prescribed. All of these shortcomings of the early systems of the 1990s, effectively have become desiderata for the next two decades of research: the 2000s and 2010s, which we are in at the moment. Thus, in this paper, we will start by providing a discussion giving motivation to the need for existence of interactive r obots with natural human-robot communication capabilities, and then we will enlist a number of desiderata for such systems, which have also effectively become areas of active research in the last decade. Then, we will examine these desiderata one by one, and discuss the research that has taken place towards their fulfillment. Special consideration will be given to th e socalled “symbol grounding problem” [25], which is central to most endeavors towards natural language communication wit h physically embodied agents, such as robots. Finally, after a discussion of the most important open problems for the futur e, we will provide a concise conclusion. II. M OTIVATION : INTERACTIVE ROBOTS WITH NATURAL LANGUAGE CAPABILITIES BUT WHY? There are at least two avenues towards answering this fundamental question, and both will be attempted here. The first avenue will attempt to start from first principles and derive a rationale towards equipping robots with natural language . The second, more traditional and safe avenue, will start fro m a concrete, yet partially transient, base: application dom ains existing or potential. In more detail: Traditionally, there used to be clear separation between design and deployment phases for robots. Application-spec ific robots (for example, manufacturing robots, such as [26]) were: (a) designed by expert designers, (b) possibly tailor programmed and occasionally reprogrammed by specialist engineers at their installation site, and (c) interacted wi th their environment as well as with specialized operators during ac tual operation. However, the phenomenal simplicity but also the accompanying inflexibility and cost of this traditional set ting is often changing nowadays. For example, one might want to have broader-domain and less application-specific robot s, necessitating more generic designs, as well as less effort b y the programmer-engineers on site, in order to cover the vari ous contexts of operation. Even better, one might want to rely less on specialized operators, and to have robots interact a nd collaborate with non-expert humans with little if any prior training. Ideally, even the actual traditional programmin g and re-programming might also be transferred over to non-exper t humans; and instead of programming in a technical language, to be replaced by intuitive tuition by demonstration, imita tion and explanation [27], [28], [29]. Learning by demonstratio n and imitation for robots already has quite some active resea ch; but most examples only cover motor and aspects of learning, and language and communication is not involved deeply. And this is exactly where natural language and other forms of fluid and natural human-robot communication enter the picture: Unspecialized non-expert humans are used to (and quite good at) teaching and interacting with other humans through a mixture of natural language as well as nonverbal signs. Thus, it makes sense to capitalize on this existing ab ility of non-expert humans by building robots that do not require humans to adapt to them in a special way, and which can fluidly collaborate with other humans, interacting with the m and being taught by them in a natural manner, almost as if they were other humans themselves. Thus, based on the above observations, the following is one classic line of motivation towards justifying efforts f or equipping robots with natural language capabilities: Why n ot build robots that can comprehend and generate human-like interactive behaviors, so that they can cooperate with and b e taught by non-expert humans, so that they can be applied in a wide range of contexts with ease? And of course, as natural language plays a very important role within these behaviors, why not build robots that can fluidly converse wit h humans in natural language, also supporting crucial non-ve rbal communication aspects, in order to maximize communication effectiveness, and enable their quick and effective applic ation? Thus, having presented the classical line of reasoning arriving towards the utility of equipping robots with natural language capabilities, and having discussed a space of possibilities regarding role assignment between human and rob ot, let us now move to the second, more concrete, albeit less general avenue towards justifying conversational robots: nam ely, specific applications, existing or potential. Such applica tions, where natural human-robot interaction capabilities with v erbal and non-verbal aspects would be desirabl