Neural Circuits for Any-Time Phrase Recognition with Applications in Cognitive Models and Human-Robot Interaction

Neural Circuits for Any-Time Phrase Recognition with Applications in Cognitive Models and Human-Robot Interaction Richard Veale 1 and Matthias Scheutz 2 1 Cognitive riveale@indiana.edu Science Program, Indiana University Abstract Humans are remarkably good at recognizing spoken language, even in very noisy environments. Yet, artificial speech rec- ognizers do not reach human level performance, nor do they typically even attempt to model human speech processing. In this paper, we introduce a biologically plausible neural model of real-time spoken phrase recognition which shows how the time-varying spiking activity of neurons can be integrated into word tokens. We present a proof-of-concept implementation of the model, which shows promise both in terms of recog- nition accuracy as well as recognition speed. The model is also pragmatically useful to cognitive modelers who require robust any-time speech recognition for their models such as real-time models of human-robot interaction. We thus also present such an example of embedding our model in a larger cognitive model, along with offline analysis of its performance on a speech corpus. Keywords: Liquid State Machine; Neural Network Model; Any-Time Speech Recognition Introduction The mechanisms that convert physical signals such as light and sound into firing rates of neurons are well-studied. How- ever, the way these signals subsequently influence behavior, especially cognitive behavior, is less well understood. In particular, the progression from continuous real-time input streams at the physical transducer level to high-level cognitive processes that abstract over many physical characteristics is challenging, so much so that classical cognitive architectures simply assume higher-level representations such as word to- kens instead of modeling the processes that generate them. While these assumptions do not pose problems for disembod- ied models (e.g., of higher-level cognitive processes such as analogical reasoning in language), they are critical showstop- pers for embodied situated models that depend on being im- plemented on robots that interact with their environments in real-time (e.g., in the context of human-robot interactions in natural language). In such models, sensory processing must be performed one way or another, and while it is sometimes possible to substitute engineering solutions for biologically plausible sensory modules (e.g., artificial speech recognizers instead of biologically plausible models of speech recogni- tion), those substitutions often come at a price that assump- tions have to be made about the nature of the interface be- tween those parts of the model that are meant to be biologi- cally plausible and those parts that function as proxies for yet- to-be-developed biologically plausible parts. Specifically, the sensory modules must be able perform their function (e.g., recognizing words or objects) at least as well as humans, or else other components of the model have to account for er- rors in perceptual processing (e.g., word recognition errors). mscheutz@cs.tufts.edu of Computer Science, Tufts University 2 Department Moreover, the sensory module must be able to perform its task and make its result available at least as fast as the corre- sponding module would in humans to be able to respect hu- man timing (e.g., the human expectation to hear a verbal ac- knowledgment at the right time in response to an utterance). As a result, these two requirements often pre-empt the use of traditional computational methods that perform sensory- input-to-token conversion. In this paper, we will address the problem of biologically plausible real-time sensory processing of speech signals with a two-fold goal: (1) to provide a biologically plausible neu- ral model of human spoken phrase recognition, and (2) to provide a sensory module that can be embedded in classi- cal cognitive architectures for the study of embodied situated models of natural language interactions. Specifically, we pro- pose a new approach to robust speech recognition which gives continuous access to meaningful partial results, and returns word or phrase tokens that can be directly used by higher- level cognitive models. The model includes several parts of the early auditory processing system in humans: the cochlea (converting time to frequency domain), parts of the olivary complex (applying several filters to the cochlear-processed signal), and a sensory cortical area (comprising a recurrent spiking neural circuit to integrate the signal). Category sepa- ration is performed by “readout neurons” (one per category) that respond continuously to ongoing activity in the recur- rent circuit based on the particular weighted projection they receive, as is customary for the employed liquid-state ma- chine (LSM) neural model (Maass, Natschlager, & Markram, 2002). The weighted projection received by each readout neuron is determined before-hand based on offline training on a speech corpus. The main contributions are thus the im- plementation of the speech processing neural architecture and the method of converting the instantaneous output of read- out neurons to the token-type output for use by subsequent cognitive processes. The paper starts by laying out the back- ground and the problems to be solved by a model that has to convert time-varying instantaneous neural-readout behav- ior to discrete categories. The subsequent model section then describes the neural speech-encoding and integration parts of the model. Then an analysis of the performance of the model on subsets of a speech-corpus from human-human interac- tion experiments is presented together with links to videos in which the model is used as part of a larger cognitive model for situated embodied human-robot interaction experiments where speech utterances are used to control the robot’s ongo- ing behavior.