Abstract:
An apparatus for testing user interface integrity of speech-enabled devices includes a processor and a storage medium coupled to the processor. A set of voiced utterances is stored in the storage medium. A software module is executed by the processor to determine a state of the voice recognizer and provide a response to the voice recognizer in accordance with the determined state. The response may be to produce at least one voiced utterance in accordance with the state. The apparatus may be acoustically coupled to the voice recognizer. The apparatus may also, or in the alternative, be electrically coupled by a cable to the voice recognizer. The set of voiced utterances may include multiple sets of voiced utterances, each set having been spoken by a different person. The set of voiced utterances may also, or in the alternative, include multiple sets of voiced utterances, each set of voiced utterances having been spoken under different background noise conditions. The software module may also be executable to monitor the performance of the voice recognizer.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    I. Field of the Invention  
           [0002]    The present invention pertains generally to the field of communications, and more specifically to testing user interface integrity of speech-enabled devices.  
           [0003]    II. Background  
           [0004]    Voice recognition (VR) represents one of the most important techniques to endow a machine with simulated intelligence to recognize user or user-voiced commands and to facilitate human interface with the machine. VR also represents a key technique for human speech understanding. Systems that employ techniques to recover a linguistic message from an acoustic speech signal are called voice recognizers. The term “voice recognizer” is used herein to mean generally any spoken-user-interface-enabled device. A voice recognizer typically comprises an acoustic processor, which extracts a sequence of information-bearing features, or vectors, necessary to achieve VR of the incoming raw speech, and a word decoder, which decodes the sequence of features, or vectors, to yield a meaningful and desired output format such as a sequence of linguistic words corresponding to the input utterance. To increase the performance of a given system, training is required to equip the system with valid parameters. In other words, the system needs to learn before it can function optimally.  
           [0005]    The acoustic processor represents a front-end speech analysis subsystem in a voice recognizer. In response to an input speech signal, the acoustic processor provides an appropriate representation to characterize the time-varying speech signal. The acoustic processor should discard irrelevant information such as background noise, channel distortion, speaker characteristics, and manner of speaking. Efficient acoustic processing furnishes voice recognizers with enhanced acoustic discrimination power. To this end, a useful characteristic to be analyzed is the short time spectral envelope. Two commonly used spectral analysis techniques for characterizing the short time spectral envelope are linear predictive coding (LPC) and filter-bank-based spectral modeling. Exemplary LPC techniques are described in U.S. Pat. No. 5,414,796, which is assigned to the assignee of the present invention and fully incorporated herein by reference, and L. B. Rabiner &amp; R. W. Schafer,  Digital Processing of Speech Signals  396-453 (1978), which is also fully incorporated herein by reference.  
           [0006]    The use of VR (also commonly referred to as speech recognition) is becoming increasingly important for safety reasons. For example, VR may be used to replace the manual task of pushing buttons on a wireless telephone keypad. This is especially important when a user is initiating a telephone call while driving a car. When using a phone without VR, the driver must remove one hand from the steering wheel and look at the phone keypad while pushing the buttons to dial the call. These acts increase the likelihood of a car accident. A speech-enabled phone (i.e., a phone designed for speech recognition) would allow the driver to place telephone calls while continuously watching the road. And a hands-free car-kit system would additionally permit the driver to maintain both hands on the steering wheel during call initiation.  
           [0007]    Speech recognition devices are classified as either speaker-dependent or speaker-independent devices. Speaker-independent devices are capable of accepting voice commands from any user. Speaker-dependent devices, which are more common, are trained to recognize commands from particular users. A speaker-dependent VR device typically operates in two phases, a training phase and a recognition phase. In the training phase, the VR system prompts the user to speak each of the words in the system&#39;s vocabulary once or twice so the system can learn the characteristics of the user&#39;s speech for these particular words or phrases. Alternatively, for a phonetic VR device, training is accomplished by reading one or more brief articles specifically scripted to cover all of the phonemes in the language. An exemplary vocabulary for a hands-free car kit might include the digits on the keypad; the keywords “call,” “send,” “dial,” “cancel,” “clear,” “add,” “delete,” “history,” “program,” “yes,” and “no”; and the names of a predefined number of commonly called coworkers, friends, or family members. Once training is complete, the user can initiate calls in the recognition phase by speaking the trained keywords. For example, if the name “John” were one of the trained names, the user could initiate a call to John by saying the phrase “Call John.” The VR system would recognize the words “Call” and “John,” and would dial the number that the user had previously entered as John&#39;s telephone number.  
           [0008]    Speech-enabled products must be tested by hundreds of users, many times during the product development cycle and during the product validation phase, in order to test the integrity of the user interface and the application logic. A statistically significant, repeatable test of such magnitude is prohibitively expensive for the manufacturer to undertake. For this reason, many VR products undergo limited testing in the lab and extensive testing in the marketplace—i.e., by consumers. It would be desirable for manufacturers to provide consumers with fully tested VR products. Thus, there is a need for a low-cost, repeatable, non-intrusive testing paradigm for testing and improving speech-enabled products and speech-enabled services.  
         SUMMARY OF THE INVENTION  
         [0009]    The present invention is directed to a low-cost, repeatable, non-intrusive testing paradigm for testing and improving speech-enabled products and speech-enabled services. Accordingly, in one aspect of the invention, a device for testing and training a voice recognizer advantageously includes a processor; a storage medium coupled to the processor and storing a plurality of voiced utterances; and a software module executable by the processor to determine a state of the voice recognizer and provide a response in accordance with the state.  
           [0010]    In another aspect of the invention, a method of testing and training a voice recognizer advantageously includes the steps of storing a plurality of voiced utterances; determining a state of the voice recognizer; and providing a response to the voice recognizer in accordance with the state.  
           [0011]    In another aspect of the invention, a device for testing and training a voice recognizer advantageously includes means for storing a plurality of voiced utterances; means for determining a state of the voice recognizer; and means for providing a response to the voice recognizer in accordance with the state. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0012]    [0012]FIG. 1 is a block diagram of a conventional voice recognition system.  
         [0013]    [0013]FIG. 2 is a block diagram of a testing system for voice recognition systems such as the system of FIG. 1.  
         [0014]    [0014]FIG. 3 is a flow chart illustrating method steps performed by a voice recognition system when the testing system of FIG. 2 saves a voice entry into the voice recognition system.  
         [0015]    [0015]FIG. 4 is a flow chart illustrating method steps performed by a voice recognition system when the testing system of FIG. 2 dials a voice entry in the voice recognition system.  
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0016]    As illustrated in FIG. 1, a conventional voice recognition system  10  includes an analog-to-digital converter (A/D)  12 , an acoustic processor  14 , a VR template database  16 , pattern comparison logic  18 , and decision logic  20 . The VR system  10  may reside in, e.g., a wireless telephone or a hands-free car kit.  
         [0017]    When the VR system  10  is in speech recognition phase, a person (not shown) speaks a word or phrase, generating a speech signal. The speech signal is converted to an electrical speech signal s(t) with a conventional transducer (also not shown). The speech signal s(t) is provided to the A/D  12 , which converts the speech signal s(t) to digitized speech samples s(n) in accordance with a known sampling method such as, e.g., pulse coded modulation (PCM).  
         [0018]    The speech samples s(n) are provided to the acoustic processor  14  for parameter determination. The acoustic processor  14  produces a set of parameters that models the characteristics of the input speech signal s(t). The parameters may be determined in accordance with any of a number of known speech parameter determination techniques including, e.g., speech coder encoding and using fast fourier transform (FFT)-based cepstrum coefficients, as described in the aforementioned U.S. Pat. No. 5,414,796. The acoustic processor  14  may be implemented as a digital signal processor (DSP). The DSP may include a speech coder. Alternatively, the acoustic processor  14  may be implemented as a speech coder.  
         [0019]    Parameter determination is also performed during training of the VR system  10 , wherein a set of templates for all of the vocabulary words of the VR system  10  is routed to the VR template database  16  for permanent storage therein. The VR template database  16  is advantageously implemented as any conventional form of nonvolatile storage medium, such as, e.g., flash memory. This allows the templates to remain in the VR template database  16  when the power to the VR system  10  is turned off.  
         [0020]    The set of parameters is provided to the pattern comparison logic  18 . The pattern comparison logic  18  advantageously detects the starting and ending points of an utterance, computes dynamic acoustic features (such as, e.g., time derivatives, second time derivatives, etc.), compresses the acoustic features by selecting relevant frames, and quantizes the static and dynamic acoustic features. Various known methods of endpoint detection, dynamic acoustic feature derivation, pattern compression, and pattern quantization are described in, e.g., Lawrence Rabiner &amp; Biing-Hwang Juang,  Fundamentals of Speech Recognition  (1993), which is fully incorporated herein by reference. The pattern comparison logic  18  compares the set of parameters to all of the templates stored in the VR template database  16 . The comparison results, or distances, between the set of parameters and all of the templates stored in the VR template database  16  are provided to the decision logic  20 . The decision logic  20  selects from the VR template database  16  the template that most closely matches the set of parameters. In the alternative, the decision logic  20  may use a conventional “N-best” selection algorithm, which chooses the N closest matches within a predefined matching threshold. The person is then queried as to which choice was intended. The output of the decision logic  20  is the decision as to which word in the vocabulary was spoken.  
         [0021]    The pattern comparison logic  18  and the decision logic  20  may advantageously be implemented as a microprocessor. The VR system  10  may be, e.g., an application specific integrated circuit (ASIC). The recognition accuracy of the VR system  10  is a measure of how well the VR system  10  correctly recognizes spoken words or phrases in the vocabulary. For example, a recognition accuracy of 95% indicates that the VR system  10  correctly recognizes words in the vocabulary ninety-five times out of 100.  
         [0022]    In accordance with one embodiment, as shown in FIG. 2, a testing system  100  for VR products includes a processor  102 , a software module  104 , and a storage medium  106 . The processor  102  is advantageously a microprocessor, but may be any conventional form of processor, controller, or state machine. The processor  102  is coupled to the software module  104 , which is advantageously implemented as RAM memory holding software instructions. The RAM memory  104  may be on-board RAM, or the processor  102  and the RAM memory  104  could reside in an ASIC. In an alternate embodiment, firmware instructions are substituted for the software module  104 . The storage medium  106  is coupled to the processor  102 , and is advantageously implemented as a disk memory that is accessible by the processor  102 . In the alternative, the storage medium  106  could be implemented as any form of conventional nonvolatile memory. Input and output connections allow the processor to communicate with a VR device (not shown) to be tested. The input and output connections advantageously comprise a cable that electrically couples the testing system  100  with the VR device. In addition to a cable, the input and output connections may include a digital-to-analog converter (D/A) (not shown) and a loudspeaker (also not shown), allowing the testing system  100  to communicate audibly with the VR device.  
         [0023]    The testing system  100  simulates hundreds of speakers using a VR device, thereby providing an end-to-end, repeatable, non-intrusive test for VR devices. The storage medium  106  contains digital samples of a set of utterances, each utterance having been repeated by many different speakers. In one embodiment  150  words are spoken by each speaker, and 600 speakers are recorded, yielding 90,000 digital samples that are stored in the storage medium  106 . The software instructions held in the software module  104  are executed by the processor  102  to anticipate the state of the VR device (which is received at the input connection) and provide an appropriate response via the output connection. The software instructions may advantageously be written in a scripting language. The cable from the output connection may advantageously interface with the VR device through a normal serial port, or diagnostic monitor port, of the VR device, and/or through a PCM port of the VR device. In one embodiment, in which the VR device is a wireless telephone, the serial port is used to command the VR device to emulate pressing buttons on a keypad of the telephone and to retrieve characters displayed on the LCD display of the telephone. In another embodiment, in which the VR device is a hands-free car kit (and an associated phone), the PCM port of the car kit is used to input speech to the car kit and to receive voice prompts and voice responses from the car kit. In another embodiment, the speech may be provided audibly to the VR device by means of a D/A and a loudspeaker. Hence, the testing system  100  appears to the VR device to be a human user, generating results in real time. Moreover, the software module  104  includes instructions to monitor the recognition accuracy of the VR device and report the recognition accuracy to the user.  
         [0024]    In one embodiment the user interface integrity of a VR device may be tested according to the method steps depicted in the flow chart of FIG. 3. Those skilled in the art would appreciate that the algorithm steps shown in FIG. 3, which are performed by a testing system (not shown), are tailored to a particular VR user interface being assumed. Other and different VR user interfaces could yield different algorithm steps. In accordance with the embodiment of FIG. 3, a voice entry is saved in a VR device (not shown) by a testing system that appears to the VR device to be a human user.  
         [0025]    In step  200  the prompt “Add a Voice Tag?” is generated on the LCD screen of a VR device. This feature, which often is found in VR devices, allows a user to add a voice tag to a previously entered numeric telephone number, so that by saying the name corresponding to that number, the user can initiate dialing. The testing system receives the prompt and selects either “OK” to add the voice tag or “Next” to add another voice tag, through a cable electrically coupling the testing system to the diagnostic, or serial, port of the VR device.  
         [0026]    In step  202  the command “Place Phone to Ear and Follow Instructions” appears on the LCD screen of the VR device and is received by the testing system. In step  204  the testing system waits two seconds, simulating the response time of a human user. In step  206  the command “Please Speak a Name” appears on the LCD screen of the VR device and is received by the testing system. In step  208  the VR device audibly generates the words “Name Please,” followed by a beep.  
         [0027]    In step  210  the testing system audibly generates a name taken from a stored database of names, and the VR device “captures” the utterance. The VR device may fail to capture the utterance, i.e., an error condition may occur. Error conditions include, e.g., more than two seconds elapsing before a name is spoken, the name spoken being too short, e.g., less than 280 msec in duration, or the name spoken being too long, e.g., greater than two seconds in duration. If the VR device fails to capture the utterance, the VR device repeats the prompt of step  208 . If a predefined number of failures, N, occurs in succession, the VR devices aborts, returning to step  206 .  
         [0028]    If the VR device captures the utterance given in step  210 , the VR device audibly generates the captured utterance in step  212 . In step  214  the VR command “Again, Please” appears on the LCD screen of the VR device and is received by the testing system. In step  216  the VR device audibly generates the word “Again,” followed by a beep.  
         [0029]    In step  218  the testing system audibly repeats the name. If the VR device fails to capture the utterance, i.e., if an error condition occurs, the VR device repeats the prompt of step  216 . If a predefined number of failures, N, occurs in succession, the VR devices aborts, returning to step  206 .  
         [0030]    If the VR device captures the utterance given in step  218 , the testing system compares, or “matches,” the two utterances captured in steps  210  and  218 . If the two responses do not match, the second response is rejected and the VR device repeats the prompt of step  216 . If a predefined number of failures, M, to match the two utterances occurs, the VR devices aborts, returning to step  206 . The testing system records the number of failures in order to provide a user with an accuracy measure of the VR device.  
         [0031]    If a successful match occurs, the VR devices audibly repeats the second captured utterance in step  222 . In step  224  the words “Voice Tag Saved Successfully” appear on the LCD screen of the VR device and are received through the cable by the testing system. In step  226  the LCD screen of the VR device indicates that the number was stored in a particular memory location. In step  228  the LCD screen of the VR device indicates the number of memory locations used and the number of available memory locations. The VR device then exits VR mode.  
         [0032]    In one embodiment the user interface integrity of a VR device may be tested according to the method steps depicted in the flow chart of FIG. 4. Those skilled in the art would appreciate that the algorithm steps shown in FIG. 4, which are performed by a testing system (not shown), are tailored to a particular VR user interface being assumed. Other and different VR user interfaces could yield different algorithm steps. In accordance with the embodiment of FIG. 4, a voice entry is dialed in a VR device (not shown) by a testing system that appears to the VR device to be a human user.  
         [0033]    In step  300  the testing system sends a command through a cable electrically coupling the testing system to the diagnostic, or serial, port of the VR device. The command simulates a human user pressing a SEND button on the VR device. In step  302  the VR device emits two audible beeps in succession. In step  304  the words “About to Start VR” and “Send=Redial” appear on the LCD screen of the VR device and are received by the testing system through the cable. The testing system has the option of selecting either “Redial” to redial a call or “VR” to enter VR mode, through the cable. The SEND key is used to initiate VR mode, which happens if the user does not perform any action for two seconds after pressing SEND. However, the user has the option of redialing the previously called number by pressing SEND again within two seconds of pressing it the first time. The VR device is indicating that VR mode is able to be started, but that the user can instead redial if he or she hits SEND again. In step  306  the testing system waits two seconds, simulating the response time of a human user.  
         [0034]    In step  308  the testing system has selected “VR” through the cable and the VR device enters VR mode. The command “Please Speak Voice Tag” is generated on the LCD screen of the VR device and received by the testing system through the cable. In step  310  the VR device audibly generates the words “Name Please,” followed by a beep.  
         [0035]    In step  312  the testing system audibly generates a name taken from a stored database of names, and the VR device “captures” the utterance. The VR device may fail to capture the utterance, i.e., an error condition may occur. Error conditions include, e.g., more than two seconds elapsing before a name is spoken, the name spoken being too short, e.g., less than 280 msec in duration, or the name spoken being too long, e.g., greater than two seconds in duration. If the VR device fails to capture the utterance, the VR device repeats the prompt of step  310 . If a predefined number of failures, N, occurs in succession, the VR devices aborts, returning to step  308 .  
         [0036]    In step  314  the VR device compares, or “matches,” the captured utterance with every name on the list of names stored in the vocabulary of the VR device. If no match is found, the VR device repeats the prompt of step  310 . If a predefined number of failures, M, to find a match occurs, the VR devices aborts, returning to step  308 . The testing system records the number of failures in order to provide a user with an accuracy measure of the VR device.  
         [0037]    If more than one match is found in step  314 , the VR device proceeds to step  316 , employing an n_best algorithm to resolve the match, as known in the art. With the n_best algorithm, the VR device allows the testing system to choose between a predefined number n, which is advantageously two, of matches selected from the vocabulary of names in the VR device. For example, the VR device audibly asks the testing system whether the testing system “said” the voice corresponding to the best match. The VR device also generates the same question on its LCD screen, along with the choices of selecting either YES or NO. The testing system receives this information through the cable and selects either YES or NO through the cable. If the testing system selects NO, the VR device repeats the questions, referencing the next-closest match. The process is continued until a match is chosen by the testing system, or until no match is chosen and the list of matches is exhausted, at which point the VR device would abort and repeat step  308 .  
         [0038]    After a successful match in either step  314  or step  316 , the VR device proceeds to step  318 . In step  318  the LCD screen of the VR device indicates that the VR device is calling the stored telephone number associated with the name. This indication is received by the testing system through the cable. In step  320  the VR device audibly indicates that it is calling the selected name.  
         [0039]    In step  322  the VR device captures any utterance made by the testing system, which is typically silence. The testing system might also audibly generate the word “Yes” via a loudspeaker coupled to the testing system. Or the testing system could generate the word “No.” If the VR device captures nothing, the call is made (i.e., silence is assumed). If the VR device captures an utterance that matches successfully with the word “Yes,” which is stored in the vocabulary database of the VR device, the call is made. If, on the other hand, an error condition occurs, such as a too-long utterance or a too-short utterance being captured, the VR device questions whether the testing system wants the call to be made. If the VR device captures an utterance that matches successfully with a word other than “Yes,” the VR device questions whether the testing system wants the call to be made. If the Testing system responds affirmatively, the call is made. If the testing system responds negatively, the VR device aborts, returning to step  308 . The testing system could respond through the cable. In the alternative, or in addition, the testing system could respond audibly through the loudspeaker, in which case the response would have to be captured and matched in similar fashion to the methods described above.  
         [0040]    In the embodiments described with reference to FIGS.  3 - 4 , commands are sent from the testing system to the VR device through a cable electrically coupling the testing system to the diagnostic, or serial, port of the VR device. The commands are sent by the testing system. In another embodiment, a computer monitor may be coupled to testing system to display a graphical rendition of the user interface of the VR device, including the current display shown on the LCD screen of the VR device. Simulated buttons are provided on the monitor screen on which the user may mouse-click to send key-press commands to the VR device to simulate a user physically pressing the same buttons. Using the monitor, the user can control the VR device without actually touching it.  
         [0041]    Thus, a novel and improved method and apparatus for testing user interface integrity of speech-enabled devices has been described. Those skilled in the art would understand that many other aspects of a VR user interface, such as, e.g., a voice memo feature, could be tested with the testing system described above. Those of skill in the art would understand that the various illustrative logical blocks and algorithm steps described in connection with the embodiments disclosed herein may be implemented or performed with a digital signal processor (DSP), an application specific integrated circuit (ASIC), discrete gate or transistor logic, discrete hardware components such as, e.g., registers and FIFO, a processor executing a set of firmware instructions, or any conventional programmable software module and a processor. The processor may advantageously be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The software module could reside in RAM memory, flash memory, registers, or any other form of writable storage medium known in the art. Those of skill would further appreciate that the data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description are advantageously represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.  
         [0042]    Preferred embodiments of the present invention have thus been shown and described. It would be apparent to one of ordinary skill in the art, however, that numerous alterations may be made to the embodiments herein disclosed without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited except in accordance with the following claims.