Abstract:
A method and system for teaching an object of a deictic reference to a machine. A processor of the machine teaches the object of the deictic reference to the machine which results in the machine learning the object. The teaching includes: the processor finds an item in a region indicated by a physical pointing gesture, by the user, that points to the object; the processor shines a laser light on the item and in response, the processor receives a negative spoken indication from the user that the item shined on by the laser light is not the object; in response to the negative spoken indication from the user, the processor interacts with the user in an iterative procedure wherein the machine learns the object in a final iteration of the procedure. The processor stores the learned object in a storage repository.

Description:
[0001]    This application is a continuation application claiming priority to Ser. No. 12/357,353, filed Jan. 21, 2009. 
     
    
     FIELD OF THE INVENTION 
       [0002]    The present invention relates to machines and more specifically to a user-guided technique for teaching deictic references and objects of deictic references to a conversational command and control system installed in a vehicle or a machine for automating the execution of deictic references for performing action and/or tasks with respect to the referent objects that have been added to the conversational command and control system by the conversational system of the vehicle or machine. 
       BACKGROUND OF THE INVENTION 
       [0003]    It is widely held that the physical act of pointing, particularly with the index finger, is a uniquely human action and one of the foundation elements of human language. Computing systems provide the means by which expertise can be built into tools, such as, vehicles and unspecialized tools, such as robots. Efforts to build computing systems that support human modes of communication, such as, spoken dialog have not addressed this human communication modality since the domain of human computer interaction has not required this modality. As such, there is a need to provide a computing system that supports such human modes of communication. 
       SUMMARY OF THE INVENTION 
       [0004]    The present invention resides in a machine, system and method for user-guided teaching of a deictic reference and an object of a deictic reference to a conversational command and control system installed in a vehicle or machine, in accordance with an embodiment of the invention. In an embodiment, the deictic reference tool further includes a verification module for confirming understanding of the one or more objects of the deictic reference learned from the user in order to detect and correct any errors pertaining to the one or more objects of the deictic reference learned, wherein the confirming understanding of the one or more objects of the deictic reference learned includes one or more of: displaying the object of the deictic reference utilizing a graphical user interface for permitting correction by spoken or graphical or deictic interactions, utilizing a laser scanner subsystem to outline or paint the object of the deictic reference and utilizing a computer-driven manipulator to point the object of the deictic reference. In an embodiment, the deictic reference module observes the physical pointing gesture via one or more image capturing devices coupled to the computer system and decodes the one or more objects of the deictic reference utilizing the conversational system. The deictic reference tool further includes a decoder module for decoding the physical pointing gesture to locate a reference region and an object recognition module for determining whether or not a object is a correct object that the physical pointing gesture points to. In an embodiment, the conversational system further includes a speech recognition module for decoding any spoken utterances received from a user using a respective spoken user interface of the one or more respective machine subsystems coupled to the data and control bus, a gesture recognition module for decoding indications received via a respective physical user interface of the one or more respective machine subsystems coupled to the data and control bus and a haptic module for processing haptic interactions received via a respective physical user interface of the one or more respective machine subsystems coupled to the data and control bus. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0005]    The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention: 
           [0006]      FIG. 1  is a schematic block diagram illustrating one embodiment of a computer infrastructure of a machine, in particular, a vehicle coupled to a computer system having installed thereon a conversational command and control system that includes a deictic reference tool or program for user-guided teaching of deictic references and referent objects of deictic references to the command and control system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. 
           [0007]      FIG. 2  is a schematic block diagram illustrating one embodiment of a computer infrastructure in a house or office that includes a conversational command and control system having a deictic reference tool or program for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. 
           [0008]      FIG. 3  is a schematic block diagram illustrating one embodiment of a computer infrastructure in a machine, such as, a tool or a robot, that includes a conversational command and control system having a deictic reference tool or program for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. 
           [0009]      FIG. 4  depicts an embodiment of a computer system in a vehicle or a machine having deployed thereon a conversational command and control system having a deictic reference tool or program for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. 
           [0010]      FIG. 5  depicts a flowchart outlining the overall steps performed by a deictic reference tool or program of a conversational command and control system installed in a vehicle or machine for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. 
           [0011]      FIG. 6  depicts a flowchart outlining the overall steps performed by a deictic reference tool or program of a conversational command and control system installed in a vehicle or machine for user-guided correction of deictic references and referent objects of deictic references taught to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. 
           [0012]      FIG. 7  depicts a flowchart outlining the detailed steps performed by a deictic reference tool or program of a conversational command and control system in a machine or vehicle for user-guided teaching of a referent object of a deictic reference received from a user, in accordance with an embodiment of the present invention. 
           [0013]      FIG. 8  depicts a flowchart outlining the detailed steps performed by a deictic reference tool or program of a conversational system in a machine or vehicle for executing actions and/or tasks on a referent object of a deictic reference learned, in accordance with an embodiment of the present invention. 
           [0014]      FIG. 9  depicts an embodiment of a teaching or learning dialog between a human operator or user and a conversational system in a machine that contains a deictic reference tool or program for user-guided teaching of a referent object of a deictic reference received from a user, in accordance with an embodiment of the present invention. 
           [0015]      FIG. 10  depicts an embodiment of an execution dialog between a human operator or user and a conversational command and control system in a machine that contains a deictic reference tool or program for execution of a deictic reference learned by the conversational system for performing or executing a sequence of actions and/or tasks on a referent object of the deictic reference, in accordance with an embodiment of the present invention. 
           [0016]      FIG. 11  depicts an embodiment of an execution dialog between a human operator or user and a conversational command and control system in a machine that contains a deictic reference tool or program for execution of a deictic reference learned by the conversational system for performing or executing a sequence of actions and/or tasks on a referent object of the deictic reference and for correcting the referent object learned, in accordance with an embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0017]    Many of the functional units described in this specification have been labeled as modules, in order to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices or the like. 
         [0018]    Modules may also be implemented in software for execution by various types of processors. An identified module or component of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions which may, for instance, be organized as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different locations which, when joined logically together, comprise the module and achieve the stated purpose for the module. 
         [0019]    Further, a module of executable code could be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices, over disparate memory devices, and may exist, at least partially, merely as electronic signals on a system or network. 
         [0020]    Furthermore, modules may also be implemented as a combination of software and one or more hardware devices. For instance, a module may be embodied in the combination of a software executable code stored on a memory device. In a further example, a module may be the combination of a processor that operates on a set of operational data. Still further, a module may be implemented in the combination of an electronic signal communicated via transmission circuitry. 
         [0021]    Reference throughout this specification to “one embodiment,” “an embodiment,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment. 
         [0022]    Moreover, the described features, structures, or characteristics of the invention may be combined in any suitable manner in one or more embodiments. It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit and scope of the invention. Thus, it is intended that the present invention cover the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents. Reference will now be made in detail to the preferred embodiments of the invention. 
         [0023]    In one embodiment, the invention provides a vehicle or machine or a home automation computer system that has deployed thereon a conversational command and control system having a deictic reference tool or program for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. Reference is now made to  FIG. 1 , reference numeral  100 , which schematically illustrates an embodiment of a machine, namely, a vehicle  102  having a computing system that includes a conversational system  105  (also referred to as a “conversational command and control system” or “spoken language command and control system” or simply as “command and control system”) having a deictic reference tool or program  107  for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects that have been added to the command and control system by the conversational system, in accordance with an embodiment of the present invention. It will be obvious to one of ordinary skill in the art that a system for user-guided teaching of deictic references may be combined with a conversational system that provides a spoken language command and control system, regardless of whether or not the spoken language command and control system includes user-guided teaching of command vocabulary and action sequences. The deictic reference tool provides the means by which the referent objects to be acted upon may be specified, while the spoken language command and control system provides the means by which the actions and/or tasks to be carried out on a referent object may be specified. In an embodiment, as shown in  FIG. 1 , the vehicle or car is a land vehicle having wheels. In one embodiment, the conversational system  105  is a spoken language command and control system that uses a commercially available speech recognition program or software, such as, the IBM® Embedded Speech Recognition software that is commercially available from International Business Machines Corporation. In an embodiment, the conversational system  105  includes a deictic reference tool or program  107  for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system. The conversational system  105 , which includes the deictic reference tool  107 , comprises a logic unit that contains a plurality of modules configured to functionally execute the necessary steps of teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system  105 , as discussed further herein below with respect to  FIG. 2 . Further, as shown in  FIG. 1 , the vehicle computing system  100  having the conversational system  105  includes a vehicle data and control bus  180 . In an embodiment, the vehicle data and control bus comprises a Controller Area Network (CAN) bus. However, the vehicle data and control bus may alternatively comprise any of the following: MOST (Media Oriented Systems Transport), SAE J1850 interface, MI (Motorola Interconnect) bus, DSI (Distributed Systems Interface) bus, BST (Bosch-Siemens-Temic) bus and Boeing Intellibus. In an embodiment, the vehicle data and control bus  180  is configured to interface with one or more devices or components within the vehicle  102 , such as, the climate control system  120 , the entertainment system  130  and/or the light control system  140  within the vehicle  102 . Further, the vehicle data and control bus  180  is coupled to a user interface adapter  150 , a reader or scanner device  160  and a display adapter  170 , as explained further herein below. In particular, a user may operate or control the various devices or components within the vehicle  102  by interfacing with one or more devices coupled to the user interface adapter  150  coupled to the vehicle data and control bus  180 . Further, in an embodiment as shown in  FIG. 1 , the computing system  100  includes a computer system bus  190  that is coupled to the various components of the computing system  100  in the vehicle  102  and is further coupled to the vehicle data and control bus  180  via an intermediate bridge module  192 , which in an embodiment supports two way communication with buffering, signal level and code translation support. In an embodiment, the system bus  190  connects a central processing unit (CPU)  106 , a flash memory  104 , a main memory comprised of RAM (Random Access Memory)  108 , ROM (Read-Only Memory)  110 , an input/output (I/O) adapter  112  and a communications adapter  118 , which can interconnect computer bus  190  with an external network  194  through a suitable connection. The communications adapter  118  is configured, in one embodiment, to facilitate network communications of the system  100  over a communications channel of a network  194  that the conversational system is connected to, such that, the system  100  can communicate with other systems. Further, in an embodiment, the input/output (I/O) adapter  112  communicates with devices, such as, the hard disk  114  and/or a computer readable storage medium or media  116 , such as, a magnetic tape or disk, optical media, DVD, memory stick, semiconductor memory, etc., which contains machine readable instructions that can be loaded into the hard disk  114  containing the conversational system  105  with a deictic reference tool  107  for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system  105 . Further, in an embodiment, the vehicle computer system  100  can download machine readable instructions stored on a server coupled to the vehicle computer system  100  via the network  194 . Furthermore, the communications adapter  118  is configured to permit communication between the various components or modules of the vehicle conversational system  100 . Turning to the vehicle data and control bus  180 , in an embodiment, the vehicle data and control bus  180  within system  100  is coupled to the climate control system  120 , the entertainment system  130 , the light control system  140 , the user interface adapter  150 , a reader or scanner  160  and a display adapter  170 . In an embodiment, the user interface adapter  150  is coupled to one or more devices, such as, a video camera  151 , a microphone  152 , a joystick  154 , an audio amplifier  156  and a touch screen  158 . The user interface  150 , in one embodiment, is configured to enable a user or car operator to interact with the computer system  100  for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, for instance, using the video camera  151  to observe the referent objects of the deictic references being taught to the conversational system using the deictic reference tool or program  107 . In an embodiment, the climate control system  120  that is coupled to the vehicle data and control bus  180  controls devices, such as, the air conditioner  124 , the heater  126 , and the windows  122 . Further, in an embodiment, the entertainment system  130  that is coupled to the car bus  190  controls audio devices, such as, the MP3 (Moving Picture Experts Group (MPEG) Audio Layer 3) player  136 , the CD (Compact Disc) player  134  and the radio  132 . In addition, the light control system  140  that is coupled to the vehicle data and control bus  180  controls devices, such as, the interior lights  142  and the map light  144 . In particular, a user may use devices, such as, the video camera  151 , the speakerphone  152  or the touch screen  158  in conjunction with the conversational system  105  having the deictic reference tool  107  for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system. For instance, a user may teach the conversational system  105  a deictic reference by physically pointing to a referent object, which is detected by the video camera  151 , and by the spoken words, commands or utterances received by the spoken interface of the spoken language command and control system in the conversational system  105 . The conversational system learns the deictic reference by decoding the spoken voice command using the spoken language command and control system and learns the deictic reference by associating the deictic reference with a referent object of the deictic reference, as discussed further herein below with respect to  FIGS. 9-11 . It will be obvious to one of ordinary skill in the art that the forgoing description contains more functional elements and modules than are needed for a minimal implementation of this invention and that complement of functional elements may be reduced or enlarged as needed in specific applications without departure from the teaching of this invention. 
         [0024]    Further, reference is now made to  FIG. 2 , reference numeral  200 , which schematically illustrates an embodiment of a machine utilized in home or office automation that includes a computing system having a conversational system  105  that includes a deictic reference tool or program  107  for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects within the house or office, in accordance with an embodiment of the present invention. The computer system  200  is similar to that described herein in  FIG. 1 , except that instead of a vehicle data and control bus  180  in  FIG. 1 , the home computer system  200  includes at least one home or office data and control bus  182  that is configured to interface with one or more devices or components within the home  202 , such as, the climate control system  120 , the entertainment system  130  and/or the light control system  140  within the vehicle  102 . It will be obvious to one skilled in the art that multiple bus types may be employed by the home computer system  200  for interfacing with additional devices and/or components and/or subsystems within the home  202 . Further, the home data and control bus  182  is coupled to a user interface adapter  150 , a reader or scanner device  160  and a display adapter  170 , as explained further herein below. In particular, a user may operate or control the various devices or components within the home  202  by interfacing with one or more devices coupled to the user interface adapter  150  coupled to the home or office data and control bus  182 . Further, in an embodiment as shown in  FIG. 1 , the computing system  200  includes a computer system bus  190  that is coupled to the various components of the computing system  200  in the home or office  102  and is further coupled to the home or office data and control bus  182  via an intermediate bridge module  192 , which in an embodiment supports two way communication with buffering, signal level and code translation support. In an embodiment, the home or office data and control bus  182  within system  200  is coupled to the climate control system  120 , the entertainment system  130 , the light control system  140 , the user interface adapter  150 , a reader or scanner  160  and a display adapter  170 . In an embodiment, the user interface adapter  150  is coupled to one or more devices, such as, a video camera  151 , a microphone  152 , a joystick  154 , an audio amplifier  156  and a touch screen  158 . The user interface  150 , in one embodiment, is configured to enable a home owner or an office employer to interact with the computer system  200  for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to referent objects known to or learned by the conversational system, for instance, using the video camera  151  to observe the referent objects of the deictic references being taught to the conversational system using the deictic reference tool or program  107 . In an embodiment, the climate control system  120  that is coupled to the home or office data and control bus  182  controls devices, such as, the air conditioner  124 , the heater  126 , and the windows  122 . Further, in an embodiment, the entertainment system  130  that is coupled to the data and control bus  182  controls home or office audio devices, such as, the MP3 (Moving Picture Experts Group (MPEG) Audio Layer 3) player  136 , the CD (Compact Disc) player  134  and the radio  132 . In addition, the light control system  140  that is coupled to the data and control bus  182  controls devices, such as, the interior lights  142  and exterior lights  146 . In particular, a user may use devices, such as, the video camera  151 , the speakerphone  152  or the touch screen  158  in conjunction with the conversational system  105  having the deictic reference tool  107  for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system. For instance, a home owner may teach the conversational system  105  a deictic reference by physically pointing to a referent object, which is detected by the video camera  151 , and by the spoken words, commands or utterances received by the spoken interface of the spoken language command and control system in the conversational system  105 . Since the remainder of the devices connected to the computer system bus  190  in  FIG. 2  are the same as that described in  FIG. 1 , these devices will not be described herein with respect to  FIG. 2 . It will be obvious to one of ordinary skill in the art that the forgoing description contains more functional elements and modules than are needed for a minimal implementation of this invention and that complement of functional elements may be reduced or enlarged as needed in specific applications without departure from the teaching of this invention. 
         [0025]    Turning to  FIG. 3 ,  FIG. 3 , reference numeral  300 , which schematically illustrates an embodiment of a tool or machine  302 , such as, a robot that includes a computing system having a conversational system  105  that includes a deictic reference tool or program  107  for teaching deictic references and referent objects of deictic references to the conversational system for performing actions with respect to the referent objects known to or learned by the conversational system  105 . The computer system  300  includes a computer system bus  190  that is coupled to a user interface adapter  150 , a reader or scanner device  160 , and a display adapter  170 , as explained further herein below. In an embodiment, the system bus  190  connects a central processing unit (CPU)  106 , a flash memory  104 , a main memory comprised of RAM (Random Access Memory)  108 , ROM (Read-Only Memory)  110 , an input/output (I/O) adapter  112  and a communications adapter  118 , which can interconnect computer bus  190  with an external network  194  through a suitable connection. The communications adapter  118  is configured, in one embodiment, to facilitate network communications of the system  100  over a communications channel of a network  194  that the conversational system is connected to, such that, the system  100  can communicate with other systems. Further, in an embodiment, the input/output (I/O) adapter  112  communicates with devices, such as, the hard disk  114  and/or a computer readable storage medium or media  116 , such as, a magnetic tape or disk, optical media, DVD, memory stick, semiconductor memory, etc., which contains machine readable instructions that can be loaded into the hard disk  114 . In an embodiment, the program instruction modules of the conversational system  105  having a deictic reference tool  107  for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system  105  are stored on the hard disk or other non-volatile memory, in whole or in part, while the whole or parts are not in use, and are copied into working memory when the program instructions are in use. Further, in an embodiment, the computer system  300  can download machine readable instructions stored on a server coupled to the vehicle computer system  300  via the network  194 . Furthermore, the communications adapter  118  is configured to permit communication between the various components or modules of the computer system  300 . In an embodiment, the tool or machine  302  utilizes one or more devices coupled to the user interface adapter  150 , such as, a video camera  151  and/or a laser scanner device or subsystem  153  for learning deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system. Additionally, the user interface adapter  150  may be coupled to one or more devices, such as, a microphone  152 , a joystick  154 , an audio amplifier  156  and/or a touch screen  158 . The user interface  150 , in one embodiment, is configured to enable a user to interact with the computer system in the tool or machine  300  for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, for instance, using the video camera  151  to observe the referent objects of the deictic references being taught to the conversational system using the deictic reference tool or program  107 . For instance, to help in learning or understanding a deictic reference, the tool or machine  300  could use a prehensile manipulator for pointing or could use a tool operated through a conversational interface that is equipped with a low power visible laser scanner subsystem  153  that can draw or paint patterns and images, where an output subsystem would permit the conversation learning system to express its understanding of the extent of the referent object by using the laser scanner subsystem  153  to outline, draw and/or paint the object, utilizing the conversational system  105  to ask for confirmation. For example, a robotic surgeon, may indicate the line of a to-be-performed scalpel cut, using a laser to draw or paint or outline the deictic referent while asking for confirmation of its understanding. Alternatively, the robot&#39;s visual field can be displayed on a video screen or display with the assumed referent highlighted or outlined. The conversational system  105  learns the deictic reference by decoding the spoken voice command using the spoken language command and control system and the deictic reference tool  107  learns the deictic reference by associating the deictic reference with a referent object of the deictic reference, as discussed further herein below with respect to  FIGS. 9-11 . 
         [0026]    Reference is now made to  FIG. 4 , reference numeral  400 , which depicts an embodiment of a computer system in a vehicle or machine that includes a computer program product, namely, a conversational system  412  having a deictic reference tool or program  420  for user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. Further, in an embodiment, the conversational system  412  includes a command modification tool or program  416  for teaching and modifying voice commands and actions executed by the conversational system. In an embodiment, the conversational system  412  is installed within a local memory  410  of the computer system  400 , for instance, a computer system used in a speech enabled car or in home automation or in a robot, as described herein above with respect to  FIGS. 1-3 . Preferably, the computer system  400  includes a central processing unit (CPU)  404 , a local storage device  402 , a multi-modal user interface  406 , a network interface  408 , and a memory  410 . The CPU  404  is configured generally to execute operations within the computer system  400 . The multi-modal user interface  406 , in one embodiment, is configured to enable a user or operator to interact with the computer system  400 , including allowing input of voice commands (via a device connected to the user interface adapter  150  of the vehicle data and control bus  180  or the home data and control bus  182 , shown in  FIGS. 1 and 2 , respectively) and/or indicated data inputted (for instance, via a touch screen  158  or joystick  154  coupled to the user interface adapter  150 , shown in  FIGS. 1 and 2 , respectively) for teaching deictic references and referent objects of deictic references to the conversational system for performing actions with respect to the referent objects known to or learned by the conversational system  400 . The network interface  408  is configured, in one embodiment, to facilitate network communications of the computer system  400  over a communications channel of a network, such as the network  194 , shown in  FIGS. 1-3 . In an embodiment, the memory  410  is configured to store one or more applications or programs, such as, the conversational system  412 , which includes the deictic reference tool or program  420  and a command modification tool or program  416 . In an embodiment, the command modification tool  416  includes a speech recognition module, a speech enhancer module, a haptic module, a gesture recognition module, a dialog management module, a knowledge manager module, a command learning module, a repository module, an execution module, a verification module and a communication module. In an embodiment, the speech recognition module is configured to recognize spoken language or utterances and to semantically interpret incoming user requests and utterances. In particular, the speech recognition module converts speech signals into a sequence of words. The speech enhancer module is configured to remove noises and acoustic signals. In an embodiment, the haptic module is configured to receive haptic input via devices connected to the data and control bus in the vehicle or home system. The gesture recognition module is configured to decode indications made via a respective physical user interface of the conversational system, such as, a touch screen. The dialog management module is configured to semantically interpret incoming user requests and utterances and to manage interactions between the user and the conversational system in order to modify a pre-determined voice command set and/or actions of the conversational system. Further, the knowledge manager module is configured to control access to knowledge systems. The command learning module is configured to learn a set of commands to be performed upon the utterance of a voice command or upon receiving an indication or input made via a physical user interface. The repository module is configured to store the voice commands and/or actions learned or modified in a local storage  402 . The execution module is configured to execute performance of a set of commands, tasks and/or actions learned and/or modified upon the utterance of a voice command. The verification module is configured to record responses received from one or more respective systems in the vehicle or machine in order to detect any errors in future execution of the one or more action command sequences corresponding to the new voice command learned. The communication module is configured to permit communication between the various modules of the command modification tool, memory, local storage  402  and with external systems, as well as communication with the deictic reference tool or program  420 . Further, in one embodiment, as shown in  FIG. 4 , the deictic reference tool or program  420  which runs on the computer system  400  comprises of a logic unit that contains a plurality of modules configured to functionally execute the necessary steps of user-guided teaching of deictic references and objects of deictic references to the conversational system  412 , as explained further herein below. In an embodiment, shown in  FIG. 4 , the deictic reference tool or program  420  running on the vehicle or machine computer system  400  includes a decoder module  422 , an observation module  424 , a deictic reference module  426 , an image recognition module  428 , a verification module  430 , a repository module  432 , an execution module  434 , and a communication module  438 . In an embodiment, the decoder module  422  utilizes the conversational system  412  loaded in memory  410  to decode voice commands and/or utterances and to semantically interpret incoming user requests and utterances. The observation module  424  utilizes devices, such as, the video camera  151  to observe and decode physical acts of pointing in a deictic reference received from a user. The deictic reference module  426  associates referent objects of a deictic reference learned with the existing command vocabulary set of the conversational system to be able to distinguish referent objects for accomplishing tasks and/or actions with respect to referent objects taught to or learned by the conversational system, as requested by a user. The image recognition module  428  utilizes the image recognition tool  414  for recognizing referent objects in a deictic reference. In an embodiment, the image recognition tool  414  comprises recognition algorithms and recognition data. In an embodiment, the image recognition tool  414  processes raw data, such as, a collection of images of a particular object, into a form that can be used by the image recognition tool to determine whether or not a class of object is present in the system&#39;s visual field. The verification module  430  verifies understanding of the deictic references and the referent objects of the deictic references based on responses received from the user confirming or correcting understanding of the deictic references and the referent objects of the deictic references. The repository module  432  stores the referent objects  415  learned for the deictic references  413  in the local storage  402 . Further, the repository module  432  stores voice commands  416  and/or actions and/or tasks  417  learned or modified by the conversational system  412  in the local storage  402 . The execution module  434  is configured to execute performance of actions and/or tasks with respect to the referent objects of the deictic references learned by the conversational system  412 . The communication module  436  is configured to permit communication between the various modules of the deictic reference tool  420 , memory  410 , local storage  402  and with external systems connected to the computer system  400 . 
         [0027]    In another embodiment, the invention provides a method or process for teaching deictic references and referent objects of deictic references to the conversational system for performing actions with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. Reference is now made to  FIGS. 5 and 6 , which together illustrate an embodiment of a method or process for teaching deictic references and referent objects of deictic references to the conversational system for performing actions with respect to the referent objects known to or learned by the conversational system. In particular,  FIG. 5 , reference numeral  500 , outlines the overall learning process steps performed by a conversational system having a deictic reference tool installed thereon for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, whereas,  FIG. 6 , reference numeral  600 , outlines the overall steps performed by a conversational system for executing a deictic reference that has been learned upon receipt of the deictic reference from a user, in accordance with an embodiment of the present invention. Turning now to  FIG. 5 ,  FIG. 5  outlines the overall learning process steps performed by a conversational system having a deictic reference tool installed thereon for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system. As shown in  FIG. 5 , in step  502 , upon being prompted by a user (explained further herein below with respect to  FIGS. 7 and 8  and discussed herein below with examples shown in  FIGS. 9-11 ), the deictic reference tool provided in the conversational system begins or starts in step  502  the learning process for learning deictic references and referent objects of deictic references for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system. In an embodiment, in step  504 , the conversational system processes the deictic reference received for learning the referent objects of the deictic reference. Further, in step  506 , the conversational system verifies that the referent objects of the deictic reference is properly understood by the conversational system. If needed, the conversational system refines in step  508  the learning of the referent objects of the deictic reference learned. Finally, the conversational system updates data in step  510  pertaining to a deictic reference learned to be able to perform actions and/or tasks with respect to the referent objects of deictic references learned by the conversational system. In an embodiment, the present invention provides a process for deploying computing infrastructure, wherein the process comprises integrating computer-readable code into a computer system of a machine, and wherein the code in combination with the computer system is configured to perform a method for teaching an object of a deictic reference to a machine having a conversational system. 
         [0028]    Further, referring to  FIG. 6 , reference numeral  600 , depicts a flowchart outlining the overall steps performed by a conversational system having a deictic reference tool for correcting perception of a referent object of a deictic reference, in accordance with an embodiment of the present invention. As shown in  FIG. 6 , in step  602 , the conversational system starts correction of a referent object of a deictic reference uttered by a user. In step  604 , the conversational system verifies that the referent objects of the deictic reference is properly understood by the conversational system. As needed, the conversational system refines in step  606  the learning of the referent objects of the deictic reference learned. Finally, the conversational system updates data in step  608  pertaining to a deictic reference learned to be able to perform actions and/or tasks with respect to the referent objects of deictic references learned by the conversational system. 
         [0029]    Reference is now made to  FIG. 7 , reference numeral  700 , which outlines the steps performed by the deictic reference tool in the conversational system for teaching deictic references and referent objects of deictic references to the conversational system for performing actions and/or tasks with respect to the referent objects known to or learned by the conversational system, in accordance with an embodiment of the present invention. In step  702 , the conversational system having a deictic reference tool decodes the voice command issued or an utterance that is spoken by a user about to teach the conversational system a deictic reference input, as described in an example further herein below with respect to  FIG. 9 . Further, in step  704 , a determination is made by the deictic reference tool in the conversational system as to whether or not the voice command decoded is to learn a new object of a deictic reference or to execute a command with respect to an object of a deictic reference already learned. If the conversational system determines in step  704  that the voice command decoded is to execute a command that exists within the set of voice commands for acting on an object already learned by the conversational system, then the conversational system executes the command in step  706 , which is discussed herein below with respect to  FIG. 8 . However, if the conversational system determines that the voice command decoded is to learn a new object, then the conversational system acknowledges that it is ready by saying “Ready” or some other similar acknowledgement. In step  710 , the conversational system stores the object name in acoustic form and further determines in step  712  whether or not a pointing action is detected in the deictic reference received from a user. If a pointing action is not detected then the conversational system waits until a pointing action is detected in step  712 . However, if a pointing action is detected in step  712 , then in step  714 , the deictic reference tool processes the pointing action to find a reference region. The deictic reference tool processes the reference region image in step  716  to located a candidate object of the deictic reference. In step  718 , the deictic reference tool indicates the candidate object to the user for confirmation. The deictic reference tool determines whether or not the object has been confirmed by the user. If the object is not confirmed by the user, then the deictic reference tool returns to step  714  to process the pointing action to find the reference region of the pointing action. However, if in step  720 , the object is confirmed by the user, then the deictic reference tool optionally searches an image library stored on the computer system for additional examples of the object. Further, in step  724 , the deictic reference tool utilizes the original and any additional images to train image recognition of the object. The deictic reference tool adds in step  726  the object name to a vocabulary set in the conversational system. In step  728 , the deictic reference prompts the user that the learning process is complete and waits, in step  730  for the next user command. 
         [0030]    Reference is now made to  FIG. 8 , reference numeral  800 , which depicts a flowchart outlining the detailed steps performed by a conversational system having a deictic reference tool for performing or executing the command or action sequences associated with a learned deictic reference upon receiving a deictic reference, in accordance with an embodiment of the present invention. In step  802 , the deictic reference tool decodes the voice command received from a user. The deictic reference tool makes a determination in step  804  whether or not the voice command is to learn a new command. If the deictic reference tool determines that the voice command is to learn a new command, then the deictic reference tool begins the learning process at step  708 , as described in  FIG. 7 . However, if the deictic reference tool determines in step  804  that the voice command is not to learn a command, then in step  806 , the deictic reference tool determines whether or not the voice command uttered for the deictic reference is found in the list of commands maintained by the conversational system. If the conversational system determines that the voice command uttered is not found in the list, then the conversational system executes in step  808  a recognition error dialog, where the user is notified of the error, in particular, of the error of not finding or recognizing the command in the existing list of commands, ending the process of executing the voice command uttered for the deictic reference. However, if the voice command uttered is found in the list, then the deictic reference tool in the conversational system determines in step  810  whether or not the voice command contains a deictic reference. Further, in step  812 , the deictic reference tool dereferences the deictic reference to an object of the deictic reference. Furthermore, the conversational system constructs a plan by referencing the command list (based on the command uttered) and substitutes the specified object in step  814  into the plan in appropriate places. A planning module is then used to expand the command into a command list, which takes the specifics of the command and the circumstances into account. Such planning systems are well known within the robotics and autonomous vehicle communities and are outside the scope of this invention. The completed plan for the step containing the deictic reference is then executed in step  816 . In step  818 , the deictic reference tool determines whether or not the command executed is the last command in the list and, if so, the deictic reference tool in the conversational leaning system says “all done” in step  822 . However, if in step  818 , the deictic reference tool determines that the last command executed is not the last command in the list, then the message for the voice command is not sent, then the deictic reference tool in the conversational system increments in step  820  the command list element pointer to execute the next command in the list in step  816 . Again, after the deictic reference tool determines that the last command has been executed in the list in step  818 , the deictic reference tool in the conversational leaning system says “all done” in step  822 . 
         [0031]    Reference is now made to  FIGS. 9, 10 and 11 , reference numerals  900 ,  1000  and  1100 , respectively, which depict examples of various dialogs between a user and a machine, such as, a robot. Turning to  FIG. 9 ,  FIG. 9  shows an example of a training dialog that occurs between a user and a machine, such as, a robot having a computer system with a conversational system that includes a deictic reference tool or program for performing user-guided teaching of deictic references and referent objects of deictic references to the conversational system for performing actions with respect to the referent objects known to or learned by the conversational system. The training dialog  900  begins with the user (reference numeral  902 ) speaking to the robot named Tobor, saying “Tobor, learn a new object.” The robot, Tobor, responds (reference numeral  904 ), “I&#39;m ready to learn a new object.” The user (reference numeral  906 ) points in the direction of an apple on a nearby table and speaks or utters the words “That is an Apple.” Tobor responds (reference numeral  908 ) “Let me see if I&#39;ve got this straight”, where Tobor uses an image recognition software installed on Tobor&#39;s computer system to analyze the pointing gesture in order to determine the region indicated by the pointing act. In an embodiment, the computer system, engaged in deictic reference learning or decoding a command containing deictic reference searches its video field of view for a hand in a pointing posture (similar to a manner in which such systems find faces in video images) by detecting appropriate features. The video image processing may then take any object intersecting the plane defined by the camera location, the pointing finger tip and the average location of the wrist as a possible target of the pointing gesture provided they lay in the finger tip side of the plane. Such a video processing system is well understood in the field of autonomous vehicle path planning and is outside the scope of this invention. Returning to the example in  FIG. 9 , finding a table in the region of where the user is pointing, and not having a “table” reference in its vocabulary, Tobor uses a laser scanner to illuminate or outline the tabletop beginning a confirming deictic reference, “that is an Apple.” The user responds (reference numeral  910 ) with a “No”. Tobor, in return asks (reference numeral  912 ), “How about this? by illuminating a table leg. The user replies (reference numeral  914 ) with “Colder”, whereby Tobor illuminates the bowl of fruit on the table and asks (reference numeral  916 ) “How about this?” The user replies (reference numeral  918 ) “Warmer”. Tobor further illuminates an apple visible over the top of the bowl (reference numeral  920 ) and asks “Is this it?” to which the user replies “Yes” (reference numeral  922 ). To further understand the meaning of “Apple”, Tobor may (reference numeral  924 ) move to the table and pick up the apple to develop a larger database of apple images and may additionally measure the weight and surface friction that an apple presents. Furthermore, Tobor may use a network connection and/or a stored image database to collect additional images of apples in order to obtain more training data and the spelling of the word “apple”. Following image processing and storage of the new vocabulary word, Tobor completes the dialog by stating (reference numeral  926 ) “I think I understand the word ‘apple’.” 
         [0032]    Turning to  FIG. 10 ,  FIG. 10  represents a command and control dialog on a subsequent occasion with Tobor, where the user uses deictic reference. It is assumed here that Tobor&#39;s command vocabulary includes the phrase “Bring x to me”, where “x” is an object in Tobor&#39;s vocabulary. As such, the user says to Tobor (reference numeral  1002 ) “Open the refrigerator.” Tobor replies (reference numeral  1004 ) with “OK” and locates and opens the refrigerator. The user further speaks (reference numeral  1006 ) by pointing at the interior of the refrigerator, “Bring me that apple.” Tobor uses (reference numeral  1008 ) image processing to distinguish an apple in the referenced region, and picks up the apple and executes the “Bring x to me” command and speaks “Here you are.” The user replies (reference numeral  1010 ) “Thanks”. 
         [0033]    Turning to  FIG. 11 ,  FIG. 11  depicts a correction dialog on a subsequent occasion, where the user has a command and control dialog with Tobor and depicts a dialog in which Tobor is incorrect in its understanding of a deictic reference. In the example of  FIG. 11 , the user has arrived in the produce department of a supermarket with Tobor and the user points to a produce stand with apples, oranges and grapes and speaks (reference numeral  1102 ) “Bring me three of those apples.” Tobor (reference numeral  1104 ) applies its recognition software and is unable to distinguish the red apples from the oranges and brings three oranges from the produce stand. The user responds (reference numeral  1106 ) with “Wrong” As such, Tobor would go back to using the image recognition software to analyze the pointing gesture in order to determine the region indicated by the pointing act and would carry on a dialog with a user until Tobor correctly brings three apples from the produce stand to the user. 
         [0034]    Furthermore, the user-guided teaching of deictic references to the conversational system can be combined with haptic commands sent using a haptic interface of the vehicle or machine. The user may combine two modalities (verbal and haptic) in a single utterance for teaching the equivalence between utterances and actions to a conversational system. For example, uttering the phrase ‘I want to go there’ means that the user wishes to receive driving directions to a location that is indicated (not spoken) via a second modality (in this case, a touch-screen). Training the conversational system may include the user saying to the system “Learn something new” and the system may respond saying “what&#39;s it called”? The user says “I want to go there” and the system responds “Show me what to do”. The user turns to the navigation system and says “Navigation”, which responds by saying “Ready”. The user pushes ‘plot route’ screen button on the navigation system (haptic) and the navigation system asks “Where to?” The user speaks “Gather parameters” (a command to the learning module to expect data for each script execution). Further, the user points to a location (by touching a map) and the navigation system captures point coordinate message. The user speaks “Calculate route” and finishes by saying “Teaching complete. Executing the script for this example may proceed by the user speaking the phrase “I want to go there” and the system creates an asynchronous software process, such as, a Java “Listener” for the point coordinate message from the navigation system. Further, the system creates an asynchronous software process, such as, a Java “Listener” for the “Ready” message from the navigation system and also creates an asynchronous software process, such as, a Java “Listener” for the “where to?” message from the navigation system and sends the navigation command to start execution. The navigation system returns “ready”, where the “ready” listener triggers next phase of the script. The system sends plot route command to the navigation system and the navigation system returns “where to?” Further, the “where to?” listener triggers the next phase of the script and the user touches location on a map in the navigation system and the navigation system returns the touched map coordinates. The system inserts coordinates in command and sends “navigate route” to the map coordinates. It will be obvious to one of ordinary skill in the art that some learned commands cannot be described without including a step which gathers input from the user at each specific execution instance. The invention as described includes the capability for incorporating commands or sequences of commands given through the spoken language command and control system. Case specific user input requirements can be accommodated by including a spoken command such as “gather parameters” or “get user input” in the command set of the spoken language command and control system. 
         [0035]    Accordingly, the invention provides a machine, system and method for teaching deictic references and referent objects of deictic references to a spoken or conversational command and control system in a tools or machines, such as, cars and unspecialized tools, such as, robots, where the expertise can be built into the tool or machine itself by adding a user-guided deictic reference learning mode to such tools or machines to simplify their operation and greatly increase their utility. Accordingly, the invention provides means by which objects of a deictic reference can be taught to a conversational command and control system having a deictic reference tool that can be engaged or evoked by the user to increase the set of objects that can be distinguished by a computer system, where the deictic reference tool can verify its understanding of a deictic reference, and can correct any errors in its understanding of a deictic reference. Further, the distinguishable objects learned may be associated with a spoken name, where the name is used in the vocabulary of the conversational or spoken language command and control system to accomplish tasks that were not anticipated by the command and control system designer. As a deictic reference and the referent objects of the deictic reference are taught to a computing system, the computing system may be informed of errors in its discernment of the referent by the human teacher or by means built into its own software for measuring the certainty of identification of the referent. In such cases, the computing system may be equipped to perform its own act of deictic reference. It will be obvious to one of ordinary skill in the art that a system for user-guided teaching of deictic referents may be combined with a spoken language command and control system whether or not that system includes user guided teaching of command vocabulary and sequence. The taught deictic reference tool provides the means by which the objects to be acted upon may be specified while the command and control system provides the means by which the actions to be carried out on that object may be specified. Further, technology for object recognition may rely on any of several means well known to those skilled in the art, which include, but are not limited to, neural networks and hidden Markov models, which are recognition technologies composed of recognition algorithms and recognition data. It is commonly the case that some auxiliary software tool is used to process raw data, such as, processing a collection of images of a particular object into a form that can be used by the recognition algorithm to determine whether or not that class of object is present in the system&#39;s visual field. Further, a personal assistant robot equipped with a spoken language command and control system and the deictic reference system described herein may be taught to retrieve an object as an assistive act for a handicapped user. The set of objects that the user may need to deal with through the robot&#39;s mediation are likely to differ from the objects which other users deal with and, therefore, the robot will need to use the deictic reference system to be maximally useful. 
         [0036]    The foregoing descriptions of specific embodiments of the present invention have been presented for the purpose of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.