Abstract:
An information system that determines the apparent emotional state of the user and, based upon that determination, reacts appropriately. Appropriate reactions of the system include marking user text input to convey the apparent emotional state of the user to recipients. Other embodiments incorporate reactions of the system that warn or prompt a user, prior to system execution of a user directed operation, when the system has determined that the apparent emotional state of the user merits a warning or a prompt.

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     This invention relates generally to control of data processing or information systems, and in particular to control of such systems based in part upon the detected apparent affective or emotional state of a user. 
     2. Description of the Related Art 
     In the history of human interaction with information systems, system reaction to the user actively engaged with the system generally has been based upon programmatic response to a user&#39;s intentional communicative acts, such as keyboard entry, mouse manipulation, voice input and the like. While certain information systems and applications respond to forms of information from a user that are not intentional communicative acts by the user, such as autonomic conditions in medical monitoring systems or biometric parameters for user identification in security systems, these system responses are generally not within the context of active engagement of the user with the system. 
     Yet for humans actively engaged with each other, an individual&#39;s response to another involves considerably more than reacting to the mere intentional communicative acts of the parties. In particular, effective interaction with others requires perceiving and reacting to the apparent affective or emotional states of others as well as to their intentional communicative acts. 
     An information system capable of determining a user&#39;s apparent affective state would be capable of altering its behavior in response to that apparent state, resulting in more effective interaction with the user. For example, in applications such as email, when a user is in communication with other users, a user&#39;s rash emotional reaction in communication may result in the user&#39;s transmitting an undesirable message. The system in this example, then, having determined that the user&#39;s apparent emotional state is such that the user&#39;s communication may be a rash response, may prompt the user for confirmation prior to transmission. What is needed for more effective human interaction with information systems is for information systems to determine and react appropriately to the apparent affective or emotional state of humans with whom they are actively engaged. 
     In addition to human-computer interaction, humans also use information systems as intermediaries in communication with other humans, as in email and word processing applications. Because effective interaction with other humans requires conveying affective or emotional states, a number of means of conveying such states have been developed by convention for written communications. For example, a correspondent&#39;s typing in all capital letters may serve to convey to the recipient the same emotional state that yelling conveys in spoken communications. Additionally or in the alternative, special combinations of common characters may be typed by a correspondent to convey an emotional state to a recipient, such as the now familiar colon-right parenthesis combination :) used to convey happiness or humor. Acronyms typed by a correspondent and understood in common with the recipient can serve the same function, for example “LOL” (laughing out loud) and “ROTF” (rolling on the floor). Depending upon display capabilities of the information systems of correspondent and recipient, the correspondent may convey emotional or affective state to the recipient by typing keystrokes or using function keys to insert special characters (such as a smiley face            ), to change a font to bold or italic or to underline for emphasis, or to color font in appropriate portions of text. To convey affective or emotional state in any of these and like conventions used in written communications, the correspondent must intentionally direct an input device, such as a keyboard or a mouse, to insert characters or mark text to convey the correspondent&#39;s emotional state. What is needed is for the correspondent&#39;s information system, without any intentional direction from the correspondent, to determine the apparent affective or emotional state of the correspondent and react by marking appropriate text in such a way that by convention conveys the emotional state of the correspondent to the recipient.
     SUMMARY OF THE INVENTION 
     The present invention shows a way for information systems to determine the apparent emotional or affective state of a user engaged with the system and, further, for information systems, based upon such determination, to react accordingly. 
     It is an object of the present invention to provide means for information systems to determine the apparent emotional or affective state of a user. It is a further object of the present invention to enable information systems to make intelligent decisions in applications programs based upon a determined apparent emotional or affective state of a user. It is a further object of this invention, having determined the apparent emotional or affective state of a correspondent, to convey that apparent state in communications transmitted by the correspondent to one or more recipients. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Other objects, advantages, features and characteristics of the present invention, as well as methods, operation and function of related elements of structure, and the combination of parts and economies of manufacture, will become apparent upon consideration of the following description and claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures, and wherein: 
         FIG. 1  is a block diagram of a general purpose information handling system, which may be used to implement the invention. 
         FIG. 2  is a flow chart of a process for applying a determination of apparent affective state to marking of text in a text composition application. 
         FIG. 3  is a flow chart of a process for applying a determination of apparent affective state in an interpersonal assistant application that warns a user of possible inadvisability of sending email. 
         FIG. 4  is an exemplary warning output of the process shown in  FIG. 3 . 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     In the following detailed description of the preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific preferred embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the spirit or scope of the invention. To avoid detail not necessary to enable those skilled in the art to practice the invention, the description may omit certain information known to those skilled in the art. The following description is, therefore, not to be taken in a limiting sense, and the scope of the invention is defined only by the appended claims. 
       FIG. 1  shows a block diagram of an information handling system  100  in accordance with the present invention. In this embodiment, processor  102 , system controller  112 , cache  114 , and data-path chip  118  are each coupled to host bus  110 . Processor  102  is a microprocessor such as a 486-type chip, a Pentium®, Pentium II®, Pentium III®, or the like suitable microprocessor. Cache  114  provides high-speed local-memory data (in one embodiment, for example, 512 KB of data) for processor  102 , and is controlled by system controller  112 , which loads cache  114  with data that is expected to be used soon after the data is placed in cache  112  (i.e. in the near future). Main memory  116  is coupled between system controller  112  and data-path chip  118 , and in one embodiment, provides random-access memory of between 16 MB and 128 MB of data. 
     In one embodiment, main memory  116  is provided on SIMMs (Single In-line Memory Modules), while in another embodiment, main memory  116  is provided on DIMMs (Dual In-line Memory Modules), each of which plugs into suitable sockets provided on a motherboard holding these components and many of the other components shown in  FIG. 1 . Main memory  116  includes standard DRAM (Dynamic Random-Access Memory), EDO (Extended Data Out) DRAM, SDRAM (Synchronous DRAM), or the like suitable memory technology. System controller  112  controls PCI (Peripheral Component Interconnect) bus  120 , a local bus for system  100  that provides a high-speed data path between processor  102  and various peripheral devices, such as video, disk, network, etc. Data-path chip  118  is also controlled by system controller  112  to assist in routing data between main memory  116 , host bus  110 , and PCI bus  120 . 
     In one embodiment, PCI bus  120  provides a 32-bit-wide data path that runs at 33 MHz. In another embodiment, PCI bus  120  provides a 64-bit-wide data path that runs at 33 MHz. In yet other embodiments, PCI bus  120  provides 32-bit-wide or 64-bit-wide data paths that run at higher speeds. In one embodiment, PCI bus  120  provides connectivity to I/O bridge  122 , graphics controller  127 , and one or more PCI connectors  121 , each of which accepts a standard PCI card (not shown). In one embodiment, I/O bridge  122  and graphics controller  127  are each integrated on the motherboard along with system controller  112 , in order to avoid a board-to-connector-to-board signal crossing interface, thereby providing better speed and reliability. In the embodiment shown, graphics controller  127  is coupled to a video memory  128  that includes memory such as DRAM, EDO DRAM, SDRAM, or VRAM (Video Random-Access Memory), and drives VGA (Video Graphics Adapter) port  129 . VGA port  129  can connect to VGA-type or SVGA (Super VGA)-type displays or the like. Other input/output (I/O) cards having a PCI interface can be plugged into PCI connectors  121 . 
     In one embodiment, I/O bridge  122  is a chip that provides connection and control to one or more independent IDE connectors  124 , to one or more SCSI connectors  125 , to one or more USB (Universal Serial Bus) ports  126 , and to an ISA (Industry Standard Architecture) bus  130 . In this embodiment, IDE connector  124  provides connectivity for up to two or more standard IDE-type devices, in particular those for non-volatile memory storage ano/or retrieval such as hard disk drives, CD-ROM (Compact Disk-Read-Only Memory) drives, DVD (Digital Video Disk or Digital Versatile Disk) drives, or TBU (Tape-Backup Unit) devices. As will be appreciated by those skilled in the art, client systems in a network, such as web pads, need not be equipped with any such non-volatile memory storage devices, relying instead upon the function of such devices in a server to which the client is connected. 
     In one similar embodiment, two IDE connectors  124  are provided, each providing an EIDE (Enhanced IDE) compliant architecture. In the embodiment shown, Small Computer System Interface (SCSI) connector  125  provides connectivity for preferably up to seven or fifteen SCSI-type devices depending on the version of SCSI supported by the respective embodiment. In one embodiment, I/O bridge  122  provides ISA bus  130  having one or more ISA connectors  131  (in one embodiment, three connectors are provided). In one embodiment, ISA bus  130  is coupled to I/O controller  152 , which in turn provides connections to two serial ports  154  and  155 , parallel port  156 , and FDD (Floppy-Disk Drive) connector  157 . In one embodiment, FDD connector  157  is connected to FDD  158  that receives removable media (floppy diskette)  159  on which data and/or program code  160  is stored. 
     In one such embodiment, program code  160  includes code that controls programmable system  100  to perform an application program as described in accordance with the invention. In an embodiment typical for client systems, and characteristic of “thin clients” such as web pads, serial port  154  is connectable to a computer network such as a local network or the Internet, and such network has program code  160  that controls programmable system  100  to act as a client, receiving and interpreting data sent by a matching server computer application. In another such embodiment characteristic of server systems, serial port  154  is connectable to a computer network such as a local network or the Internet, and special program code  160  within programmable system  100  executes that causes programmable system  100  to act as a server, providing data and applications over the network to a matching client computer program that is capable of properly interpreting that data and applications. 
     In one embodiment, ISA bus  130  is connected to buffer  132 , which is connected to X bus  140 , which provides connections to real-time clock  142 , keyboard/mouse controller  144  and keyboard BIOS ROM (Basic Input/Output System Read-Only Memory)  145 , and to system BIOS ROM  146 .  FIG. 1  shows one exemplary embodiment of the present invention, however other bus structures and memory arrangements are specifically contemplated. It should be appreciated that modification or reconfiguration of information handling system  100  of  FIG. 1  by one having ordinary skill in the art would not depart from the scope or the spirit of the present invention. 
     An information system may determine the apparent affective or emotional state of a user based upon a variety of sources of information, which may be used singly or in combination for the information system to make its determination. See generally Picard, Rosalind W. “Teaching Computers to Recognize and Express Emotion” in: Stork, David G. (ed.),  Hal&#39;s Legacy:  2001 &#39;s  Computer as Dream and Reality  (Cambridge, Mass. MIT Press. 1997), Chapter 13, Section 4. 
     In applications where the user is inputting textual content, such as word processing or email composition, the actual content of the input may give a good indication of the user&#39;s affective state. For example, the incidence of vulgar language or derogatory epithets in text may indicate anger or frustration, with frequency of such words providing an indication of the intensity of the user&#39;s emotional state. By monitoring the incidence and frequency of words indicating a user&#39;s emotional state, an information system may make a determination of the user&#39;s apparent affective state. 
     As previously discussed in reference to the prior art, the user may mark textual input in such a way as to indicate the user&#39;s emotional or affective state with respect to the text or context in question. Such marking may be by way of highlighting the text in some way, such as alteration of font appearance by color, underlining, bold, italic, capitals, etc. In the alternative, such marking may be by way of special character combinations or special characters inserted in appropriate textual locations. In any case, the correspondent conveys his emotional state to the recipient by such marking on the assumption that the recipient shares the convention that associates that marking with the affective state of the correspondent. An information system may determine a correspondent&#39;s apparent emotional state by monitoring text input for markings that, by convention, convey emotion. 
     The facts surrounding the user&#39;s composition and transmission of text in such applications may be indicative of the user&#39;s affective state. For example, if the period of time between a user&#39;s first access of an email message and the user&#39;s reply to that message is very short, it is possible that the user has reacted emotionally to the received message and has responded rashly based upon that emotional response. Similarly, a rash response may be indicated if the length of time used to compose the message is very short but the recipient list of the message is very long or the recipients include key individuals such as the head of the sender&#39;s organization. An information system may, in such circumstances, make a determination of the user&#39;s apparent emotional state in the composition of email by monitoring the time to reply, the time to compose, and the distribution list of the email. 
     A user&#39;s input characteristics may also be indicative of the user&#39;s emotional state. For applications using standard manual input devices, pronounced force of keystrokes on a keyboard or mouse clicks on a mouse may indicate anger or frustration. Input devices such as keyboards and computer mice that are capable of detecting and reporting the degree of force in user input are well known in the art (see, for example, U.S. Pat. No. 5,579,238 to Krugman, U.S. Pat. No. 5,982,357 to Burgett and U.S. Pat. No. 5,995,026 to Sellers). By employing such input devices, an information system may determine apparent user affective state by monitoring the force exercised by the user in input operations. 
     A user&#39;s input characteristics may also be determined in applications using speech input by analysis of the user&#39;s speech. As set forth in the Hal &#39;s Legacy reference, vocal intonation is a common way to communicate strong feelings. Several features of speech are modulated by emotion; we divide them into such categories as voice quality (e.g., breathy or resonant, depending on individual vocal tract and breathing), timing of utterance (e.g., speaking faster for fear, slower for disgust), and utterance pitch contour (e.g., showing greater frequency range and abruptness for anger, a smaller range and downward contours for sadness). Analysis of these characteristics in a user&#39;s speech by an information system provides the information system with an indication of the user&#39;s emotional state. 
     Autonomic responses are also indicative of a user&#39;s emotional state. While some autonomic responses may be difficult to monitor unobtrusively, galvanic skin response of a user&#39;s hands or fingers may be monitored by input devices such as keyboards or computer mice. Electrodes on such input devices at appropriate points of contact with the user&#39;s hands may be used by an operatively connected information system to monitor the user&#39;s skin conductivity. As is well known in the art (see, for example, Kasa et al., U.S. Pat. No. 4,625,732), increased skin conductivity is associated with a user&#39;s affective state. Monitoring the user&#39;s skin conductivity thereby provides the information system with an indication of the user&#39;s apparent emotional state. 
     Other autonomic responses that may be measured fairly unobtrusively are the dilation of the user&#39;s pupils and the rate at which the user blinks his eyes. It is well known in the art that pupil dilation and eye blink rate, just as skin conductivity, are associated with a person&#39;s affective state (see, for example, U.S. Pat. No. 4,931,865 to Scarampi). By means of an operatively coupled video camera, an information system may monitor a user&#39;s pupil dilation and/or blink rate and thereby make a determination of a user&#39;s apparent emotional state. 
     Face recognition software technology may also be employed by this invention. Typically analyzing a computer graphic of a face, such software, used in conjunction with a video camera operatively coupled with the information system, can determine the identity of the user and the user&#39;s apparent emotional state. At present, several algorithmic approaches to face recognition provide sufficiently accurate performance in face recognition for application in this invention. Some approaches are based upon eigenvectors generated from the autocorrelation matrix of a computer image of a face. These eigenvectors, termed “eigenfaces” by persons skilled in the art, may be analyzed by vector transform followed by discriminative modeling to yield reliable face recognition determinations. In such approaches, some algorithms employ a linear discriminant, while others, such as that used in the Viisage Gallery application of Viisage Technology, Inc. of Littleton, Mass., employ a quadratic discriminant. In the alternative, within these approaches using eigenfaces, some algorithms, such as those used in FaceIt® software by Visionics Corp. of Jersey City, N.J., may employ a sparse transform of the eigenfaces, followed by a discriminative neural network analysis. 
     Illustrating yet another approach, TrueFace software from Miros, Inc. of Wellesley, Mass., relies upon a method that extracts and analyzes local features from a variety of faces. Local features of a face are extracted from an eigenface by a determination of which features of a face differ most from features of other faces in a database. By applying neural network technology to the analysis of extracted local features in a database derived from computer images, this software improves its face recognition success rate with experience. 
     In any case, the present state of the art allows those of ordinary skill in the art, without undue experimentation or testing, to use face recognition software to analyze a user&#39;s facial expression and thereby determine the apparent emotional or affective state of the user. 
     Turning now to the remaining drawings,  FIG. 2  is a flow chart of a procedure for applying a determination of apparent affective state to marking of text in a keyboard text input and composition application. This procedure, which processes text input prior to formatted output, commences at  201 . In  202 , the user&#39;s previously determined affective state is initially set by default to “neutral”. A loop commencing at  203  scans the keyboard for input. Upon obtaining keyboard input  204 , the system proceeds to determine the user&#39;s apparent affective state  205 , by way of means well known to those of ordinary skill in the art, as set forth by example above. Using, for example, the message content, input device force detection, autonomic indicators or facial expression, or any combination of methods well known to those of ordinary skill in the art to determine the user&#39;s apparent affective state at  205 , the system compares the newly determined affective state to the previously determined affective state at  206 . If these states are the same, the system proceeds to output text at  210  without a change in formatting. 
     Considering the initial condition of this aspect of the invention, the initial setting of text formatting by default corresponds with communicating a “neutral” affective state in this embodiment. Since the initial setting of the user&#39;s previously determined affective state is “neutral” at  202 , if there is no difference between this initial state and the current affective state newly determined at  205 , step  207  will cause the initial output of text that is appropriately marked “neutral” by default at  210 . If, on the other hand, the current affective state differs from the initial setting of the user&#39;s previously determined affective state (“neutral” in this embodiment) step  207  will direct step  208  to change text formatting appropriately and the initial output of text will be appropriately marked at  210 . 
     As set forth by example above, text formatting indicating emotional state may be indicated  208  by way of highlighting the text in some way, such as alteration of font appearance by color, underlining, bold, italic, capitals, etc. In the alternative, such marking may be by way of special character combinations or special characters inserted in appropriate textual locations. In any case, the information system will change text formatting at  208  in a manner well known in the art, marking the text in such a way as to be likely to convey the user&#39;s affective state to the intended recipient of the composition. 
     Subsequent to the initial condition, if the current affective state determined at  205  differs from the previous affective state (which is to say the user&#39;s apparent emotional state has changed), the system changes text formatting appropriately at  208  as described above. At  209  the variable indicating the user&#39;s present affective state is reset to indicate the current state as determined in step  205 . Appropriately formatted text is output at  210 . After this procedure has output text, it continues to scan the keyboard for input, monitoring the user&#39;s apparent affective state and modifying text markings as necessary and appropriate. 
     Referring next to  FIG. 3 , a flow chart is shown of a procedure for applying a determination of apparent affective state to prompting whether to transmit a user&#39;s electronic mail composition. This procedure, which is an application operating between the composition and the transmission of email, commences upon the user&#39;s activation  301  of a “send” function for email from the user directed to one or more recipients. The information system determines the user&#39;s apparent affective state at  302  by using, for example, the message content, input device force detection, autonomic indicators, facial expression, or any combination of such methods. In the alternative, or additionally, the information system may make its determination of the user&#39;s apparent affective state at  302  by facts surrounding the user&#39;s composition and transmission of text. As set forth above, such facts may include the period of time between a user&#39;s first access of an email message and the user&#39;s reply to that message, or the length of time used to compose the message related to the intended recipients of the message. In any case, the information system, by methods well known to those of ordinary skill in the art, makes a determination of the apparent affective state of the user at  302 . If the apparent affective state of the user is such that the user would be well-advised to exercise particular prudence in transmission of email  303 , the system will warn the user and prompt for confirmation at  304 .  FIG. 4  is illustrative of a typical warning and prompt, showing the circumstances leading the information system to determine that the user&#39;s affective state might be questionable. If and only if the user confirms the prompt  305 , the email is transmitted at  306  and the program exits  307 . 
     Although the invention has been described with a certain degree of particularity, it should be recognized that elements thereof may be altered by persons skilled in the art without departing from the spirit and scope of the invention. Accordingly, the present invention is not intended to be limited to the specific forms set forth herein, but on the contrary, it is intended to cover such alternatives, modifications and equivalents as can be reasonably included within the scope of the invention. The invention is limited only by the following claims and their equivalents.