# Presentation: 512792

## Psychologically Plausible Models that Drive Synthetic Character Behavior within Applications for Assessing Interaction Skills

- <footer>

**Technology Assisted Learning Division**

- Rob Hubal, Paul Kizakevich, Geoff Frank,
- 	Curry Guinn
- 	RTI International
- Sandia Workshop
- 	30 June 2004

## Technology Assisted Learning Group at RTI

- Develop custom, practical PC-based solutions for military, commercial, government clients.
- Active in maintenance training, visualization, and other systems since 1993.
- Active in creating synthetic character applications since 1996.
- Staff of over 40 specialists in computer science and engineering, computer graphics, 3D programming, instructional design, learning analysis, project management. Also a cognitive psychologist.
- Work closely with Research in Education group at RTI.

## RTI International(Research Triangle Institute)

- RTI International(Research Triangle Institute)

- Non-profit founded in 1958 by Duke, UNC, NCSU as seed organization in Research Triangle Park.
- Multidisciplinary teams provide applied R&D services.
- Revenues $447m in FY2003.
- Over 2300 staffpersons.
- Projects and offices nationwide and worldwide.

## Interaction Skills Assessment Issues

- Interaction skills assessment increasingly complex, expensive, necessary.
- Skills often assessed through questioning, questionnaires, or on-the-job.
- Limited use of or ability to set up role-playing.
- Formal methods (e.g., standardized patients) useful but restricted.
- For practical and ethical reasons, conditions often not conducive to use of role-playing.
- One solution is “virtual role-playing” using synthetic characters.

## Virtual Role-playing Architecture

- Environment
- Adaptation

- User

- Knowledge Base

- Natural Language
- Processing

- Behavior
- Engine

- Synthetic Character

_*Models*_
*Affective*
*Cognitive/social*
*Physiological*
*Language*
*Gesture*

## Three Applications

- JUST-TALK.
- Trauma Patient Simulator.
- “What Works Best” Study.

## JUST-TALK Goals

- Research methods for using natural language processing and virtual reality to improve police training.
- Determine how to use virtual role-playing for police training.
- Determine methods for providing useful computer-based training using NLP and VR technology to the broadest range of law enforcement agencies.

## JUST-TALK Scenario Organization

- Introduction.
- Dispatch call.
- Verbal interaction:
- Introduction.
- Standard procedure (name, department, dispatch).
- But also common variations.
- De-escalation - Crisis Stages model.
- Interview (name, family, illness, medication, event).
- Resolution.
- After-action review.

## Possible Resolutions

- No action is taken (correct outcome for 2 of 5 scenarios).
- Subject is persuaded to get into the patrol car to be taken to a mental health facility for diagnosis, observation, or treatment (correct outcome for 3 of 5 scenarios).
- Subject runs away (bad outcome).
- Subject attacks the trainee (bad outcome).
- Subject goes catatonic and refuses to talk (bad outcome).

## Crisis Stages Model

- Developed by Randy Dupont at Memphis in conjunction with Memphis CIT.
- Describes escalation stages that roughly correspond to emotional states.
- At each stage, specifies good and bad actions.
- Being tested by equating escalation stages with emotional states and forcing gestural, expressive, and verbal output based on the current stage.

- Stage

- Uncertainty

- Questioning

- Refusal

- Demanding

- Recovery

- Post-crisis

- What to Try

- Address relevant questions

- Provide structure

- Use a simple request

- Provide legitimate support

- Reinforce calm behavior

- Rebuild rapport

- What to Avoid

- Defensiveness

- Passivity, counter-transference

- Power struggle

- Intervening prematurely

- Re-escalation

- Blame, guilt inducement

## After Action Review

## Three Applications

- JUST-TALK.
- Trauma Patient Simulator.
- “What Works Best” Study.

## TPS Goals

- Develop virtual patients for practicing assessment, diagnosis, treatment, and interpersonal skills for people with medical, physical, and psychological disorders.
- Develop and deploy software using virtual patientsfor initial training, knowledge and skills validation,and sustainment training for all levels of medical care.
- Investigate the efficacy of virtual patient technologies for acquiring, maintaining, and validating competency for all levels of medical care.

## Virtual Patient Visualization

*Abdominal penetration*

- Full-body medically-relevant animations.
- Interactive body regions (e.g., wrist, left) and interactive medical devices.
- Skin texturing of clinical signs & injuries.
- Multi-layered, deformable & removable clothing.
- Breathing integrated with real-time physiology.
- Set pupil size and animate pupil response.

*Extremity fracture*

*Vomiting*

*Nausea*

*Convulsions*

## Virtual Patient Characteristics

- Cognitive (level of consciousness).
- Social/historical:
- Medical history (health states, medications, findings).
- Temporal history (recent events, meals, travel, activity).
- Emotive (anxiety, fright, sadness, quiescent).
- Linguistic (dynamic speech production).
- Physiology:
- Pharmacokinetic drug model.
- Normal, disease, trauma, mental states.
- Protocol structure.

## Physiology Modeling

*Real-Time Data and Waveforms*

*Trend Data*

- Core capability:
- Cardiovascular model, with physiological waveforms .
- Respiratory gas exchange model.
- Pharmacokinetic models (over 70 medications).
- Visualization of natural and aided respiration.
- Variations:
- Airway obstruction, respiratory arrest.
- Hemorrhage, cardiac tamponade, cardiac arrest.
- Pneumothorax (simple, tension, open, hemo).
- Relief provided by appropriate care.
- Cyanide exposure and treatments.
- Based on the Starko & Smith BODY model.

**Notes:**
The STATcare dual use science & technology program will build on the successful ACT-II Trauma Patient Simulator to provide casualty care simulation for both civilian and military emergency personnel.

FEATURES:
*  TPS is unique: no other product provides real-time, interactive trauma care training on a personal computer. 
*  Physiologically-accurate -- based on anesthesiology simulator, with models in scientific literature
*  Includes pharmacokinetic models for over 70 drugs
*  Reconfigurable scenarios --free-play  simulation, not driven  by predetermined storyboards or protocol trees
*  Lifelike caregiver-patient dialog
*  Integrated  web-based courseware and reference (i.e, medic& EMT distance learning materials)

SCREEN SHOT:
*  Pop-up videos/web pages of medial procedures 
*  Flexible database of medical procedures
*  Real-time cardiac and respiratory signals
*  3D interactive model using Windows-95 compatile VR

VALUE & BENEFITS:
*  PRACTICE infrequent trauma care situations
*  SUSTAIN rapid assessment &  decision-making skills
*  ASSESS  skills for Field MEdical Badge
*  ENABLE new training capabilities and objectives
*  Truely a dual-use medical  R&D  program

## Protocol Structure

## Three Applications

- JUST-TALK.
- Trauma Patient Simulator.
- “What Works Best” Study.

## WWB Goals

- Part of study of executive cognitive functioning deficits that may contribute to differential responses to preventive interventions.
- Audience – at-risk inner-city teenage males, assessed with/without conduct disorder.
- Monitor how participants handle vignettes designed to elicit risky behavior.
- Social/cognitive skills.
- Emotional control.
- Decision-making ability.
- Synthetic character – teenage male.

## PACT Content

- Positive Adolescents Choices Training developed by Hammond & Yung at Wright State.
- Social skills training program developed specifically for African American youth.
- Intended to reduce disproportionate risk for becoming victims or perpetrators of violence.
- Uses videotapes containing vignettes of a conflict situation and demonstrating skills that could diffuse a potentially violent situation.
- Target skills include:
- Giving negative feedback or expressing criticism calmly.
- Reacting appropriately to the criticism or anger of others.
- Identifying problems and solutions.

## Scenarios & Outcomes

- Three pairs of scenarios:
- Stolen goods.
- Party, car-ride invitation.
- Hallway, girlfriend confrontation.
- Participant behavior analyzed in terms of:
- Engagement with vignettes.
- Body language.
- Verbalizations.
- Number of conversational turns.
- Response time.
- Outcome.
- Positive outcome is to decline or back away.
- Negative outcome is to agree or escalate confrontation.

## Findings

- Participants agree to help/go along with/escalate confrontation with the synthetic character one-third to half of time.
- All queried participants stated their virtual decision mirrored what would be their real-life decision.
- Factorial validity: performance measures load on two factors corresponding to emotional control, interpersonal communication skills.
- Some support for criterion, construct validity.

## Model Specifics

- Cognitive.
- Emotional.
- Linguistic.
- Gestural.
- Methods used to gather modeling data.

## Cognitive Modeling

- Track domain knowledge via:
- State variable settings.
- Planning structure inherent in augmented transition network.
- What is known about the world:
- What is the intent of the scenario (e.g., user has to check breathing with a stethoscope; user is asked to store stolen goods).
- Reason about social roles and conventions:
- What can be stated or asked at any point in the dialog.
- How it gets stated or asked.
- Application-specific parameters and initial conditions drive how synthetic characters express knowledge.

## Emotion Modeling

- Our own mix sampled from lots of offerings (Five Factor Model, Circumplex theory, OCC cognitive theory of emotions):
- Maintain continuous base emotion variables anchored by modifiable descriptors.
- Map emotional states onto base variables and enable updating:
- Emotion mapping:
- AFRAID  (FEAR >= SOMEWHAT_HIGH)
- Emotion setting:
- APPEASE  (ANGER -= VOLATILITY * MEDIUM)
- Separate personality (stable; e.g., VOLATILITY) from mood (dynamic; e.g., FEAR).
- Updates based on student’s input:
- Format (command, request, statement).
- Lexical analysis (politeness, degree of compromise, detachment, personalization).
- Semantic content (topic relevance, sensitivity).
- Actions in the virtual world (expected or not, desired or not).

## Linguistic Modeling

- Understand structure, flow of conversations.
- Interpret inputs:
- Parser relies on a minimum distance translator algorithm.
- Interpret semantics; grammars tagged with linguistic, social variables (politeness, complexity, personalization).
- Syntactic analysis determines type of utterance (command, query or request, informative statement, statement of appreciation or understanding).
- Lexical analysis determines specific particularly bad phrases (insults, threats).
- Generate appropriate responses:
- Reply, question, challenge, deny, zone out.
- Along the lines of Linguistic Style Improvisation (but not as comprehensive).
- Appropriateness to particular synthetic character.
- Provide informative gestural animations:
- Whole body, head, arms/hands, facial expression (using emotional state mappings).
- Following pragmatics of conversation, as understood by particular synthetic character.

## Gesture Modeling

- Gestures:
- Most verbal utterances accompanied by gestures.
- Most representational, rest beats, idle motions.
- Eye gazes:
- Looking at conversational partner indicates feedback sought.
- Staring, averted gaze imply intimidation, embarrassment, etc.
- Posture:
- Sustained glance and a smile invites contact.
- Head bent and shoulders down typical of unease, sadness.
- Heads up for deference to authority.
- Eyes, mouth wide open, torso back indicates fear.
- Visualization engine:
- Expressions implemented using percentage blending of morph-targets.
- Body movement implemented using key-framed animations based on motion capture.

## Methods Used to Gather Modeling Data

- Subject-matter expert guidance and feedback.
- Videotapes:
- Of paranoid-schizophrenic behavior.
- Of police responses to calls involving persons with mental illness.
- Of adolescents in natural settings.
- Used in existing intervention programs.
- Direct observation (of users of simulations).
- Scripted and unscripted interactions.
- Motion capture.
- Literature on:
- Normal behavior characteristics (gesture, facial expression, language, conversational pragmatics).
- Abnormal behavior characteristics.
- Children’s behavior.

## Agency for Healthcare Research and Quality.
National Institute of Justice.
National Institute on Drug Abuse.
National Institutes of Health.
National Science Foundation.
RTI Internal Research and Development.

- National Institute of Justice.
- National Institute on Drug Abuse.
- National Institutes of Health.
- National Science Foundation.
- RTI Internal Research and Development.

- Funding Acknowledgements

## Questions?