# Presentation: 016400

## A Framework for the Comprehensive Computational Representation of Human Decision Processes 

Chris Forsythe
Sandia National Laboratories
Albuquerque, NM
jcforsy@sandia.gov

**A Framework for the Comprehensive Computational Representation of Human Decision Processes**** **
**Chris Forsythe**
- Sandia National Laboratories
- Albuquerque, NM
- _[jcforsy@sandia.](mailto:jcforsy@sandia.gov)__[gov](mailto:jcforsy@sandia.gov)_

**RPD-Based Cognitive Modeling**

## Acknowledgements

**Sandia National Laboratories Team**
** ****Patrick Xavier, Eric Parker, Ed Thomas, Dave Schoenwald, Sabina Jordan, Caren Wenner, Nathan Brannon, Elaine Raybourn, Howard Hirano, .....**
**University of New Mexico**
** ****Tim Goldsmith**
**Northern Illinois University**
** ****Joe Magliano, Mary Anne Britt, Christoph Guess**

## Program Objectives

**Objective: collection of projects focused on development of a framework for the **_**comprehensive**_** modeling of factors that shape human decision processes in naturalistic settings**

**Technical Challenge: **_**realistically**_** model decision making processes in a manner that is comprehensive, defensible, extensible and doable.**

**Problem Space: (1) Synthetic humans for training and analytic tools; (2) Intelligent machines; and (3) Core technology for augmented cognition solutions**

## Critical Requirements for Realism of Synthetic Entities

**  ****Representation of emotional processes including the interaction between emotions, arousal and cognitive processes**
**  ****Representations of knowledge that provide a broad range of relevant, and also, irrelevant behavioral responses**
**  ****Mechanisms to address variations in knowledge and emotional associations attributable to cultural differences**
**  ****Mechanisms that enable non-linear patterns of behavior and reasonably realistic reactions to non-linear behavioral responses**** **

## Theoretical Synthesis

**The simulation incorporates a theoretical synthesis of Klein’s RPD model of naturalistic decision making (Klein, 1993), oscillating systems theory of semantic and episodic memory processes (Klimesch, 1996) and LeDoux’s (1998) model of the relationship between cognitive and emotional processes (Forsythe, 2001).  **

## Modeling Recognition Primed Decision Making

- Semantic Knowledge

- Situational Knowledge
-     - Situation A
-     - Situation B
-     - Situation C
-     -
-     -

- Pattern Recognition

- Interpretation of Situation

- Cues and Knowledge of Ongoing Events

** ****Cues in the environment activate concepts in semantic net**

**Patterns of activation are recognized that correspond to known situations**

**Knowledge of appropriate actions (e.g., scripts) are implicit to situation recognition**

**Situations and corresponding knowledge**

## Cognitive Model: Initial Pattern Recognition Approach

**Cognitive Model: Initial Pattern Recognition Approach**

**Environmental Cues and**
**Knowledge of Ongoing Events**

**Concept A**

**Concept B**

**Concept C**

**Concept D**

**Concept E**

**Situation 1**

**Situation 2**

**Situation 3**

**Begin Snapshot**

**End Snapshot  **

**Semantic Network**
- Perceptual events and knowledge lead to activation of concepts in a semantic network with activation spreading to related concepts

**Pattern Recognition**
- The pattern of activation in the semantic network is matched with patterns associated with known situations

**Situation Recognition**
- Sequential snapshots increment or decrement evidence for alternative situations in a race model

## Physiology-Based Engine: Foundation in Oscillating Systems Theory

**Knowledge Network**

**Situation Recognition**

**Oscillators represent each node in the network.  In operation, the activity of these oscillators may vary in frequency and amplitude.**

**A single oscillator underlies the pattern recognition process.  Operating at a lower frequency (10-13 Hz for semantic processes versus 4-7 Hz for recognition) it provides a relatively broad window for recognition of patterns in the activation of the semantic network.**

**Notes:**
Physiological model based on oscillating systems theory of the brain.  We are not building an artificial brain.  We are mimicking in order to have the right machinery to be able to model the effects of a variety of organic factors.  

## Properties of Semantic Processes: Reverse Engineering and First Steps Toward V&V

_**Example of Specification**_**:**
**The rate of information processing corresponds to the dominant frequency in the 10-13 Hz Bandwidth**
- (Klimesch 1996)

**Frequency of Oscillations for Neural Units (Hz)**

**Decision Time (sec)**

**Simulated Event-Related Potential**

**Actual Event-Related Potential**

**Notes:**
One output of the model is a simulated EEG, used as a validation tool

Model based on a collection of neural assemblies.  Cognitive operations are a product of activity in these neural assemblies.  A second product is an electrophysilogical response (EEG).

Example is a study we simulated with our model and the figure shows the results of the simulation which are consistent with those reported from the study with human subjects.  EEG response was also consistent with that reported with human subjects  - Semantic memory task (e.g., is dog a mammal)

## Emotions: Fear

**Emotion leads to increased activation of neural assemblies corresponding to the stimulus event or situation associated with emotional reaction, and active inhibition otherwise **(LeDoux 1998)

**Activation Observed in the Absence of Fear-Inducing Stimulus**

**No association**

**Association**

**No association**

**Association**

**Activation Observed with Fear-Inducing Stimulus Presented at Approximately 2 Seconds.**

**Notes:**
Emotion leads to increased activation associated with emotionally charged stimulus and active inhibition of everything else.  If talking about cognitive models and a snake falls out of the ceiling, likely will no longer be thinking about cognitive models.

## Issues Now Occupying Our Minds

** ****Population of Models with Knowledge**
**  ****Knowledge Elicitation**
**  ****Knowledge Representation**
** ****Pattern Recognition**
** ****Episodic Memory**
** ****Perceptual Processes**
** ****Scalability, Interoperability**
** ****Population Models**
** ****Details, Characterization and Validation**

## Population of Models with Knowledge: Elicitation for Semantic Knowledge

**Approach developed and validated by Schvaneveldt, Goldsmith and colleagues based on Pathfinder Networks (Schvaneveldt, 1990)**
**Identify critical concepts**
**Collect relatedness data for concept pairs**
**Develop Pathfinder Network identifying associative links between concepts and associative strength of these links**
**Implement in model where concepts represented as nodes with associative links and strengths of associative links provide basis for spreading activation**

## Representation in Model: Semantic Module

**Activation routed to individual nodes**

**Each concept represented as separate node, with oscillatory properties**

**Timer operates as pacemaker sending periodic pulses to network**

## Larger Semantic Spaces: Latent Semantic Content Analysis

**Goal to develop customized knowledge structures based on domain, culture, knowledge of individual, etc.**

**Books, Magazines, Newspapers, Movie & Television Scripts, etc.**

**Text Parser**

**Concept**
**Concept**
**Concept**
**Concept**

**Sentences/Paragraphs**

**0  1  1  1  0**
**1  1  0  1  0**
**0  0  0  0  1**
**0  0  1  0  0**

**1  2  3  4  5  .  .  .**

**A**

**B**

**C**

**D**

**Singular Value Decomposition**

**Points in Multi-Dimensional Space**

**Semantic Network Representation**

## Population of Models with Knowledge: Elicitation for Situational Knowledge

**Utilized approach wherein task analysis emphasizes actions and associated cues, not unlike CDM (Klein, et.al., 1989)**
**Identify actions and develop associated flowcharts**
**Identify cues associated with actions providing basis for discrimination of situations**
**??? Relatedness cues-to-situations, so far done analytically**
**Implement in model where pattern recognition based on cues and endpoints of flowchart serve as separate actions**
**??? Pattern recognition, initially template match, subsequently used information accumulation and neural net approaches**

## Representation in Model: Situation Recognition Module

**Timer operates as pacemaker sending periodic pulses to network**

**Assign evidence based on semantic activation –**
**??? Knowledge Elicitation**

**Situation activation if continuation of current situation –**
**??? Assume bias for current situation**

**Assess evidence if not continuation of current situation**

**Assess cues contrary to expectations – **
**??? Knowledge Elicitation**

**Introduce top-down activation for semantic concepts – **
**??? Knowledge Elicitation**

**Placeholder for Level 2 & 3 RPD**

## Episodic Memory

**The objective is a computational model capable of the following:**
**  ****Store representations of experience**
**  ****Retrieve representations of experience**
**  ****Apply experience in meaningful ways to recognize the solution to current problems**
**  ****Apply experience as a basis for recalling knowledge relevant to current circumstances**
**  ****Communicate on the basis of shared experience**
**  ****Equally applicable to synthetic human and intelligent machine applications**

## Life Experience Generator: Machine Cognitive Model

- Source of Smoke

- Alarm Coming from Pasages

- Detect Smoke

- Detect Smoke

- Smoke Thickens

- Smoke Thickens

- Smoke Lessens

- Smoke Lessens

- Smoke Lessens

- Robot 1

- Robot 2

**Employ Umbra simulation environment to generate a range of representative experiences for robotic entities**

## Semantic Activation Across Time

**For a duration of time, there is a sequential progression in the patterns of concept activation**

**Smoke**

**Passage**

**Hallway**

**Smoke**

**Passage**

**Hallway**

**Smoke**

**Passage**

**Hallway**

**Progression of Time**

**t****1**

**t****2**

**t****3**

**t****4**

## Recognition of “Schema” based on Patterns of Semantic Activation

## Episodic Representation Based on Recurrent Sequences of Schema (i.e., Thremes)

## Statistical Derivation Schema & Themes

**Use high-dimensional data vector of robot status (each semantic node represented as vector)**
**Construct representative training set of data**
**Use unsupervised learning of training set to develop clusters (each cluster representing general robot state)**
**Use supervised learning (classification trees) to interpret clusters**
**Analyze cluster space: by robot, by run, and temporally**
**Transitions (in time) from cluster to cluster define schema**
**Consistently occurring sequences of schema define themes **

## Neural Nets for Recognition of Schema and Themes

- Semantic Knowledge

- Situational Knowledge
-     - Situation A
-     - Situation B
-     - Situation C
-     -
-     -

- Cues and Knowledge of Ongoing Events

**Schema Recognition**

**Theme Recognition**

**Interpretation Situation**

**Episodic Memory Record**

**Notes:**
Information regarding schema and themes leads to a ‘genetic sequence’ of weights, connections, and response functions that are competitively optimized in a genetic algorithm cycle.

INTELLIGENT ALGORITHMS
Before proceeding it is useful to describe the components of the intelligent algorithms used in data analysis.  Genetic algorithms and neural networks are both based on biological models and have been used successfully in a variety of data analysis applications32,33.   

Neural Networks
Neural Networks are based on a simplified model of the mammalian brain34.  In this model, neurons, which consist of a stoma (base) and numerous synapses (connections to other stoma), are represented by neurodes and their interconnections.  Each neurode receives multiple inputs and gives a single output.  The individual input signals are each assigned a relative weight and then summed to give the total weighted input. Individual inputs can either excite the neurode to fire or inhibit the neurode from firing.  Excitatory inputs are positive, while inhibitory inputs are negative.  The output signal from the neurode is computed as the result of a transfer function applied to the weighted sum of the inputs.  If the output is above a certain threshold the neurode fires, sending an input to each neurode in the following layer.  The transfer function can take many forms: step functions, ramp functions, sigmoid functions, etc.  
The structure of a NN is defined by the way in which neurodes and connections are arranged. The neural networks used in this work are comprised of three layers: an input layer containing a neurode for each piece of information passed into the NN, a middle layer, and an output layer. The middle layer is called a “hidden layer” since it has no direct connections to the outside world.  We use a fully-connected feed-forward architecture, meaning that each neurode in the input layer is connected to each neurode in the hidden layer, and each neurode in the hidden layer is connected to each neurode in the output layer.  No neurodes are allowed to send signals to neurodes in preceding layers or in the same layer.
Traditionally, neural networks learn to recognize and interpret patterns through a process of supervised training.  The connections between neurodes in a NN are initialized with random weights.  The NN is then shown many training sets, each consisting of  a set of inputs and expected outputs.  After a training set is shown to the NN the actual and expected outputs are compared.  The difference between the expected and actual output is calculated for each output neurode, and the weights of the connections to that neurode are adjusted to give the correct output.  This training method is called back-propagation of errors.  In this work we do not use back-propagation to train NNs.  Instead we use a genetic algorithm (GA) to train a population of NNs. 

Genetic Algorithms 
Genetic algorithms are global optimization algorithms based on mathematical models of the genetic process35. GA’s use the principles of natural selection and “survival of the fittest” to drive the evolution of a population of NNs.  Rather than starting the search for an optimum from a single point in the search space (as is done in a traditional optimization method), the GA starts with a statistical population of NNs.  Each member of the population is initialized with a random set of weights on its connections, scattering the members of the population randomly throughout the search space. 
Each individual neurode in the NN is represented by a set of chromosomes describing its’ features.  In our work each chromosome consists of a list containing the identity of a neurode, the connections between that neurode and each of the neurodes in the following layer, and the weights of those connections.  Reproduction (or “crossover”) occurs through a process of selection and recombination of chromosomes.  When two parents have been selected, a member of the next generation is created by taking part of each parents’ list of connections and weights and combining them to make an offspring.  The number of crossover points is a parameter which can be varied, while the position of each crossover point is chosen randomly. Offspring may or may not have higher fitness values than their parents due to the random nature of the crossover processes.  
During a generation (an iteration of the GA) each NN in the population is shown a set of training cases and then ranked according to how well its’ responses compare to the expected responses.  For this work the expected response for bacteria is 1, while the expected response for non-bacteria is 0.  Individuals with higher rank are deemed to be more fit than individuals of lower rank.  Highly fit individuals have a greater probability of being selected to reproduce, while the least fit are rarely chosen and thus tend to die out. 
To ensure that the search space is thoroughly explored the GA must maintain genetic diversity.  This is done both through the process of selection and recombination and through mutation.  Mutation introduces changes into the population during reproduction by randomly changing the weights on one or more connections.  This effectively takes the new NN into a different part of the search space, thus reducing the chance that the population will converge to a local optimum.  
Better chromosomes tend to dominate as the selection/recombination/mutation process is repeated over many generations.  As a result, the overall fitness of the population increases and the most promising areas of the search space are explored. The GA continues producing and evaluating new generations of NNs until the population converges (all members of the population have the same fitness) or reaches some predetermined stopping criteria such as a maximum number of generations. Typically the process is more efficient for a larger population, and less susceptible to premature convergence.  There is a general power underlying the simplicity of this method: it is stochastic and does not depend on making the right initial choice to avoid being fooled by local optima. 

## Training Neural Nets for Recognition of Schema and Themes

**Inputs**

**Output Episode Recognition**

**Input layer**
**Hidden layer(s)**
**Output layer**

- Neural Net
- Model

**i****1**

**i****2**

**i****n**

**. . .**

**w****1**

**w****2**

**w****n**

***Response Function***

**Notes:**
Information regarding schema and themes leads to a ‘genetic sequence’ of weights, connections, and response functions that are competitively optimized in a genetic algorithm cycle.

INTELLIGENT ALGORITHMS
Before proceeding it is useful to describe the components of the intelligent algorithms used in data analysis.  Genetic algorithms and neural networks are both based on biological models and have been used successfully in a variety of data analysis applications32,33.   

Neural Networks
Neural Networks are based on a simplified model of the mammalian brain34.  In this model, neurons, which consist of a stoma (base) and numerous synapses (connections to other stoma), are represented by neurodes and their interconnections.  Each neurode receives multiple inputs and gives a single output.  The individual input signals are each assigned a relative weight and then summed to give the total weighted input. Individual inputs can either excite the neurode to fire or inhibit the neurode from firing.  Excitatory inputs are positive, while inhibitory inputs are negative.  The output signal from the neurode is computed as the result of a transfer function applied to the weighted sum of the inputs.  If the output is above a certain threshold the neurode fires, sending an input to each neurode in the following layer.  The transfer function can take many forms: step functions, ramp functions, sigmoid functions, etc.  
The structure of a NN is defined by the way in which neurodes and connections are arranged. The neural networks used in this work are comprised of three layers: an input layer containing a neurode for each piece of information passed into the NN, a middle layer, and an output layer. The middle layer is called a “hidden layer” since it has no direct connections to the outside world.  We use a fully-connected feed-forward architecture, meaning that each neurode in the input layer is connected to each neurode in the hidden layer, and each neurode in the hidden layer is connected to each neurode in the output layer.  No neurodes are allowed to send signals to neurodes in preceding layers or in the same layer.
Traditionally, neural networks learn to recognize and interpret patterns through a process of supervised training.  The connections between neurodes in a NN are initialized with random weights.  The NN is then shown many training sets, each consisting of  a set of inputs and expected outputs.  After a training set is shown to the NN the actual and expected outputs are compared.  The difference between the expected and actual output is calculated for each output neurode, and the weights of the connections to that neurode are adjusted to give the correct output.  This training method is called back-propagation of errors.  In this work we do not use back-propagation to train NNs.  Instead we use a genetic algorithm (GA) to train a population of NNs. 

Genetic Algorithms 
Genetic algorithms are global optimization algorithms based on mathematical models of the genetic process35. GA’s use the principles of natural selection and “survival of the fittest” to drive the evolution of a population of NNs.  Rather than starting the search for an optimum from a single point in the search space (as is done in a traditional optimization method), the GA starts with a statistical population of NNs.  Each member of the population is initialized with a random set of weights on its connections, scattering the members of the population randomly throughout the search space. 
Each individual neurode in the NN is represented by a set of chromosomes describing its’ features.  In our work each chromosome consists of a list containing the identity of a neurode, the connections between that neurode and each of the neurodes in the following layer, and the weights of those connections.  Reproduction (or “crossover”) occurs through a process of selection and recombination of chromosomes.  When two parents have been selected, a member of the next generation is created by taking part of each parents’ list of connections and weights and combining them to make an offspring.  The number of crossover points is a parameter which can be varied, while the position of each crossover point is chosen randomly. Offspring may or may not have higher fitness values than their parents due to the random nature of the crossover processes.  
During a generation (an iteration of the GA) each NN in the population is shown a set of training cases and then ranked according to how well its’ responses compare to the expected responses.  For this work the expected response for bacteria is 1, while the expected response for non-bacteria is 0.  Individuals with higher rank are deemed to be more fit than individuals of lower rank.  Highly fit individuals have a greater probability of being selected to reproduce, while the least fit are rarely chosen and thus tend to die out. 
To ensure that the search space is thoroughly explored the GA must maintain genetic diversity.  This is done both through the process of selection and recombination and through mutation.  Mutation introduces changes into the population during reproduction by randomly changing the weights on one or more connections.  This effectively takes the new NN into a different part of the search space, thus reducing the chance that the population will converge to a local optimum.  
Better chromosomes tend to dominate as the selection/recombination/mutation process is repeated over many generations.  As a result, the overall fitness of the population increases and the most promising areas of the search space are explored. The GA continues producing and evaluating new generations of NNs until the population converges (all members of the population have the same fitness) or reaches some predetermined stopping criteria such as a maximum number of generations. Typically the process is more efficient for a larger population, and less susceptible to premature convergence.  There is a general power underlying the simplicity of this method: it is stochastic and does not depend on making the right initial choice to avoid being fooled by local optima. 

## Perceptual Processes

**Exploring concept wherein a federation of perceptual agents provide input to perceptual synthesis leading to semantic activation - ****??? Concept Exploration**

**Sensor**

**Sensor**

**Data**

- Perceptual Agent

- Perceptual Agent

- Perceptual Agent

- Perceptual Synthesis

- Federation Perceptual Agents

_**Perceptual Processes**_

## C++
Semantic Association/Activation Network expected to be large in real applications (60-80 vs 1,000-10,000 nodes).
Real-time (or faster) interactivity desired.
Umbra framework for highly modular simulations and systems.
Extends data-flow model  to better support interacting agents and interaction phenomena.
Integration with Tcl/Tk for user interactivity.
Easy to interface to devices, C/C++ libraries.
MPI support for parallelism within Umbra, e.g system of interacting Human Emulators.
HLA support for simulation in the large and federation.

**Semantic Association/Activation Network expected to be large in real applications (60-80 vs 1,000-10,000 nodes).**
**Real-time (or faster) interactivity desired.**
**Umbra framework for highly modular simulations and systems.**
**Extends data-flow model  to better support interacting agents and interaction phenomena.**
**Integration with Tcl/Tk for user interactivity.**
**Easy to interface to devices, C/C++ libraries.**
**MPI support for parallelism within Umbra, e.g system of interacting Human Emulators.**
**HLA support for simulation in the large and federation.**

**Scalability, Interoperability**

## Population Models

**Exploring concepts in which populations of distinctly different cognitive models interact – **
**???  Generative Cognitive Models, Parallelization**

**Population**

## Details, Characterization and Validation

- Numerous assumptions underlie model, numerous parameters require characterization, and a methodology is required for validation
- -         ???  differential contribution of cues
- -         ???  additive and inhibitory influences of cues
- -         ???  redundancy of cues
- -         ???  differential saliency of cues
- -         ???  facilitory priming of cues
- -         ???  inhibitory priming of cues
- -         ???  differential sequencing of cues
- -         ???  top-down facilitation by preceding situation(s)
- -         ???  top-down inhibition by preceding situation(s)
- -         ???  situation release and situation nesting
**	**

## Development of Experimental Protocol for Validation

**Study 1:  College students will generate the specific instances of general situations with ratings of familiarity and frequency **
**Study 2:  College students will generate cues present when situation is occurring**
**	****	****Events**
**	****	****Knowledge states (expectations, beliefs, and goals)**
**	****	****Emotions**
**	****	****Actions **

|  |
| --- |

| General Problem or  Situation  | Specific Instance of Situation  |
| --- | --- |
| Car Problems  | For example, run out of gas,, flat tire, engine problem, etc. |
| Health problems  | For Example, heart attack, drunk, head injury, etc. |

## Conclusion

** ****Developed test case models to prove in-principle approach**
** ****Characterization and validation effort commencing with NIU collaborators**
** ****Emerging capability for simulations based on synthetic humans, although intelligent machine may be more immediate application (e.g., intelligent building based on embedded experts)**
** ****Exploring ideas associated with real-time knowledge elicitation**

## Additional Slides

## Outline

**Simulation Setup**
**Summary of Analysis Strategy**
**Data Vector**
**Cluster Analysis**
**Distribution of Robot States**

## Simulation Setup

**20 simulations using RMSEL**
**8 robots per simulation**
**Fire/smoke location varies across runs**
**Initial robot status (e.g., location) is consistent across simulations**
**Fire/smoke location influences robot behavior   **

## Robot Search Paths - Simulation#5

## Robot Search Paths - Simulation#18

## Analysis Strategy - Summary

**Construct high-dimensional data vector(continuous and binary dimensions) that describes robot status**
**Construct representative set of high dimensional data**
**reduced temporal resolution (1/20 sampling rate)**
**800 randomly chosen observations (10 per robot per 10 simulations)**
**Develop clusters using unsupervised learning**
**cluster represents general robot state**
**Interpret clusters using classification trees (supervised learning)**

## Analysis Summary (continued)

**Analyze state space**
**Over all runs/robots**
**By robot**
**By run**
**Temporally: By robot within run**

## High-Dimensional Data Vector

**1. Time (t)**
**2. X****t**
**3. Y****t**
******abs(********X****t)**
******abs(********y****t****)**
**6. Smoke****t**
**7. ********Smoke****t ****= Smoke****t****-Smoke****t-1**
**8. Smoke****t**** -  max(robot smoke)**
**9. Smoke****t ****- max(system smoke)**

**10. IS_Beacon**
**11. IS_Last**
**12. IS_Rover**
**13. RF_Hear_Beacon**
**14. RF_PING**
**15. STOP**

## Cluster Analysis

**Use representative data set**
**Unsupervised learning**
**Candidate cluster analysis methods**
**Partitioning methods: Kmeans, medoids**
**Agglomerative methods**
**Divisive methods**
**Example uses Kmeans method (5 clusters)**
**Ref: Finding Groups in Data: An Introduction to Cluster Analysis, L. Kaufman, P. J. Rousseeuw, Wiley, 1990. **** **

## Gross Interpretation of Clusters (Robot States) Via Classification Tree (KMEANS)

- 5

- 3

- 1

- 2

- 4

- Moving

- Stop

- SmallX

- Large X

- Away  from Robot’s
- Max Smoke

- Near Robot’s Max Smoke

- No RF_Ping

- RF_Ping

## Robot State DistributionOver all 20 runs and all 8 robots

## Robot State DistributionBy Robot ID

## Robot State DistributionBy Run (1:10)

## Robot State DistributionBy Run (11:20)

## Gross Interpretation of Clusters (Robot States) Via Classification Tree (Divisive “DIANA”)

- RF_Hear_Beacon OFF

- RF_Hear_Beacon ON

- Lo Smoke

- Hi Smoke

- Small X

- Large X

- Small Y

- Large Y

- Small X

- Large  X

- 1

- 2

- 6

- 5

- 3

- 4

## Robot State DistributionBy Robot ID

## Robot State DistributionBy Run (1:10)

## Robot State DistributionBy Run (11:20)

## Timeline for Sandia National Laboratories Cognitive Modeling and Simulation

_**Next Generation**_** **_**Security Simulation**_
**Concept for cognitive model of human naturalistic decision making**

**FY99**

**FY00**

**FY01**

_**Organic Model**_
**Systems model with human as source of organic properties**

**Comprehensive **_**Representation**_
**Merged cognitive and organic models to create prototype framework for comprehensive model**

_**Extensible Knowledge-**_
_**Based Agents**_
**Modeling domain knowledge and practical application of cognitive models**

_**Human-Like Episodic Memory**_
**Model knowledge derived through collective  life experiences**

_**Intermediate Layer**_
**Models for perceptual and action generation processes**

_**Cognitive-Driven Augmented Analyst**_
**Integrates cognitive architecture with data exploitation for next-level capability**

**History of Human Reliability and Vulnerability Analysis for High Consequence Systems**

## General Cognitive Architecture for Agents

**Mismatch**

**Sensor**

**Sensor**

**Data**

**Perceptual Agent**

**Perceptual Agent**

**Perceptual Agent**

**Situation/Contextual Knowledge**
**Situation**
**Situation**

**Comparator**

**Selective Attention**

**Action Generation**

**Drive Mechanism**

**Emotional Processes**

**Human Interface:**
**Visual, Auditory, Haptic, Kinesthetic, Etc.**

**Perceptual Synthesis**

**Federation Perceptual Agents**

**Perceptual Processes**

**Notes:**
World model is your blackboard.  Your representation of the world derived from sensory data.  You don’t have to process anymore sensory data unless something changes.

Comparator:  Situation has some prototype Goal-action sequence gets activated from recognizing a situation, including any intermediate states you might experience prior to the goal state.  When the comparator regognizes a mismatch along the way or at the end two things happen:  1) emotional response of surprise (which has an arousal component) and 2) selective attention (perceptual focus on mismatch).  This prompts a resolution of the mismatch.  This is not the right situation and it channels the perceptual processes in an way to resolve the discrepancy.

Nesting of situations:  clear house, entry, stack up
Nesting of situations is represented in the Contextual knowledge that feeds into the action generation.  (See vg.#    on procedural knowledge;  procedural knowledge is a highly form of situational knowledge)