# Presentation: 134405

## Trade-offs between automatic postural adjustments and orienting responses as indices of cognitive engagement

**Carey D. Balaban**
**Departments of Otolaryngology, Neurobiology and Communication Sciences & Disorders**
**University of Pittsburgh**

## Postural Control and Cognitive States

**Multimodal: integration of visual, vestibular, proprioceptive, haptic and auditory information**
**Automatic **
**Context-dependent**
**Predictive or reactive**
**Reflects trade-off between voluntary movement and postural maintenance**

##  Real-time Dynamic Analysis of Seated Posture 

**Detect voluntary movements**
**Task-related**
**Spontaneous (including ‘fidgeting’)**
**Identify automatic postural responses**
**Reactive (vehicle, self or substrate mov’t)**
**Predictive (vehicle, self or substrate mov’t) **
**‘****Background’ activity**

## Dynamic Postural Assessment Chair

**Computer Workstation Environment**
**16 X 16 arrays of capacitive pressure sensors in seat and seat back**
**1 inch sensor spacing**
**Sampling rate: 4.5 Hz**
**Ultrasonic head tracker**
**6 degrees of freedom (x,y,z,roll angle, pitch angle and yaw angle)**
**Sampling rate: 50 Hz**
**Referenced to computer monitor**
**Motion sensor (to be added)**

## Dynamic Postural Assessment Chair

**DCC S-class Mercedes**
**16 X 16 arrays of capacitive pressure sensors in seat and seat back**
**1 inch sensor spacing**
**Sampling rate: 4.5 Hz**
**Ultrasonic head tracker**
**6 degrees of freedom (x,y,z,roll angle, pitch angle and yaw angle)**
**Sampling rate: 50 Hz**
**Transmitter mounted on ceiling above driver**

## Dynamic Postural Assessment Chair Patent Application Serial No. PCT/US04/14158

## Dynamic Postural Assessment ChairPatent Application Serial No. PCT/US04/14158

## Dynamic Postural Assessment Chair Patent Application Serial No. PCT/US04/14158

## Approach

**Dynamic rather than static analysis of posture**
**Use normal automatic behavioral templates**
**Postural sway (e.g., spontaneous anteroposterior and lateral weight shifts in seat)**
**Anticipatory shifts (e.g., leaning prior to turns or accelerating of a car)**
**Cognitive engagement responses (e.g., orienting movements relative to computer monitor)**
**Integrate sensors with Cognitive Models:  Active contextual gauges**

## Sensors versus Gauges

**Sensors **
**Single modality**
**Concrete ‘read-out’**
**Ambiguous without context**
**Gauges**
**Integrated from multiple sensors**
**Requires context to infer cognitive state**
**Can vary in degree of abstraction**

## Gauge Development: Stage 1

**Identify automatic postural responses during task**
**Static platform**
**Warship Commander Task environment**
**Balaban et al., Int. J. Human-Computer Interaction, 17 (2), in press.**
**Transferred to LM-ATL Aegis platform **

## Seat Sensor Example:  TH1

- Pseudocolor of instantaneous pressure

## Engagement Response: Head position and seat center of pressure (COP) a linear function of number of tracks 

## Engagement Response: Head position

## Epochs (8 sec duration) where at least 50% of variance (r≥0.71) explained by linear relationship between AP head position and number of tracks or by left (blue dots) or right (red circles) seat COP.

**Epochs (8 sec duration) where at least 50% of variance (r****≥0.71) explained by linear relationship between AP head position and number of tracks or by left (blue dots) or right (red circles) seat COP.**
**Head and body engagement are partially independent**

## Back Bracing Gauge: SD of dP/dt across Back Pad

## Transfer to LMATL Aegis Task

**For WCT the waves are:**
**Cued**
**Dominant feature of display**
**Rapid changes in number of tracks (mean ******** 2 sec)**
**For LMATL Aegis Environment the waves are:**
**Uncued (from operator’s perspective)**
**Embedded in a more complex display**
**Slow change in number of tracks (mean ******** 12 sec)**

## AugCog December LM CVE

**Postural engagement gauge**
**Detected linear association between head position or seat COP (center of pressure) movement (re: screen) and the number of tracks**
**Time window: 12 sec (mean & median target change duration)**
**Criterion for engagement: coefficient of determination = 0.50 (explains 50% of variance**)
**Calculated the percentage of sub-sampled time points (at 2 Hz) exceeding criterion**

## Engagement Gauge Predicts Next User Action

- •Overall engagement gauge: Maximum of head and seat COP values
- •Example: episodic engagement during scenario (upper left).
- •Predicts with high probability user action within next 5 seconds (upper right).
- •Distribution by type of next action (lower left) and hazard plots (lower right).

## Engagement Gauge  Predicts Next User Action

**Engagement gauge value >0.5 indicates that 50% of COP or head motion relative to computer monitor is explained by the number of targets**
**Positive engagement value predicts user action with high probability and reliability across subjects and scenarios**
**Hook, Select or ID within 2 seconds:  median p=0.616**
**Hook, Select or ID within 3 seconds:  median p=0.766**
**Hook, Select or ID within 4 seconds:  median p=0.860**
**Hook, Select or ID within 5 seconds:  median p=0.946**

## AugCog DCC CVE

**Seat sensors and head tracker in Mercedes S500**
**Road driving conditions**
**Autobahn**
**Two lane highway**
**Urban **
**Interactive gauge development and cognitive (contextual) modeling with Sandia National Laboratories**

## Chair Signals for Contextual Modeling

**Statistical properties of time derivatives of seat and back pressure from left and right sides of body**
**Body COP torsion (yaw) gauges**
**Angle of body in seat from left and right body centers of pressure/derivatives of centers of pressure**
**Linear translation of global COP**
**Correlation between seat and back velocity RMS in  a time window**
**Head yaw velocity and RMS(time window)**

## Seat Sensor Example:  TH1

- Pseudocolor of instantaneous pressure

## Left

**Left**

**Right**

**Back**

**Front**

## New Algorithms

**Stability gauges**
**Based upon statistical properties of time derivatives of seat and back pressure from left and right sides of body**
**Body torsion (yaw) gauges**
**Angle of body in seat from left and right body centers of pressure/derivatives of centers of pressure**
**Linear translation from global center of pressure**
**Correlation of head yaw and body yaw**

## Body Torsion (Yaw) Gauge

- Angle of body in seat from left and right body centers of pressure/derivatives of centers of pressure

**Seat back**

**Left**
**COP**

**Right**
**COP**

****

## Seat Yaw re: Left A-P COP

## Seat Yaw re: Right A-P COP

## Seat COP ‘Yaw” 

- Multiple regression analysis: 98% VAF (r2) by front-back COP shift on right and left seat
- Seat yaw angle = -0.1733 (±0.0006)* left COP + .1676 (±0.001)*right COP – 0.0566(±0.0098)
- No significant contribution of lateral COP

## Seat Yaw: Deterministic and Residual

## Temporal Relationships

## Head and Body Yaw: Example in Traffic and Lane Change

**1100**

**1110**

**1120**

**1130**

**1140**

**1150**

**1160**

**1170**

**1180**

**-0.5**

**0**

**0.5**

**1**

**Time (sec)**

**Yaw (radians re: leftward positive)**

- Seat

- Head

- Prepare &

-   Change

-     Lanes

- 1155-1161

- Overtaken

- repeatedly

-  by traffic

- 1100-1136

- Looking toward

-     blind spot

-    Execute

- Lane Change

-   1158-1161

## 3600: Emerge from tunnel in right lane on Autobahn (2 lanes)

**3606: Overtaken (car);  3644: Left lane to pass RV; 3651: To right;**
**3667: Left lane, Pass car/trailer (3676);  3680: Right lane **

***TS1***

## 3600: Emerge from tunnel in right lane on Autobahn (2 lanes)

**3606: Overtaken (car);  3644: Left lane to pass RV; 3651: To right;**
**3667: Left lane, Pass car/trailer (3676);  3680: Right lane **

***TS1***

## 6772: Merge on Autobahn behind truck, overtaken by cars

**6772: Merge on Autobahn behind truck, overtaken by cars**
**6777: To center lane to pass truck; pass truck at 6780**
**6800:  Pass car; 6802: Move to left lane; 6812: Pass car**

## In left lane on Autobahn

**Pass car at 6831, 6845, 6851, 6860, 6878, **
**6883, 6892, 6906, 6909, 6925, 6932**

## In left lane on Autobahn, Tunnel at 6949-53; Light traffic

**To center lane: 6960; Left: 6967; Center: 6980; Left: 6988**
**Pass car (6992), truck (6994), car (7000), car (7010)**
**To center lane: 7018**

## Postural Measurements in Driving

**Chair sensors detect orienting responses to vehicles, signs and situations requiring vigilance**
**Being overtaken by vehicles**
**Lane change scenarios**
**Passing large vehicles**
**Chair sensors detect relative quiescent periods with lower vigilance demands**

## Postural Measurements in Driving

**Need to optimize sensor-derived gauges with cognitive model**
**Considerations:**
**Different time scales (e.g., derivatives, correlation history, RMS history)**
**Quasi-independent measurements to facilitate optimization**
**Physiologically sensible and plausible**