v Emergency	medical	first	responders	and	firefighters	have	access	

to	substantial	amount	of	data	at	the	incident	scene:

• Observations	and	communications	with	the	center/other	responders

• Sensor	data	from	wearables,	mobile,	and	IoT devices

• Physiological	data	from	patient	monitors	and	medical	devices

v Challenges	in	Analysis	of	Emergency	Response	Data

v Making	sense	of	substantial	amount	of	data:

• Inaccuracies	and	incompleteness	of	manually	collected	data	

v Cognitive	overload:	

• Significant	human	cognitive	effort	required	for	manual	interpretation	and	

logging	of	data	at	the	incident	scene,	which	could	be	better	used	to	assess	

most	effective	response	actions.

v Resiliency:	

• Disconnections	due	to	poor	network	connectivity	or	equipment	faults	may	

compromise	first	responder	decision	making	and	response.

MOTIVATION

PROPOSED	SYSTEM

MAJOR	MILESTONES

Towards	Cognitive	Assistant	Systems	for	Emergency	Response

Homa	Alemzadeh,	Jack	Stankovic,	Ronald	Williams,	Sarah	Masud Preum

Electrical	and	Computer	Engineering	and	Computer	Science

School	of	Applied	Science	and	Engineering	(SEAS),	University	of	Virginia

v Get	the	right	data,	at	the	right	time,	to	the	right	person

• Real-time	collection	and	analysis	of	data	from	incident	scene	

• Aggregation	of	in-situ	and	public	data,	e.g.,	video,	audio,	social	media

• Providing	dynamic	data-driven	feedback	on	effective	response	actions

v Develop	a	cognitive	assistant	system	for	emergency	response

• Resilient	data	analytics

• Anytime	real-time	sensing	and	computing

OBJECTIVES

EXPECTED	IMPACT

Task1 Resilient	Data	Analytics	Algorithms	

T1.1 Develop	labeled	data	sets	of	EMS	incident	reports	and	a	target	

set	of	EMS	protocol	guidelines.

T1.2 Design	and	validation	of	a	framework	for	modeling	and	execution	

of	EMS	protocol	guidelines.

T1.3 Develop	speech-to-text	recognition	and	text-to-speech	features	

to	target	EMS	first	responder	tasks	and	context.

T1.4 Develop	the	lexicon	of	EMS	concepts	and	design	NLP	algorithms	

for	incident	narrative	summarization	and	conflict	detection.

T1.5 Integrate	the	data	analytics	components	and	evaluate	the	

performance	of	cognitive	assistant	software.

Task	2 Embedded	System	for	Real-time	Sensing	& Computing	

T2.1 Design	and	test	the	embedded	architecture	executing	real-time	

data	analytics	developed	in	Task	1.

T2.2 Design	the	monitoring	mechanisms	for	detection	of	failures	and	

dynamic	reconfiguration	of	computing	resources.

Task	3 Evaluation	of	System	Performance	and	Resiliency

T3.1 Develop	the	testbed	by	integrating	embedded	device	with	a	

private	cloud	platform	and	workload	and	failure	simulators.

T3.2 Evaluate	the	resiliency	and	performance	of	the	system	in	the	lab	

settings	using	the	testbed.

T3.3 Evaluate	the	prototype	system	in	simulated	EMS	training	

scenarios	involving	first	responders.

Resilient	Data	Analytics

v Incident	Data	Collection	and	Summarization	

• Speech	to	Text	Recognition

• Natural	Language Processing	(NLP)

• Event Filtering	and	Synchronization

v Cognitive	Inference	and	Assistance

• Modeling	Protocol	Guidelines	and	Knowledge	Sources

• Conflict	Resolution

• Text	to	Speech Reminders

Real-time	Sensing	and	Computing

v Embedded	Architecture	for Real-time	Data	Analytics	

v Dynamic	Reconfiguration	for	Resiliency

PUBLIC	SAFETY	PARTNERSHIPS

Thomas Jefferson 

EMS Council (TJEMS)

Office of Emergency 

Medical Services

North Garden 

Fire Department

v Facilitate	automated	collection	and	summarization	of	EMS	data

v Reduce	stress,	increase	situational	awareness	and	safety	

v Enhance	EMS	training	modules	and	environments

• Training	first	responders	in	remote	rural	areas	

• Identifying	most	effective	response	actions

• Assessing	performance	in	simulated	scenarios	and	real	operations

v Provide	a	powerful	bridge	to	support	response	to	rare	mass	

casualty	incidents	(MCI)

Historical	Data	

Analytics	and	Models

Emergency	site

Wearable	Interface

Cardiac	Monitor

ER	community

Protocols/

Guidelines

Real-time	Sensing	and	Computing

v

Voice	recordings

Voice	feedback/reminders

Patient	Vitals

Speech	

to	Text

Signal	

Processing

Filtering	and

Cognitive	Inference

Natural	Language	

Processing

Text	to	

Speech

Feature	

Extraction

Secured	Cloud

Data	from	

other	 first

responders

EMS

Node

EMS

Node

EMS

Node

Embedded	System

Responder	Vitals

Wearable	Monitors

Center	Updates

ER	Center

References

§ H. Alemzadeh and M. V. Devarakonda, “An NLP-based Cognitive System for Disease Status 

Identification in Electronic Health Records,” IEEE Int. Conf. on Biomedical and Health 

Informatics (EMBS BHI), Feb. 2017. 

§ A. Mondol, et al., “MedRem: An Interactive Medication Reminder and Tracking System on 

Wrist Devices,” Wireless Health, Oct. 2016. 

§ S. Preum, A. Mondol, M. Ma, H.Wang, and J. Stankovic, PreCluDe: Conflict Detection in 

Textual Health Advice, Percom 2017, March 2017.