Na#onal	Ins#tute	of		

Standards	&	Technology

Data	Science		Symposium	

March	4th	and	5th,	2014	

DATA	SCIENCE	PROGRAM	

Ashit	Talukder	

Chief,	Informa;on	Access	Division,	ITL,	NIST	

Email:	Ashit.Talukder@nist.gov		
Data	Science	Measurement	Challenge	in	the	Informa;on	Age	

• Data-to-Knowledge	Pathway	(and	associated	workflows)	is	key	component	

of		decision-making	and	discovery	in	the	informa;on	age	

• Technologies	to	extract	knowledge	from	data	increasingly	used	in	all	

segments	of	business,	science,	healthcare,	and	daily	living	

• Data	Science	enables	beNer	decisions,	but	approaches	oOen	lack	the	

necessary	measures	for	understanding:		

– the	quality	and	context	of	the	analyzed	data	

– the	rigor	of	the	analy;c	process	and	tools	employed	

– the	impact	of	the	human	in	the	analy;c	process	

– the	strength	of	the	conclusions	derived,	ques;ons	answered,	hypotheses	tested,	

and	discoveries	made	

• Since	data	science	is	driving	crucial	decision	making,	it	is	cri;cal	to	

understand	the	approaches,	measure	the	performance	of	the	underlying	

technologies,	and	to	correctly	interpret	the	final	output	

3/5/14	 2	
NIST	Data	Science	Interests	

• White	House	(OSTP)	announced	mul;agency	Big	Data	R&D	Ini;a;ve	

• NIST	has	strong	history	of	fostering	progress	in	analy;cs	and	data	

science	for	specific	data	modali;es	

– NIST	conducts	open,	community-based	evalua;ons	and	benchmarking	in	

specific	data	science	domains	involving	structured	and	unstructured	data	by	

dissemina;ng	reference	data,	hos;ng	challenge	problems,	and	designing	

metrics	

– Iden;fy	gaps	in	specific	data	science	applica;ons	through	strong	community	

par;cipa;on,	and	overcome	barriers	to	progress	by	having	community	solve	

common	problems	

• Text	search	and	informa;on	retrieval	(TREC,	TAC)	

• Speaker	ID,	Machine	Transla;on,		

• Video	Analy;cs	(TRECVID),	Image-based	biometrics	

• New	Data	Science	Program
Challenge	Problems	to	Advance	Specific	

Technology	

Ini;al	

Performance		

(High	Error)	

Improved	

Performance		

(Low	Error)	 APPROPRIATE		(common)	METRIC	

Annual	Evalua;ons																TIME																			Annual	Evalua;ons	

Evalua;ons	

focus	on	ini;al	

barriers	

New	barriers	

iden;fied	

More	

challenging	

barriers	

addressed	

Technology	

matures,	new	

challenges	

possible	

Usability	

Tests	

Applied	

to	new	

data	

types	

Engineering	

for	speed	

...	
Driving	Innova#ons	in	Data	Science	Through	

Measurements:	Impacts	

• Reference	Data,	metrics,	and	challenge	tasks	to	advance	

informa;on	retrieval	from	text–	TREC	conferences	since	1992	

TREC is an annual 

benchmarking 

exercise that has 

become a de facto 

standard in 

Information Retrieval 

evaluation. 

Stephen Robertson 

Microsoft 

SIGIR 2007 

Establishes 

the research 

methodology 

TREC has proven to be a 

valuable forum in which 

IBM Research has 

contributed to an 

improved understanding 

of search, while at the 

same time the insights 

obtained by participating 

in TREC have helped to 

improve IBM’s products 

and services. 

Alan Marwick, et al. 

IBM chapter of the TREC book 

2005 

Facilitates 

technology 

transfer 

In other words, for every 

$1 NIST and its partners 

invested in TREC, at 

least $3.35 to $5.07 in 

benefits accrued to IR 

researchers...These 

responses suggest that 

the benefits of TREC to 

both private and 

academic organizations 

go well beyond those 

quantified by this study’s 

economic benefits. 

RTI International 

Economic Impact Assessment 

of NIST’s TREC Program 

December 2010 

Amortizes the 

costs of 

infrastructure 

This project [the TREC 

Legal track] can be 

expected to identify both 

cost effective and reliable 

search and information 

retrieval methodologies 

and best practice 

recommendations, which, 

if adhered to, certainly 

would support an 

argument that the party 

employing them performed 

a reasonable ESI search, 

whether for privilege 

review or other purposes. 

Magistrate Judge Paul Grimm 

Victor Stanley v. Creative Pipe 

Forms/solidifies a 

research 

community 

TREC data revitalized 

research on information 

retrieval. Having a 

standard, widely 

available, and carefully 

constructed data set laid 

the groundwork for 

further innovation in the 

field. The TREC 

conference fostered 

collaboration, innovation, 

and a measured dose of 

competition that led to 

better information 

retrieval. 

Hal Varian 

Google Chief Economist 

March 4, 2008 

Improves the 

state of art 
Driving	Innova#ons	in	Data	Science	Through	

Measurements:	Impacts	

• Catalyze	advances	in	speech	technologies	(speech	

recogni;on,	transla;on,	and	voice	biometrics)	from	infancy	to	

mature	prototypes	through	new	metrics	and	challenge	tasks	

Progressive	Advancement	of	Speech	Recognition

Read	Speech

(Lab)

Broadcast	

Speech

Conversational		Speech Meeting	Speech
Driving	Innova#ons	in	Data	Science	Through	

Measurements:	Impacts	

• Comprehensive	image-based	NIST	biometrics	evalua;ons	

with	strong	stakeholder	par;cipa;on	advanced	biometrics	

field	and	led	to	biometrics	standards,	interoperability	and	

mature	biometrics	products	since	2001	

Improve	Face	Recognition	Capabilities		Through	Evaluations

FERET	1993

(Partially	Automatic)

Data	Sets

FERET	1996

(Fully	Automatic)

FRVT	2002

(Fully	Automatic)

FRVT	2006

(Fully	Automatic)

MBE	2010

(Fully	Automatic)

FRR	at	FAR	=	0.001

Year	of	Evaluation
NIST	Data	Science	Program	Vision	and	Goals	

Vision:		A	interdisciplinary	mul;-sector	Data	Science	program	focused	on	enabling	the	

understanding	and	evalua;on	of	data	science	approaches,	iden;fying	gaps,	and	

fostering	advancements	through	benchmarking	and	rigorous	measurement	science,	

reference	datasets,	and	evalua;ons	

Goals:	

q Work	with	Data	Science	community	to	address	technology	and	measurement	

barriers	and	characterize	uncertain;es	cri;cal	for	data-to-knowledge	workflows	

q Advance	Data	Science		capabili;es	and		address	R&D	to	resolve	gaps	through:	

1) advancement	of	rigorous	measurement	techniques	

2) development	of	reference	frameworks		

3) development	of	open	challenge	problems	on	use	cases	addressing	compelling	classes	

of	technology	challenges	

4) community	collabora;on	(engaging	stakeholders	from	ALL	sectors)	

q Help	all	stakeholders	understand	the	state-of-the-art	and	facilitate	collabora;on,	

interchange,	direct	comparison	of	approaches	on	common	problems	and	

reference	data	with	standard	metrics	 8	
Data	Science	Program	Expected	Impacts	

• End	Users	will	have	an	objec;ve	resource	to	understand	the	

technology	space	and	implementa;on	considera;ons	

• Industry	will	have	a	means	for	engaging	a	broad	R&D	

community	to	understand	the	state-of-the-art	and	best	

prac;ces	for	measuring	and	integra;ng	technology	

• Academia,	Researchers	&	System	Developers	will	have	

resources	and	methods	for	objec;vely	comparing	approaches,	

performing	scalable	research,	and	understanding	performance	

from	both	a	component-	and	system-level	perspec;ve	

Ø Accelerated	R&D	transi#on,	overcome	barriers	to	progress,	

reduced	transi#on	risk,	beMer	more	reliable	analy#c	

technology	 9	
Measurement	Science		

Advance	Data	Science	Through	Measurement	and	

Evalua#on	

Metrics	

Reference	

Frameworks	

Methods	

Measurements	for	

Data	Science	

Hardening	metrics,	

Iden=fying	cri=cal	gaps,	

and	advancing	state-ofthe-art	

•Novel	Metrics	

•Massive	annotated	data	

collec;ons	

•Novel	mul;disciplinary/	mul;-

sector	communi;es	of	interest	

•Forward-thinking	analy;c	

challenge	problems	and	open	

evalua;ons	

•genera;ng	cri;cal	R&D	mass	

•accelera;ng	development	of	

advanced	analy;c	

technologies	

•Reference	evalua;on	and	R&D	

architectures	and	algorithms	

•Best	prac;ces,	consensus	

standards,	generally	

recognized	de	facto	standard	

performance	tes;ng	methods	

and	data,	and	formal	standards	

•Analy;c	Technology		

interoperability	and	wellunderstood	state-of-the-art	

•Advance	capabili;es	in	NIST	

and	stakeholder	Data	Science	

use-cases	

• Exper;se	

• Data	Science	

Use	Cases	

• Data	Sharing	

• Formal	and	

Informal	

Collabora;ons	

• Outreach	

Events	

• Evalua;ons	

• Evalua;on	

Workshops	

Data	Science	

Stakeholders	 Data	and	

Ground	

Truth	

Data	Science	

Technologies	
NIST	Data	Science	Program	Phased	Approach	

• Common	Framework	for	data	science	workflows		

– Create	common	taxonomies	and	categoriza;on	for	data	analy;cs	

– Allows	stakeholders	across	mul;ple	domains	to	speak	common	language	

– Enable	mul;-sector	experts	to	iden;fy	common	solu;ons	&	approaches	

– Assist	in	iden;fica;on	of	cri;cal	gaps	and	hurdles	

• Adop#on	of	reference	datasets	that	will	be	reusable	and	

applicable	across	mul;ple	domains	

• Foster	cross-sector	collabora#ons	to	catalyze	advances	by	

iden;fying	solu;ons	in	one	sector	reusable	in	other	domains	

• Mul#-year	open	evalua#ons	involving	communi;es	of	experts	

using	a	phased	approach	and	consensus	process	

11	
NIST	Data	Science	Program	Phased	Approach	

• Common	Framework	for	data	science	workflows		

– Create	common	taxonomies	and	categoriza;on	for	data	analy;cs	

– Allows	stakeholders	across	mul;ple	domains	to	speak	common	language	

– Enable	mul;-sector	experts	to	iden;fy	common	solu;ons	&	approaches	

– Assist	in	iden;fica;on	of	cri;cal	gaps	and	hurdles	

• Adop#on	of	reference	datasets	that	will	be	reusable	and	

applicable	across	mul;ple	domains	

• Foster	cross-sector	collabora#ons	to	catalyze	advances	by	

iden;fying	solu;ons	in	one	sector	reusable	in	other	domains	

• Mul#-year	open	evalua#ons	involving	communi;es	of	experts	

using	a	phased	approach	and	consensus	process	

3/5/14	 Data	Science	Symposium	 12	
Example	Common	Framework	for	Data	Science	Workflows	

13	

Sources	

• Observa;onal	

• Transac;onal	

• Model	output	

• Metadata	

• Experimental	

...		

Representa#ons	

• Records	

• Sequences	

• Graphs	

• Spa;o-temporal	

• Natural	language	text	

• ...		

3/5/14	 Data	Science	Symposium	

Data	
Example	Common	Framework	for	Data	Science	Workflows	

14	

Objec#ves	

• Detec;on	

• Modeling	

• Transforma;on	

• Inference	

• Predic;on	

• ...		

3/5/14	 Data	Science	Symposium	

Analy;c	Components	

Approaches	

• Regression	

• Associa=on	rule	learning	

• ...	

Techniques	

• Support	vector	machines	

• Neural	networks	

• ...	
15	

3/5/14	 Data	Science	Symposium	

Data	Source	 Data	

Representa;ons	

Analy;c	

Workflow	

Knowledge	

Example	Common	Framework	for	Data	Science	Workflows	

Illustra;on	

Transac;onal	 Records	 Fraud	

Credit	Card	

Purchase	

• Time	

• Vendor	

• Loca;on	

• Amount	

Modeling	

Detec;on	

Decline	

Purchase;	

Contact	

Cardholder	

Detec;on	
3/5/14	 Data	Science	Symposium	 16	

Data	Source	 Data	

Representa;ons	

Analy;c	

Workflow	

Knowledge	

Example	Common	Framework	for	Data	Science	Workflows	

Illustra;on	

Observa;onal	 Text	&	

Records	

Pa;ent	is	

healthy	

Visit	to		

Doctor		

&	

Lab	

Transform	

Model	

Inference	

HandwriNen	

Notes	Scan	

&	

Lab	Results	

Model	

Detect	

Inference	

Inference	
3/5/14	 Data	Science	Symposium	 17	

Example	Common	Framework	for	Data	Science	Workflows	

Illustra;on	

• Opera;ons	

• Experiments	

• Transac;ons	

• Observa;ons	

• Model	outputs	

• Metadata,		

Ontology	

• ...	

• Machine	

Generated	

• Human	

Generated	

Detec#on Grouping Modelling Predic#on Inference Transform

Records

Sequences

Graphs

Spa#al/

Image

Temporal

Spa#o-temporal

/	Video

Text

Medical	

Diagnosis	

Support	

Medical	

Diagnosis	

Support	

Medical	

Diagnosis	

Support	

Medical	

Diagnosis	

Support	

Medical	

Diagnosis	

Support	

Medical	

Diagnosis	

Support	
NIST	Data	Science	Program	Phased	Approach	

• Common	Framework	for	data	science	workflows		

– Create	common	taxonomies	and	categoriza;on	for	data	analy;cs	

– Allows	stakeholders	across	mul;ple	domains	to	speak	common	language	

– Enable	mul;-sector	experts	to	iden;fy	common	solu;ons	&	approaches	

– Assist	in	iden;fica;on	of	cri;cal	gaps	and	hurdles	

• Adop#on	of	reference	datasets	that	will	be	reusable	and	

applicable	across	mul;ple	domains	

• Foster	cross-sector	collabora#ons	to	catalyze	advances	by	

iden;fying	solu;ons	in	one	sector	reusable	in	other	domains	

• Mul#-year	open	evalua#ons	involving	communi;es	of	experts	

using	a	phased	approach	and	consensus	process	

3/5/14	 Data	Science	Symposium	 18	
Data	Science	Program	

Measurement	&	Evalua;on	Elements	

3/5/14	 Data	Science	Symposium	 19	

Measurement	&	

Evalua#on	

Focused	Events	

Technical	

Discussion	

Groups	

Metrics	&	

Measurment	

Methods	

Use	Cases,	

Challenge	

Problems,	&	

Reference	

Datasets	

Tes#ng	

Frameworks	

Open	Evalua#ons	
Metrics	Development	

3/5/14	

• Create	general,	extensible	measures	of:	

– Scalability,	resource	u#liza#on	,	speed, and accuracy	of	analy;c	

components,	end-to-end	systems	with	and	without	humans	in	the	

loop	

– Propaga#on	of	error	and	uncertainty	through	the	system	

• Develop	evalua;on	strategies	that	explore

– Transferable	methods	and	complex	data	representa#ons	

(heterogeneous,	streams,	etc.)	

– A	variety	of	analy#c	objec#ves	(predic;on,	clustering,	tracking,	

search,	filtering,	etc.)	

– User	interfaces	and	usability	

Data	Science	Symposium	 20	
Use	Case	and	Reference	Data	Development	

• U;lize	common	framework	for	data	science	workflows	to	

build consensus	across	use	cases	around	abstract	data	and	

analy;c	categories	

• Iden#fy candidate	use	cases	for	challenge	problems,	metrics	

development,	R&D,	and	evalua;ons	

– Staged	approach,	star;ng	with	low-hanging	fruit,	overcoming	barriers,	

addressing	new	challenges	

• Make	available	reference	datasets	for	use	in	research	and	

evalua;on	that	will	be	reusable	and	applicable	across	mul;ple	

domains	

3/5/14	 Data	Science	Symposium	 21	
Tes;ng	Frameworks	to	Support		

Research,	Measurement,	and	Evalua;on		

Tes#ng	framework	to:	

• develop	and	implement	data	science	evalua;ons	using	a	

common	approach		

• foster	community-contributed	open	source	tools,	libraries,	

data,	compu;ng	resources,	and	experiments	

• assist	both	data	science	prac;;oners	and	applica;on	

developers	understand	performance characteris#cs	of	

different	implementa;ons	

3/5/14	 Data	Science	Symposium	 22	
Measurement	and	Evalua;on	Focused	Events	

Data	Science	Symposium	Series	will	be	an	annual	event,	with	

the	following	objec;ves:	

• Convene	a	broad	community	of	experts	to	discuss	the	data	science	

landscape,	direc;ons,	and	measurement	and	collabora;on	needs	

• Foster	collabora#on	and	resource	sharing	

• Report	on	data	science	evalua#on	results	

• Discuss	the	state	of	the	art	

• Expose	technical	gaps	and	challenges		

• Plan	and	priori;ze	future	challenge	evalua#ons	to	focus	research	

3/5/14	 Data	Science	Symposium	 23	
Data	Science	Evalua;on	Series	

• Open	challenge	evalua#on	series	with	several	analy#c	

challenge	tracks	addressing	a	variety	of	research	challenges	

– Tracks	chosen	to	address	common	problems	across	domains		

– Coordina;on	of	tracks	to	have	components	in	common,	allowing			

cross	track	research	

– Representa#ve	data	to	be	tested	against,	and	objec#ve	metrics	

– Evalua;ons	to	include	human	interac#on	with	data	as	applicable	

• Evalua;ons	results	and	research reports	will	be	presented	

during	an	annual	symposium	series	by	both	the	track	leaders	

and	par;cipants.			

3/5/14	 Data	Science	Symposium	 24	
Data	Science	Evalua;on	Series	

Outcomes

• Establish	evalua#on	framework,	enabling	research	

coordina;on	for	mutual	benefit	by	providing	common	

• Measure	progress	in	the	state	of	the	art	

• Iden#fy	technical	hurdles	and	engage	community	to	

overcome	hurdles	

• Harden	metrics and	methods	of	measurement

3/5/14	 Data	Science	Symposium	 25	

- Objec;ves	 - Metrics	

- Datasets	 - Measurement	Methods	
Data	Science	Evalua;on	Series	

Proposed	Timeline	&	Outputs	

3/5/14	 Data	Science	Symposium	 26	

Form	

Discussion	

Groups	

Run	Pilot	

Evalua;on	

(1-3	Tracks)	

Coordinate	

Community	

Driven	

Evalua;on	

Tes;ng	

Framework	

Available	

Now	 2014	 2015/	

2016	

2016/	

2017	

• Reference	

Datasets	

• Establish	

Baseline	

Performance	

• Coordinated	

Research	

Efforts	

• Measures	of	

the	state	of	

the	art	

• Resource	for	

Research	and	

Measurement	

• Common	

Framework	

Enabled	

Approaches	

• Common	

Framework	

for	Data	Sci.	

Workflows	

• Community	

Consensus	

Plans		
Way	Forward	

1) Form	Data	Science	evalua#on	planning	group	

2) Solicit	track	proposals	and	champions	

– Priori;ze	based	on	desiderata,	evalua;on	balance,	and	risk	

3) Solicit	ini;al	expression	of	intent	for	candidate	tracks		

4) Form	track	development	groups	and	track	implementa;on	plan	

– Develop	implementa;on	schedule	

– Iden;fy/create	track	data	resources	

– Develop	evalua;on	tools	

– Define	track	evalua;on	plan	(tasks,	data,	metrics,	methods)	

5) Deploy	track	development	resources	and	tools	

6) Conduct	dry	run	evalua#ons,	tes;ng	the	pipeline	

7) Obtain	commitments	to	par#cipate	

8) Conduct	formal	evalua#on	

9) Report	results	at	annual	symposium

3/5/14	 Data	Science	Symposium	 27	
How	to	Contribute	

Join	in	symposium	ac;vi;es	

– Help	create	the	common	framework	for	data	science	

workflows		

– ANend	the	working	lunches	and	contribute	to	the	breakout	

group	discussions	

– Share	your	ideas	

Engage	in	evalua;on	planning	

– Become	a	member	of	the	evalua;on	planning	group		

– Suggest	datasets,	challenge	problems,	metrics	

– Propose	or	champion	an	analy;c	research	track	

Par#cipate	in	evalua;ons

3/5/14	 Data	Science	Symposium	 28