R A I Risk Assessment Quality Improvement

CHECKLIST v1.3

More	states	than	ever	are	using	risk	assessment	to	guide	corrections	decision making	and	supervision	

practices.	Risk	assessment	is	a	valuable	tool,	but	it	requires	routine	maintenance	to	ensure	continued	

effectiveness.	The	Risk	Assessment	Quality	Improvement	(RAQI)	protocol was	developed	by	The	Council	of	

State	Governments	Justice	Center to	help	agencies	take	stock	of	their	risk and	needs assessment	practices	

and	determine	opportunities	to	further	their	quality	improvement	efforts. While	there	are	a	variety	of	risk	

assessment tools	used	by	criminal	justice	agencies (e.g.,	custody	classification	tools or pre-release	parole	

tools),	the	RAQI	protocol	was specifically developed	to	help	agencies	evaluate	their	use of	general	

criminogenic	risk	assessment	tools.

Terms	in gold are	defined	in	the	glossary	at	the	end	of	the	checklist.

SECTION 1: Design and Structure of the Assessment Tool

1.1	|	Which risk	assessment	tool are	you currently	using? Please	include	name	and	version	number,	if	

applicable.

1.2	|Does	the	tool	measure	static	and/or	dynamic	risk	factors?

Static?	 Yes													 No

Dynamic? Yes													 No

1.3	|	If	you	are using	an	externally	developed	tool	(either	proprietary	or	public	domain),	has	it	been	modified	

in	any	way	for	your	jurisdiction?	

Yes													 No

If	yes,	how?

1.4	|	Do	you	use a	scoring	guide that	has	been	tailored	for	your jurisdiction?

Yes													 No

If	yes,	when	was	it	last	updated?

If	yes, and	you	are	using a	proprietary	tool,	were	the	scoring	guide	changes	approved	by	the	owner/developer	

of	the	tool?

Yes													 No
2

1.5	|	Is	there	a	specific	person	or	entity within	your	agency	or	department	responsible	for	monitoring	

developer-recommended	modifications	to	the	tool	or	scoring	guide?

Yes													 No

If	yes,	how	frequently	does	this	person	or	entity check for	changes	in	scoring	guidance?

1.6	|	Who	conducts	your risk	assessments (e.g.,	counselors,	classification	staff,	institutional	parole	officers)?

If	the	tool	is	fully	automated and	does	not	require	staff	to	conduct	interviews,	please	indicate	“N/A.”

N/A

												

In	total,	how	many	staff	members	conduct	the	assessments?

1.7	|	When	are risk	assessments conducted (e.g.,	at	intake	to	prison, prior	to	parole	hearing,	within	30	days	

of	starting	probation) ?

1.8	|	Does	your	department	or	agency conduct	regular	re-assessments?

Yes													 No

If	yes,	what	triggers	re-assessments	(e.g.,	updated	annually,	redone	at	significant	life	events such	as	loss	of	a	

job,	etc.)?

1.9	|	Are	different	criminogenic	risk	assessment tools	being	used	within	your system?	For	example,	do	

correctional	institutions	use	one	tool,	and	probation	or	parole agencies	use another?		

Yes													 No Don’t	Know

1.10	|	How	is	risk	assessment	information	stored?	For	example,	do	you	keep	paper	files or	use	electronic	

records to	store	the	item-by-item	scoring	and	assessment results?
3

SECTION 2: Process and Uses of the Assessment Tool

2.1	|	Has	everyone	who	conducts	assessments completed	training	on	using	the	risk	assessment	tool?	If	the	

tool	is	fully	automated	and	does	not	require	staff	to	conduct	interviews,	please	indicate	“N/A.”

Yes													 No N/A													

2.2	|	Are	staff required	to participate	in refresher	trainings? If	the	tool	is	fully	automated	and	does	not	

require	staff	to	conduct	interviews,	please	indicate	“N/A.”

Yes													 No N/A			

If	yes,	when	was the	last	time	refresher	training was	conducted?	

How	frequently	does	the	refresher	training	occur?		

2.3	|	Is	individual risk	assessment	information	shared	among agencies	working	with	the person being	

assessed	(e.g.,	DOC,	probation,	parole,	service	providers,	etc.)?			

Yes													 No

If	yes,	what information is	typically	shared	(e.g.,	total	score,	risk	category,	primary	need	areas,	responses	to	

each	item)?

2.4	|	How	is	the	risk	assessment	information	being	used	in	your	department or	agency?	Case	assignment /

supervision	determination?	Workload	determination?	Case	planning?	Classification?	Determining	responses	

to	a	person’s	behavior	(i.e.,	sanctions	and	incentives)?

2.5	|	For	what	purposes	does your	agency	share	the	risk	assessment	results	with	other	agencies	working	with	

the	individual (check	all	that	apply)?	If	risk	assessment	results	are	not	shared	with	other	agencies,	please	

select	not	applicable.

Not	applicable

Pretrial	

Decisions Sentencing Parole	Decisions

Reentry	

Planning

Treatment	

Coordination

Prosecutors/public	defenders

Judges

Police
4

Community	behavioral	health	

providers

Parole	Board

Department	of	Corrections

Community	nonprofit

Other:

2.6 |	How	would	you	rate	your	department’s	or	agency’s buy-in	for	using	the	risk	assessment, with	1	

indicating	“very	little	buy-in (staff	generally	do	not	see	risk	assessment	as	adding	value),”	3	indicating	

“neutral (staff	conduct	risk	assessments but	do	not	use	results to	inform activities	such	as	case	planning),”	

and	5	indicating	“full	staff	buy-in (staff	view	risk	assessment as	necessary	to	best	practices)”?		

1													 2													 3													 4													 5

2.7 |	Is	there	a	person	or	entity responsible	for	conducting	quality	improvement	activities	for	risk	assessment	

(e.g.,	reviewing assessments	for	scoring	accuracy,	conducting	inter-rater	reliability	exercises,	providing	

coaching	to	improve	assessor’s	skills)?

Yes													 No

Has	this person	or	entity been	trained	to	conduct	quality	improvement	activities?		

2.8 |	Does	the	quality	improvement	person	or	entity	review	risk	assessments	to	ensure	scoring	accuracy?		If	

the	tool	is	fully	automated	and	does	not	require	staff	to	conduct	interviews,	please	indicate	“N/A” and	skip	to	

Section	III.

Yes													 No N/A			

If	yes,	how	are	reviews	conducted (e.g.,	by	auditing recorded	assessment	interviews,	reviewing assessor’s	

notes	and	scoring)?	

How	frequently	do	the	reviews	occur?

2.9 |	Are	the	reviewed	samples	representative	of	the	people	being assessed	(e.g.,	people	who	are

incarcerated or on community	supervision)?
5

2.10 |	What	steps	does	your	department	or	agency take	to	improve	the	assessors’	skills?		

2.11 |	Does	your	department	or	agency regularly	conduct	inter-rater	reliability exercises,	where	staff	work	to	

improve	consistency	among	those	conducting	and	scoring	assessments?	

Yes													 No

If	yes,	how	frequently	does	this	occur?	

2.12 |	Does	your	department	or	agency regularly	analyze	data	to	monitor	the	quality	of	risk	assessment

administration? Please	note,	this	question	is	trying	to	assess	the	process by	which	assessments	are	

completed	and	how	completely	they	are	conducted as	opposed	to	testing	the	accuracy	of	the	tool.

Yes													 No

If	yes,	please	indicate	which	items you	routinely	analyze:

The	total	number	of	assessments	conducted

The	average	assessment	score	and	distribution	of	scores

The most	commonly	omitted	items	and	frequency	of	omissions

The	number of	and	reasons	for	overrides;	and	the	overall	risk/needs	profile	of	your	population)

Other,	please	specify	

2.13 |	Does	your	department	or	agency have	a	written	policy guiding	the	use	of overrides of the	assessment	

tool	(i.e.,	when	an	assessment	indicates	one	risk	level	for	an	individual	but	the	assessor	chooses	to	manage	

that	case	according	to	a	different	risk	level	based	on	aggravating	or	mitigating	circumstances)?	

Yes													 No

Does	the	override	process	require	supervisor	approval?

Yes													 No

2.13 |	Are	high-quality	assessments	used	to	model	appropriate	techniques	for	other	staff?		

Yes													 No Don’t	Know
6

SECTION 3: Validation and Statistical Properties of the Assessment Tool

3.1 | Has	a	validation	study been	conducted	for	your risk	assessment	tool?		

Yes													 No

If	yes,	when	was	the	most	recent	validation	study completed?		

3.2 | Did	the	results of	your	validation	study indicate	that	the	assessment	instrument	was	predictive	of	

recidivism	in	your	jurisdiction?

Yes													 No Don’t	Know									

3.3 | Did	the	validation	study	demonstrate	predictive	accuracy	by	race	and	by	gender?

Yes													 No									 Don’t	Know									 Study	didn’t	examine	validation	by	race	or	gender													

3.4	|	If	a	validation	study	has	been	conducted,	did	the	evaluator use	a	measure for	recidivism	that	is	

comparable	to	what	is	commonly reported	nationally	(e.g.,	re-arrest,	reconviction,	or	re-incarceration)?		

Yes													 No Don’t	Know		

3.5 |	Did	your	validation	study	consider	multiple	measures	of	recidivism?		

Yes													 No Don’t	Know		

If	yes,	were	the multiple measures	of	recidivism	considered	together	or	separately?		

Yes													 No Don’t	Know		

3.6 |	Does	the	jurisdiction	use normative	risk	categories (i.e.,	the categories	that	were	established	in	the	

development	of	the	tool)	or	site-specific	risk	categories (i.e.,	categories	that	were	established	based	on	a	

study	of	your	local	population)?

Normative	risk	categories													 Site-specific	risk	categories

If the	categories	are site-specific,	are	the	resultant	risk	categories	(e.g.,	very	high,	high,	moderate,	low,	very	

low)	based	on	site-specific	recidivism	rates?

Yes													 No

3.7 |	What	is	the	current	risk	distribution	of	your	population	(i.e.,	percentage in	each	risk	category)?	If	your	

agency	only	uses	3	risk	categories,	please	complete	boxes	1,	3,	and	5.

Risk	level	1	(lowest	risk) 						
7

Risk level	2 						

Risk	level	3	(average	risk)						

Risk	level	4 						

Risk	level	5	(highest	risk)						

If	your	agency	uses	more	than	5	risk	categories,	please	describe	below.	

3.8 |	Are	any	of	the	risk	factors	on	the	tool	“weighted”	based	on reasons	not	derived	from	empirical	analysis	

(e.g.,	policy	decisions or	resource	limitations)?

Yes													 No Don’t	Know		
8

Glossary of Terms

Actuarial

Assessor

Cut-off	scores

Criminogenic

Dynamic	risk	

factors

Inter-rater	

reliability

Normative	risk	

categories

Overrides

Proprietary	

tools

Public	domain

tools

Quality	

improvement	

activities

Risk	assessment	

tool

The	actuarial	approach	represents	a	statistical	model	of	estimating	the	risk	of	an	event’s	

occurrence	(e.g.,	the	risk	of	an	individual	committing	a	new	offense).

Line or	direct	services	staff	who administer the	risk	assessment	tool	by	conducting	the	

interview and	using the	scoring	guide	to	place	the	person in	a	risk	category.	Assessors	may	

be	probation	officers,	parole	officers,	intake	services	staff,	etc.	

The	points	in	the	distribution of	your	population’s	risk	scores at	which	one	risk	category

ends	and	another	begins.	Cut-off	scores are	sometimes tailored	to	a	particular	jurisdiction.	

Cut-off	scores	are	meant	to	produce meaningful	differences	in	risk	categories.

Causing	or	likely	to	cause	criminal	behavior.	Common	criminogenic	needs	include	antisocial	

attitudes,	antisocial	associates,	and	antisocial	behavior.

Factors	that	contribute	to	risk of	recidivism but	can	change	over	time	(e.g.,	social	networks,	

thinking	patterns,	housing,	substance	use,	finances,	etc.).	Dynamic	risk	factors	not	only	add	

to	the	predictive	ability	of	an	assessment	tool,	they	represent	those	areas	that	can	be	

changed	through	programming	and	interventions.	

The	degree	to	which	an	assessor	scores	an	assessment	consistently	and	the	degree	of	

scoring	consistency	between	different	assessors	when	assessing	the	same	person.	

Assessments	should	be	consistently	scored	the	same	way	regardless	of	who conducts	the	

assessment.

The	original	risk	categories	created	during	the	development	of	a	tool.	Externally	developed	

tools generally	come	with	guidance	about	the	risk	categories	from	the	normative	group.	

These	are	often	adopted	by	an	agency	until	a	local	validation	effort	is	conducted.

An	override	is	an	assessor’s	decision to	increase	or	decrease	a person’s	risk	category and	

supervision	and	programming recommendations	based	on	some	extenuating	information

(e.g.,	departmental	policy	requires	people	who	have	committed certain offenses	to be	

supervised	at	a	high level	of	intensity	for	an	initial	post-incarceration	period).	

Some	screening	and	assessment	tools	are	owned	by	business	entities	(e.g.,	the	Level	of	

Service	tools	are	owned	by	Multi-Health	Systems),	requiring	users	of	the	tool	to	purchase	

the	instrument.	

Some tools	are	considered	to	be	“in	the	public	domain”	(e.g.,	the	ORAS),	which	means	they	

are	not	copyrighted,	although	most	still	require	paid	training	prior	to	use.

Also	referred to	as	“quality	assurance,”	these	are	evaluations of	the	risk	assessment	

processes,	procedures,	and	scoring	to	see	whether	the	tool	is	being	used	correctly	and	

meeting	specified	requirements and	best	practices.	Audits	are	commonly conducted	

through	reviewing	files,	tapes	of	assessments,	and	direct	observation.	

Sometimes	referred	to	as	simply	“risk	tool,”	this	is	an instrument	used	to	predict	the	

likelihood	of	criminal	behavior	(often	defined	in	terms	of	reoffending).	Assessment	refers	to	

the	full	set	of	questions	that	are	contained	in	an	instrument	and	generally	surveys	more	

topics	than	a	screening tool.	Risk	tools do	not predict	the	behavior	of	specific	individuals,	

but	rather	the	probability	of	reoffending	for	people	who	share	certain	characteristics.	
9

Scoring	guide

Site-specific	risk	

categories

Static	risk	

factors

Validation study

Instructions	for	how	each	question	in	a	risk	assessment	should	be	scored	based	on	the	

information	collected.	The	scoring	guides	of	proprietary	and	public	domain	risk	assessments	

are	regularly	reviewed	and	updated	by	their	creators	to	reflect	the	latest	research	and	

practices	in	the	field.

Risk	categories	(e.g.,	low,	moderate,	high)	that	are	created	as	a	result	of	a	local	validation	

study	and	reflect	the	characteristics	of	the	local	population.	

Risk	factors	that	are	unchanging	(e.g.,	gender,	criminal	history)	or	not	subject	to	change	as	a	

result	of	treatment	interventions.	For	example,	age	of	first	arrest	is	typically	one	of	the	

factors	predictive	of	future	criminal	behavior, and	undergoing	an	intervention	cannot	

change	it.

A	validation	study	(and later	revalidation)	entails	examining	the	scores	of	a	risk	assessment	

in	relation	to	the	verified	criminal	risk	of	those	who	have	taken	the	assessment	(criminal	risk	

is	related	to	recidivism	and	may	be	defined	as	reoffending,	rearrest,	reincarceration,	etc.)	to	

ensure	that	risk	scores	are	accurately	predicting	risk.	

The	RAQI	protocol	will	help	you	structure	the	review	of	your	department’s	or	agency’s	current	risk	

assessment	practices,	and	The	Council	of	State	Governments	Justice	Center	can	provide	expert	guidance	

around	how	best	to	interpret	the	results	and	translate	them	into	system	improvements.	For	additional	

information,	please	contact The	Council	of	State	Governments	Justice	Center	at	raqi@csgjusticecenter.org.

This	project	was	supported	by	Grant	No.	2013-ZB-BX-K002 awarded	by	the	Bureau	of	Justice	Assistance.	The	

Bureau	of	Justice	Assistance	is	a	component	of	the	Department	of	Justice’s	Office of	Justice	Programs,	which	also	

includes	the	Bureau	of	Justice	Statistics,	the	National	Institute	of	Justice,	the	Office	of	Juvenile	Justice	and	

Delinquency	Prevention,	the	Office for	Victims	of	Crime,	and	the	Office	of	Sex	Offender	Sentencing,	Monitoring,	

Apprehending,	Registering,	and	Tracking.	Points	of	view	or	opinions	in	this	document	are	those	of	the	author	and	

do	not	necessarily	represent	the	official	position	or	policies	of	the	U.S.	Department	of	Justice.	To	learn	more	about	

the	Bureau	of	Justice	Assistance,	please	visit	bja.gov.	

In	partnership	with:

The	Council	of	State	Governments	Justice	Center	is	a	

nonprofit	organization	delivering	nonpartisan	advice	and

data-driven	strategies	for	safer	communities.	The	CSG	Justice	

Center’s	Justice	Reinvestment	Initiative	aims	to	improve	

public	safety	while	reducing	corrections	spending.