Risk Adjustment for Cost 

Analyses: The Comparative 

Effectiveness of Two Systems

Todd Wagner, PhD, Anjali Upadhyay, MS, 

Theodore Stefos, PhD, Elizabeth Cowgill, MPH, 

Eileen Moran, MHA, Peter Almenoff, MD

June 19, 2013
Acknowledgements 

 Yu-Fang Li, PhD, RN

 Mei-Ling Shen, MS

 Steve Asch, MD, MPH

 Bruce Kinosian, MD

 Amy Rosen, PhD

 Maria Montez-Rath, PhD

2 
Outline 

 Introduction 

 Aim 1: How do the computed risk scores 

compare? 

 Aim 2: What is gained by recalibrating 

the risk models to fit VA? 

 Limitations

 Findings 

3 
What is risk adjustment?

 Statistical method to adjust for the 

observable differences between patients

– Classify patients into homogeneous clinical 

categories

– Calculate a single dimension risk score 

using clinical categories

4 
Why Risk Adjust?

 Limited research budget and limited time 

require the use of administrative data to 

– Comparative Effectiveness

– Improving the delivery of high value care

 Risk adjustment is necessary to address 

these questions 

5 
Risk adjustment at VA

 Used by operations and research to: 

– Assess medical center efficiency and 

productivity

– Heath services research (e.g., hospital 

readmissions)

 VA contracts with Verisk to obtain 

calculate risk scores for VA data 

6 
Verisk Versions

 Risk Smart algorithm create 184 

hierarchical condition categories (HCC) 

and risk scores

 Verisk is phasing out Risk Smart and 

moving to Risk Solutions,which creates 

394 HCCs and risk scores

 We focused on the latter, more recent 

version (Risk Solution)

7 
DxCG Risk Solutions

 Hereafter DxCG refers to Risk Solution 

model 

 Model produces 3 risk scores 

– Medicare prospective risk without Rx

– Medicare concurrent risk without Rx

– Medicaid prospective risk with Rx

8 
Operation Question

 Given the transition from Risk Smart to 

Risk Solutions, should VA continue to 

contract with Verisk? 

9 
Other Risk Adjustment Systems

 Charlson co-morbidity index

 CAN score

 Adjusted Clinical Groups (ACGs)

 Chronic Illness & Disability Payment 

System (CDPS)

 CMS Risk Adjustment Model (V21)

10 
CMS PACE Version 21

 Generates 189 HCCs 

 Produces 3 prospective risk scores 

– Community

– Institution

– New enrollee

 No concurrent risk score

11
Aims 

 How do the DxCG and V21 risk scores 

compare? 

 What is gained by adding variables and 

recalibrating the risk scores to fit VA? 

12 
How do the computed

 risk scores compare? 

Aim 1:
Six Study Samples

1. General sample 

2. High cost Veterans

3. Veterans with mental health/substance 

use disorder (MH-SUD)

4. Veterans over age 65 

5. Veterans with multi-morbidity

6. Healthy Veterans 

14 
Samples 

 General sample: 2 million randomly 

selected Veterans

 High Cost Users: most costly 5% VA 

users. Most costly was based on HERC 

national costs to remove geographic wage 

variation 

15 
Samples (cont)

 MH-SUD: All patients with a MH or 

SUD diagnosis in VA. We used 

diagnostic codes from MHO. 

 Over 65: Veterans >=65

 Multi-morbidity and healthy used the 

AHRQ body indicator

16 
Body System Indicator

1 = Infectious and parasitic disease 

2 = Neoplasms 

3 = Endocrine, nutritional, and metabolic diseases and immunity disorders 

4 = Diseases of blood and blood-forming organs 

5 = Mental disorders 

6 = Diseases of the nervous system and sense organs 

7 = Diseases of the circulatory system 

8 = Diseases of the respiratory system 

9 = Diseases of the digestive system 

10 = Diseases of the genitourinary system 

11 = Complications of pregnancy, childbirth, and the puerperium 

12 = Diseases of the skin and subcutaneous tissue 

13 = Diseases of the musculoskeletal system 

14 = Congenital anomalies 

15 = Certain conditions originating in the perinatal period 

16 = Symptoms, signs, and ill-defined conditions 

17 = Injury and poisoning 

18 = Factors influencing health status and contact with health services 

17 
Samples (cont)

 Multi-morbid: 2 or more Body System 

Indicators 

 Healthy: 

– not multi-morbid

– just one body system indicator

– Had a V code for a physical 

V70x,V71x,V72x

18 
Outcomes 

 Total costs in current year (FY10) and 

prospective year (FY11) 

– VA inpatient (DSS)

– VA outpatient (DSS)

– VA pharmacy (DSS)

– Fee Basis (purchased care)

19 
Data

 VA care (utilization, cost, diagnostic information)

– NPCD 

– PTF

– DSS

– HERC Average Cost data

 Non-VA care

– Fee Basis 

20 
General Over 65 High cost MH-SUD Multi-morbid Healthy

N 1,995,620 644,524 261,487 830,832 817,951 78,032

Age (SD) 62.0 (15.9) 81.4 (4.6) 62.5 (13.4) 56.9 (15.2) 62.2 (13.8) 48.2 (17.4)

Male 94% 98% 95% 91% 94% 86%

Total Costs*

 Mean 8,819 8,067 76,920 15,067 21,345 2,435

 Median 2,563 1,908 52,954 5,637 9,337 1,093

 SD 24,976 25,624 76,697 33,560 40,603 5,203

 Maximum 1,660,240 1,597,986 2,979,525 2,476,373 2,979,525 275,166

21 

Descriptive Statistics

*Total Costs include inpatient, outpatient, pharmacy and Fee Basis care 

- Veterans whose DSS Rx costs exceed $50,000 were excluded from these analyses. 

- Negative DSS VA in/out costs and DSS Rx costs were also replaced with zeros.

- High cost sample developed using HERC average costs, not DSS costs
Comparing Risk Scores: 

regression models

 Ordinary least squares (OLS)

 Log-OLS 

 Square-root OLS

 Generalized linear model (GLM) with gamma 

distribution and log-link

 GLM with gamma distribution and square root link

 Covariates: age, age-squared, gender

22 
CMS V21 DxCG Medicare DxCG Medicaid

Prospective Prospective Concurrent Prospective 

without Rx without Rx without Rx with Rx

General 0.756 (0.730) 0.661 (0.698) 0.497 (0.879) 1.756 (2.126)

Over 65 1.065 (0.750) 0.921 (0.656) 0.504 (0.874) 2.020 (2.267)

High cost 2.234 (1.580) 2.077 (1.628) 2.684 (2.243) 7.228 (4.323)

MH-SUD 0.893 (0.850) 0.802 (0.802) 0.770 (1.092) 2.487 (2.708)

Multi-morbid 1.160 (1.024) 1.044 (1.002) 1.004 (1.343) 3.146 (3.027)

Healthy 0.295 (0.234) 0.236 (0.219) 0.152 (0.244) 0.708 (0.708)

23 

Average Risk Scores

Mean (SD)
How do Risk Scores 

Fit the VA Data?

 R-squared 

 Root mean squared error

 Mean absolute error

 Hosmer-Lemeshow goodness of fit 

24 
R-squared 

CMS V21 DCG Medicare DCG Medicaid

Prospective 

without Rx

Prospective 

without Rx

Concurrent 

without Rx

Prospective 

with Rx

General 0.4287 0.4308 0.5122 0.5682

Over 65 0.4108 0.3876 0.4802 0.5907

MH-SUD 0.3985 0.4191 0.4876 0.5738

High cost 0.1920 0.1999 0.2650 0.3779

Multi-morbid 0.3910 0.3906 0.4790 0.5377

Healthy 0.1646 0.1966 0.2694 0.2701

25 

Results shown were from an Square root OLS model
Root Mean Squared Error

CMS V21 DCG Medicare DCG Medicaid

Prospective 

without Rx

Prospective 

without Rx

Concurrent 

without Rx

Prospective 

with Rx

General 20,576 21,829 22,060 17,884

Over 65 22,018 23,377 23,761 18,464

MH-SUD 27,942 29,215 28,865 23,895

High cost 70,312 70,003 67,206 62,716

Multi-morbid 34,035 35,043 33,708 29,888

Healthy 4,945 5,045 4,782 4,605

26 

Results shown were from an Square root OLS model
Mean Absolute Error

CMS V21 DCG Medicare DCG Medicaid

Prospective 

without Rx

Prospective 

without Rx

Concurrent 

without Rx

Prospective 

with Rx

General 7,415 7,423 6,783 6,398

Over 65 7,320 7,552 6,812 6,077

MH-SUD 11,843 11,607 10,774 9,942

High cost 41,640 41,266 39,120 36,720

Multi-morbid 15,225 15,236 13,868 13,234

Healthy 2,087 2,035 1,937 1,941

27 

Results shown were from an Square root OLS model
Hosmer-Lemeshow Tests

DCG Medicare DCG Medicaid

CMS V21 prospective concurrent prospective with 

prospective without Rx without Rx Rx

Deciles

General Sample

1 -1,651 -2,530 -2,561 -1,286

2 -1,464 -2,187 -2,455 -1,287

3 -1,432 -2,059 -2,300 -1,296

4 -1,431 -1,954 -2,035 -1,636

5 -1,495 -1,701 -1,542 -1,234

6 -1,415 -1,493 -867 -335

7 -730 -567 188 179

8 301 1,014 1,115 1,049

9 2,694 3,579 4,479 1,199

10 6,645 7,922 6,000 4,663

28 
Findings 

 Concurrent risk: 

– DxCG offers concurrent risk scores; CMS V21 does not

– Concurrent risk models tend to produce better fit statistics 

than prospective risk models

 Prospective risk: 

– DxCG and V21 produce similar results across a range of 

samples and regression specifications

 Prospective risk with pharmacy:

– DxCG offers better fit than V21, which does not include 

pharmacy

29 
What is gained by adding 

variables and recalibrating the 

risk models to fit VA? 

Aim 2
Recalibration 

 Added more covariates

– Race, Marital status, Other health insurance, 

Veteran priority level status

– Exposure registry (e.g., Agent Orange)

– 46 psychiatric condition categories (Rosen)

– Pharmacy

 Re-ran analytic models from Aim 1, and 

estimated new risk score for each patient 

31 
Pharmacy 

 Prior year’s pharmacy spending

 Any use of medication in 26 drug class 

categories 

32 
Drug Class

 PBM maintains an alphanumeric list of 

580 drug types within 29 drug classes. 

 Three classes were rarely used, resulting 

in the final list of 26

33 

Abbreviation Description

AM Antimicrobials

CN Central nervous system agents

DX Diagnostic agents

RS Rectal solutions

VT Vitamins
R-squared 

DxCG 

CMS V21 DxCG Medicare Medicaid

Prospective 

Prospective with VA drug Prospective Concurrent Prospective 

without Rx class indicators without Rx without Rx with Rx

General 0.5793 0.6924 0.5819 0.6274 0.6351

Over 65 0.5728 0.6772 0.5677 0.6233 0.6397

MH-SUD 0.5820 0.6810 0.5896 0.6268 0.6509

High cost 0.3559 0.4281 0.3544 0.4244 0.4241

Multi-morbid 0.5350 0.6331 0.5326 0.5957 0.5943

Healthy 0.2922 0.4573 0.3113 0.3508 0.3778

34 
Hosmer-Lemeshow 

35 

Recalibrated V21 DCG Medicare DCG Medicaid

Deciles prospective

prospective 

with VA drug 

class

prospective 

without Rx

concurrent 

without Rx

prospective 

with Rx

General Sample

1 -1,056 -650 -1,349 -1,244 -1,066

2 -1,050 -742 -1,379 -1,251 -1,149

3 -1,037 -623 -1,322 -1,282 -1,064

4 -944 -624 -1,210 -1,138 -950

5 -796 -670 -1,017 -981 -755

6 -570 -686 -678 -659 -343

7 -312 -746 -160 -71 96

8 12 -740 468 590 612

9 693 -211 1,883 2,149 659

10 5,072 5,707 4,776 3,897 3,968
Predicting Risk

 Use regression model to predict person’s 

costs, then divide by average predicted 

costs 

 Split-sample validation

36 
Predicted Risk Scores

Sample n Mean Std. Dev Min Max

General 1,988,053 1.00 1.62 0.14 41.26

Over 65 641,048 0.91 1.56 0.14 40.75

MH-SUD 819,707 1.64 2.19 0.14 44.02

High cost 255,661 5.58 3.92 0.16 45.92

Multi-morbid 815,088 2.06 2.48 0.14 45.92

Healthy 77,357 0.38 0.36 0.15 11.20

37 
Results 

 Model matters

38 

CMS V21 DCG Medicare 

DCG 

Medicaid 

prospective

with 

pharmacy

prospective 

without Rx

concurrent 

without Rx

prospective 

with Rx

OLS 0.3141 NA 0.3371 0.4373 0.4441

SQRT OLS 0.4287 NA 0.4308 0.5122 0.5682
Results 

 Notable gain from Pharmacy 

DCG 

 CMS V21 DCG Medicare Medicaid 

with prospective concurrent prospective 

 prospective pharmacy without Rx without Rx with Rx

OLS 0.3141 NA 0.3371 0.4373 0.4441

SQRT OLS 0.4287 NA 0.4308 0.5122 0.5682

39 
Results 

 Recalibration is possible and yields 

improved fit 

40 

CMS V21 DCG Medicare 

DCG 

Medicaid 

prospective

with 

pharmacy

prospective 

without Rx

concurrent 

without Rx

prospective 

with Rx

Basic 0.4287 NA 0.4308 0.5122 0.5682

Recalibrated 0.5793 0.6924 0.5819 0.6274 0.6351
Limitations

 Comparison of only 2 systems

 Risk adjustment systems are concerned 

with gaming

– VERA (Veterans Equitable Resource 

Allocation)

– These risk models would not be used for 

payments

41
References

 Ettner SL, Frank RG, Mark T, Smith MW. Risk adjustment of capitation payments 

to behavioral health care carve-outs: how well do existing methodologies account 

for psychiatric disability? Health Care Manag Sci. Feb 2000;3(2):159-169.

 Fishman PA, Goodman MJ, Hornbrook MC, Meenan RT, Bachman DJ, Rosetti 

MCOK. Risk Adjustment Using Automated Ambulatory Pharmacy Data: The 

RxRisk Model. Medical Care. 2003;41(1):84-99.

 Montez-Rath M, Christiansen CL, Ettner SL, Loveland S, Rosen AK. Performance 

of statistical models to predict mental health and substance abuse cost. BMC 

medical research methodology. 2006;6:53.

 Pope GC, Kautter J, Ellis RP, et al. Risk adjustment of Medicare capitation 

payments using the CMS-HCC model. Health Care Financ Rev. Summer 

2004;25(4):119-141.

 Pope GC, Kautter J, Ingber MJ, Freeman S, Sekar R, Newhart C. Evaluation of the 

CMS-HCC Risk Adjustment Model. 2011.

 Sloan KL, Montez-Rath ME, Spiro A, 3rd, et al. Development and validation of a 

psychiatric case-mix system. Med Care. Jun 2006;44(6):568-580.

42
Todd Wagner, PhD

Todd.Wagner@va.gov 

Questions?