HERC Cyber Seminar, April 17, 2013

New Guidelines for Costeffectiveness Models: A Report of 

the ISPOR-SMDM Modeling Good 

Research Practices Task Force

Karen Kuntz ScD
Jaime Caro MDCM, FRCPC, FACP, Chair

Uwe Siebert MD, MPH, MSc, ScD, Co-chair

Karen Kuntz ScD, Co-chair

Andrew Briggs DPhil, Co-chair
Poll of Audience Background

 Are you familiar with decision modeling used in 

cost-effectiveness analyses?

 Yes, I have developed them

 Yes, I have participated in projects with models

 Yes, I have read studies that uses them

 No
For those with experience:

 What types of models are you most familiar with?

 Decision trees

 Cohort Markov models

 Individual-level Markov models

 Discrete event simulation

 Other
Background

 ISPOR has good infrastructure for developing best 

practice papers

 SMDM has one paper on disaster modeling

 2003 article in best practices in modeling 

(Weinstein et al., Value in Health)

 2010 decision to update that paper with a series of 

papers and involve SMDM
Working Groups

Conceptual Modeling Working Group 

Chair: Mark Roberts; Members: Murray Krahn; David Paltiel; Michael Chambers; 

Phil McEwan; Louise Russell 

State-Transition Modeling Working Group 

Chairs: Karen Kuntz; Uwe Siebert; Members: Oguzhan Alagoz; Doug Owens; 

David Cohen; Beate Jahn; Ahmed Bayoumi, 

Modeling Discrete Event Simulation Working Group

Chairs: James Stahl; Jonathan Karnon; Members: Jörgen Möller; Javier Mar; 

Alan Brennan

Dynamic Transmission Modeling Working Group 

Chairs: Richard Pitman; John Edmunds; Members: Maarten Postma; Greg 

Zaric; Marc Brisson; David Fisman; Mirjam Kretzschmar

Model Parameter Estimation & Uncertainty Working Group 

Chair: Andrew Briggs; Members: Milt Weinstein; Mark Sculpher; Elisabeth 

Fenwick; David Paltiel; Jonathan Karnon

Model Transparency and Validation Working Group 

Chairs: David Eddy; John Wong; Members: Joel Tsevat; William Hollingworth; 

Kathy McDonald
Published Papers

 Seven papers – one from each working group and 

an overview paper

 Medical Decision Making 2012 Sept-Oct Issue

 Value in Health 2012 September Issue
Review Process

 All papers underwent external review

 Broad representation

 Reviewed/approved by journal editors

 Peer review comments documented as well as 

responses

 Papers posted for members’ review & comment

 Submission jointly to MDM & ViH

 Editors review
Conceptualizing the model
Conceptual Framework

Reality: health 

care decision, 

process and 

disease

Conceptual Model of:

1) Decision/Problem

2) Disease

Data Sources

Model

Output

Model

Users/ 

Stakeholders

Mathematical

Model

Conceptualizing 

the Problem 

Conceptualizing 

the Model 
Conceptualizing the Model

 Collaborate and consult to ensure that model adequately 

addresses decision problem & disease in question 

 Clear, written statement of the decision problem, objective and 

scope 

 Conceptual structure should

 Be linked to the problem and not based on data availability

 Be used to identify key uncertainties in model structure

where sensitivity analyses could inform the impact of structural 

choices

 Follow an explicit process to convert the conceptualization into 

an appropriate model structure: Influence diagrams, concept 

mapping, expert consultations

 Model simplicity is desirable for transparency, ease of validation 

and description, but

 Must be sufficiently complex to answer the question

 Should maintain face validity
Choice of Model Type

Problem 

Characteristic

 Simple, non-dynamic

 Based on “states” of health

 State explosion

 Interactions, event-based, 

time-to-event

 Resource constraints, 

interactions 

Model Type

 Decision tree

 State transition model

 Individual microsimulation 

 Dynamic transmission 

models, DES, agent-based

 DES, agent-based, dynamic 

transition models

For some decision problems, combinations of model types, hybrid 

models, and other modeling methodologies are appropriate
Uncertainty

 All modeling studies should include an assessment of 

uncertainty as it pertains to the decision problem

 Role of decision maker should be considered

 Authors should be aware that terminology varies within 

the decision modeling & related fields

 carefully define terminology to avoid confusion 

 Identify & incorporate all relevant evidence, rather than 

cherry-picking the “best” source

 Whether employing deterministic SA methods (point 

estimate & range) or probabilistic SA (parameterized 

distribution) the link to the underlying evidence base 

should be clear
Terminology

Preferred term Concept Other terms sometimes 

employed

Analogous concept 

in regression

First-order 

uncertainty

Random variability in 

outcomes between identical 

patients

• Variability

Monte Carlo error

Unexplained heterogeneity

•

•

Error term

Parameter 

uncertainty

The uncertainty in 

estimation of the parameter 

of interest

• Second-order uncertainty Standard error of the 

estimate

Heterogeneity The variability between 

patients that can be 

attributed to characteristics 

of those patients

• Variability

• Observed or explained 

heterogeneity

The Beta coefficients 

(or variability of fitted 

dependent variable)

Structural 

uncertainty

The assumptions inherent in 

the presentation of the 

decision modeling form

• Model uncertainty The form of the 

regression model 

(linear/log-linear etc.)
Parameter estimation & 

uncertainty
Estimating Parameters

 While completely arbitrary analyses (e.g., varying an input 

parameter by +/- 50%) can be used as a measure of sensitivity, 

they should not be used to represent uncertainty

 Consider using commonly adopted standards from statistics, 

such as 95% confidence intervals, or distributions based on 

agreed statistical methods for a given estimation problem

 Where there is very little information, analysts should adopt a 

conservative approach

 In choosing distributional forms for parameters in a probabilistic 

sensitivity analysis, favor should be given to continuous 

distributions that provide a realistic portrayal of uncertainty

over the theoretical range of the parameter of interest 

 Correlation among parameters should be considered
Structural Uncertainty

 Where uncertainties in structural assumptions

were identified in the process of conceptualizing 

and building a model, those assumptions should 

be tested in a sensitivity analysis 

 Consideration should be given to opportunities to 

parameterize these uncertainties for ease of 

testing 

 Where it is not possible to perform structural 

sensitivity analysis it is nevertheless important that 

analysts be aware of the potential for this form of 

uncertainty to be at least as important as 

parameter uncertainty for the decision maker 
Reporting Uncertainty

 Uncertainty analyses can be deterministic or probabilistic

 often appropriate to report aspects of both

 When additional assumptions or parameter values are 

introduced for purposes of uncertainty analyses, these values 

should be disclosed & justified 

 When model calibration is used to derive parameters, 

uncertainty around the calibrated values should also be 

reported, & this uncertainty should be reflected

 When the purpose of a probabilistic sensitivity analysis is to 

guide decisions about acquisition of information to reduce 

uncertainty, results should be presented in terms of expected 

value of information 

 When more than two comparators are involved, CEACs for 

each comparator should be plotted on the same graph
Dynamic Transmission Models
Dynamic Transmission Models

 What are they:

 Models where the risk of infection is dependent on the number 

of infectious agents at a given point in time

 When to use:

 When evaluating an intervention for an infectious disease that 

1) has an impact on disease transmission in the population, and/or 

2) alters the frequency distribution of strains (e.g., genotypes or 

serotypes)

 Use appropriate type based on complexity of the 

interactions, size of the population, and role of chance

 Can be deterministic or stochastic, cohort or individual 

 Justification for the model structure should be given
Agent-based Models

 If using an agent-based model, 

thoroughly describe 

 the rules governing the agents, 

 the input parameter values, 

 initial conditions and all 

 sub-models
State-Transition Models
Structure 

 Cohort or individual simulation? 

 Cohort: if the decision problem can be represented with a 

manageable number of health states incorporating all 

characteristics relevant to the decision problem

 Individual: if unmanageable number of states

 Validity should not be sacrificed for simplicity 

 Specification of states and transitions should reflect the 

biological/theoretical understanding of the disease or 

condition being modeled

 States need to be homogeneous with respect to the 

observed and unobserved (i.e., not known by the decision 

maker) characteristics that affect transition probabilities

 Cycle length should be short enough to represent the 

frequency of clinical events and interventions
Parameters

 Parameters relating to the intervention 

effectiveness derived from observational 

studies should be correctly controlled for 

confounding 

 Time-varying confounding is of particular concern in 

estimating intervention effects
Reporting

 Communicate key structural elements, assumptions and 

parameters using nontechnical language and clear figures 

that enhance understanding of the model

 Depending on the problem, report not only the expected 

value but also the distribution of the outcomes of interest.

 In addition to final outcomes, intermediate outcomes 

should be presented that enhance understanding and 

transparency of the results

 Paper contains illustrative examples of both cohort & 

microsimulation
Discrete Event Simulation
Areas of Application

 Constrained resource scenarios

 Optimising the delivery of services

 technologies result in differing levels of access (e.g. 

different referral rates) and

 time to access resources can have significant 

effects on costs and/or outcomes

 Non-constrained resource scenarios

 More complex health technology assessments

 An alternative to individual state-transition models

 Provides additional flexibility in representing time
Structure 

 To simplify debugging and updating, sub-models should be 

used 

 If downstream decisions can have significant effects on 

costs or outcomes, structure should facilitate analyses of 

alternative downstream decisions

 Mechanism for applying ongoing risks should remain active 

over the relevant time horizon

 For structural sensitivity analyses, alternative structures 

should be implemented within a single DES
Parameterisation

 With competing risks, parameterisation approaches 

that represent correlations between the competing 

events are preferred

 Rather than specifying separate time to event curves 

for each event.

 Where possible, progression of continuous disease 

parameters and the likelihood of related events 

should be defined jointly 

 e.g., sample the level of the continuous measure at 

which an event occurs, then sample the time at which 

the level is reached








Implementation

Software choice depends on importance of flexibility & execution 

speed (general programming) vs. efficiency

 Spreadsheet software is inappropriate for implementing DES

Outputs should 

 be stored as attributes only when individual outcomes are required, 

otherwise aggregated values should be collected from each run

 account for the outputs required for validation 

When run times are constrained, 

 optimal combination of run size & numbers of alternative input parameter 

sets tested should be estimated empirically 

 variance reduction techniques should be implemented

 factorial design and optimum seeking approaches can be used 

 meta-modelling can be used

If system is not empty at start, use a warm-up period if: 

 it can be assumed that the key parameters have remained constant over 

time

 history of the key parameters can be incorporated into the warm-up 

period 
Reporting

 Animated representation that displays the 

experience of events by individuals is 

recommended as a means of engaging with users, 

as well to helping to debug the model through the 

identification of illogical movements

 Both general and detailed representations of a 

DES model’s structure and logic should be 

reported to cover the needs of alternative users of 

the model 
Transparency & Validation








Transparency

Every model should have non-technical documentation that 

should

 Be freely accessible to any interested reader

 Describe in non-technical terms the type of model and intended 

applications; funding sources; structure of the model; inputs, outputs, 

other components that determine the model’s function, and their 

relationships; data sources; validation methods and results; and 

limitations. 

Every model should have technical documentation that should

 Be made available at the discretion of the modelers either openly or 

under agreements that protect intellectual property

 written in sufficient detail to enable a reader with the necessary 

expertise to evaluate the model and potentially reproduce it

Modelers should identify parts of a model that couldn’t be 

validated because of lack of suitable data sources, and describe 

how uncertainty about those parts is addressed.

For multi-application models, describe criteria for determining 

when validations should be repeated and/or expanded.
Validation

 Face validity of structure, evidence, problem formulation, and 

results 

 Should be made by people who have expertise in the problem area, 

but are impartial to the results

 Process used should be described

 If questions about the model arise, these issues should be 

discussed

 Verification (internal validity/consistency)

 Should be described in the non-technical documentation

 Results should be made available on request

 Published models of same or similar problems should be sought 

and similarities and differences discussed
External Validation

 Formal process for conducting external validation that includes:

 Systematic identification & justification of data sources

 Specification of whether a data source is 

 dependent, 

 partially dependent, or 

 independent; 

 Description of which parts of the model are evaluated by each

 Simulation of each data source and comparison of results

 Measures of how results match observed outcomes

 Description of external validation & results available on request 

 When feasible, test for prediction of future events 

 Seek opportunities to conduct predictive validations as part of 

the overall validation process
Final Poll

 Which of the following recommendations do you agree 

with least?

 Structure linked to problem and not based on data 

availability

 Model simplicity is desirable

 Varying inputs arbitrarily does not represent uncertainty

 Technical documentation should be detailed enough to 

reproduce model

 I agree will all of them