Hypothesis-Based Weight of Evidence:

An approach to structured evaluation of complex evidence 

bases for regulatory judgments about risk and causality

Lorenz Rhomberg, PhD, Fellow ATS

Gradient

EPA/FWS Meeting

6 October 2014
2

Copyright Gradient 2013

 The regulatory process cannot sustain pure science’s 

suspension of judgment until ultimate resolution. A basis 

for actions is needed.

 Skepticism and consideration of alternatives as part of the scientific 

method

 diversity in interpretations among scientists

 The regulatory process needs “findings,” and judgment is 

delegated to particular people tasked with representing 

the larger body of informed scientific opinion.

 So, whose judgments and how they are justified become key

The general WoE Question is as big as “Science,” 

but the application to Chemical Regulation entails 

some special features:
3

Copyright Gradient 2013

“WEIGHT OF EVIDENCE”

 As a metaphor

 As a method

 Use all the data

 Systematic evaluation

 Aim at objective procedures that lay out the process of 

scientific professional judgment

 Question: In view of incomplete and 

contradictory evidence, how compelling is the case 

for the existence and potential magnitude of risk?
4

Copyright Gradient 2013

Desiderata:

 Systematic Approach to Literature Inclusion

 inclusion/exclusion criteria

 not just positive or “featured” outcomes of studies

 Systematic and Consistent Review of Studies

 established procedure, tabulation

 Evaluations of Study Strengths and Weaknesses

 BUT – what to do with “lesser” studies?

 omit? down-weight? interpret?

 study design, power, confounders, potential problems

An Established Process for Evaluation and Combination 

into Inferences
5

Copyright Gradient 2013

Inferences in 

HHRA vs

ECO vs

ESA
6

Copyright Gradient 2013

Rules based vs. guided judgment: 

Sailing between Scylla and Charybdis

A “Known Human 

Carcinogen” is one for which 

the evidence is sufficient to 

conclude that it is a human 

carcinogen.

A “Known Human 

Carcinogen” is one for 

which, following the 

framework, one ends up in 

the “Known Human 

Carcinogen” box.

“JUDGMENT”

 guided evaluations with recorded results

 Judgments are proposed explanations of the array of 

results

 Judgments are justified by citing basis and showing 

superiority over alternatives

“RULES”

“STRUCTURED JUDGMENT”
7

Copyright Gradient 2013

“WEIGHT OF EVIDENCE”

HOW MUCH DOES EVIDENCE WEIGH?

i.e., what process for the evaluation of evidence provides a 

means for judging how compelling it is and how to trade 

off among apparent contradictions?

 “Evidence” has no meaning except in 

relation to a specific hypothesis

 The hypothesis is evaluated w.r.t. the evidence (not 

the other way around)

 So WoE should be organized around evaluating 

specific hypotheses against data
8

Copyright Gradient 2013

Articulate an Hypothesis

What is the proposed basis for inferring that a particular 

phenomenon seen in studies of a chemical's effects will also 

happen in target populations?

 May be hypothesized MoA -- or general (“animals predict 

humans”)

 A generalization, not just an extrapolation (should apply to 

all cases within its realm)

 What manifestations of the hypothesis are expected and not 

expected? Check against all actual observed results.
9

Copyright Gradient 2013

Key WoE Questions

 Based on observed positives, what hypothesized causal 

processes are necessary? Sufficient?

 How do they generalize? What other manifestations should 

they have?

 If hypothesis were wrong, how else would one explain the 

array of outcomes?
10

Copyright Gradient 2013

For Observed Outcomes that are Candidates 

for “Evidence”

 Why we think they happened where they did.

 Why we think they didn’t happen where they didn’t.

 Why we think the “did-happen” factors would also apply to the 

target population.

 Might apply? Probably apply? Known to apply?

 Are there discrepant observations, and if so, how do we account for 

them?

 Are our “whys”

 Observable underlying causes?

 Reasonable guesses based on wider knowledge, other cases?

 Ad hoc assumptions without evidence, needed to explain otherwise 

puzzling phenomena?
11

Copyright Gradient 2013

Some Pitfalls:

 Overreliance on Conventions, Heuristics

 Narrative, citing “consistent” information

 Failure to address null or contradictory findings on same 

endpoints (especially when combining endpoints)

 Multiple Comparisons (choosing from among many parallel alternative analyses 

of same data)

 Mutually contradictory explanations of individual study results

 Ad Hoc Explanations (distinguish discovery from support)

 Accommodation 
12

Copyright Gradient 2013

Evaluate How Compelling the Case is for the 

Proposed Basis of Human Risk in View of:

“Predictions" of hypotheses that are confirmed in the 

observations

 More weight to "risky" and specific predictions

 Less weight when subsidiary assumptions or explanations are needed

 Both Positive and Negative Predictions!

Apparent Refutations (counterexamples)

 Failure to repeat result across studies

 Non-responding sexes or species

 Unpredicted but clearly relevant phenomena

A hypothesis can often be reconciled with apparent refutations 

by either modifying it or adding subsidiary assumptions – but 

this entails a weight "penalty"
13

Copyright Gradient 2013

The Perils of Implicit Accommodation

 X caused increased activity in rats

 In another lab, X caused slightly decreased activity

 but activity measured during day (instead of night)

 Is this supportive evidence?
14

Copyright Gradient 2013

Relative Credence in Competing “Accounts”

• “Account” = an articulated set of proposed 

explanations for the set of observations

• Relevant Causation – but also chance, error, confounding factors, 

general-knowledge possibilities, plausible assumptions, assertions 

of irrelevance, and “unknown reasons”

Certain Findings Indicate 

Target-Population Risk

 reasoning why

 how contradictions resolved

 why assumptions reasonable

Those Findings Do Not Indicate 

Target-Population Risk

 reasoning why not

 how findings are otherwise

explained

 why assumptions reasonable

Can we measure the weights?
15

Copyright Gradient 2013

Advantages of Hypothesis-Based WoE

 Shows which endpoints are the most compelling

 Makes reasoning explicit – shows both strengths and 

weaknesses

 Allows debate about particular data and interpretations

 Frames WoE classifications as scientific statements 

(falsifiable, points to tests)

 Does not blend all endpoints together in an 

uninterpretable, generalized “risk” statement

 Informs the QRA process
16

Copyright Gradient 2013

Sir Austin Bradford Hill

on the Hill Criteria

“. . . the fundamental question – is there any other way 

of explaining the set of facts before us, is there any other 

answer equally, or more, likely than cause and effect?”

A. Bradford Hill (1965) Proc Roy Soc Medicine 58:295.

“set of facts” = 

 all the epi (+ and -)

 mode of action

 animal studies

 other potential explanations
17

Copyright Gradient 2013

Applications

of Hypothesis-Based Weight of Evidence (HBWoE)

 Chloryprifos neurodevelopmental toxicity

 Prueitt, RL; Goodman, JE; Bailey, LA; Rhomberg, LR. 2011. Crit. Rev. Toxicol. 42(10):822-903.

 Goodman, JE; Prueitt RL; Rhomberg, LR. 2012. Dose Response 11(2):207-219.

 Methanol carcinogenicity

 Bailey, LA; Prueitt, RL; Rhomberg, LR. 2012. Regul. Toxicol. Pharmacol. 62:278-291.

 Dioxins thyroid hormone perturbation

 Goodman, JE; Kerper, LE; Petito Boyce, C; Prueitt, RL; Rhomberg, LR. 2010. Regul. Toxicol. Pharmacol. 

58(1):79-99.

 Formaldehyde as a leukemogen

 Rhomberg, LR; Bailey, LA; Goodman, JE; Hamade, AK; Mayfield, DB. 2011. Crit. Rev. Toxicol. 41(7):555-

621.

 Naphthalene carcinogenicity

 Rhomberg, LR; Bailey, LA; Goodman, JE. 2010. Crit. Rev. Toxicol. 40(8):671-696. 

 Methylmethacrylate nasal toxicity

 Pemberton, M; Bailey, EA; Rhomberg, LR. 2013. Regul. Toxicol. Pharmacol. 66(2): 217-233.

 Toluene Diisocyanate carcinogenicity

 Goodman, JE; Prueitt, RL; Rhomberg, LR. 2013. Crit. Rev. Toxicol. 43(5):391-435.
18

Copyright Gradient 2013

Publications on the HBWoE Method

 Rhomberg, LR. 2014. "Hypothesis-Based Weight of Evidence: An Approach 

to Assessing Causation and its Application to Regulatory Toxicology." Risk 

Analysis. doi: 10.1111/risa.12206.

 Rhomberg, LR; Goodman, JE; Bailey, LA; Prueitt, RL; Beck, NB; Bevan, C; 

Honeycutt, M; Kaminski, NE; Paoli, G; Pottenger, LH; Scherer, RW; Wise, KC; 

Becker, RA. 2013. "A Survey of Frameworks for Best Practices in Weight-ofEvidence Analyses." Crit. Rev. Toxicol. 43(9):753-784.

 Lutter, R; Abbott, L; Becker, R; Borgert, C; Bradley, A; Charnley, G; Dudley, 

S; Felsot, A; Golden, N; Gray, G; Juberg, D; Mitchell, M; Rachman, N; 

Rhomberg, L; Solomon, K; Sundlof, S; Willett, K. 2014. "Improving Weight 

of Evidence Approaches to Chemical Evaluations." In Press in Risk Analysis.