Alternatives To Establishing Thresholds:

A Predictive Modeling Decision Framework

January 23^rd^, 2001

Notes Prepared for Discussion at the CVM Antimicrobial Resistance
Meeting

by

Louis Anthony Cox, Jr.

Cox Associates

Denver, Colorado

*CHALLENGE: How to protect public health against food-borne
antibiotic-resistant bacteria?*

1.  Size of health risk is typically uncertain.

-   Could be zero in many applications -- but not sure!

2.  *Multiple* contributing factors How to control?

-   Effect on risk of changing one factor (e.g., at farm,
    transportation, processing, storage and preparation, treatment)
    depends on what happens at other stages.

-   Confounding and risk-attribution challenges

-   How much care should each contributor take?

3.  Non-traditional exposure models

-   Bacteria can survive and grow

4.  Non-traditional dose-response relations

5.  Susceptible populations not well understood

6.  Biologically-based risk models not yet adequately developed.

-   Initiation Growth Survival Reproduction

-   Dose-reponse for infection and illness

-   Impacts on response to treatment

RISK ANALYSIS FRAMEWORK

1.  *Identify and define* the hazard of concern.

-   Source, target population, adverse health effect

-   Causal mechanism(s) for exposure-response

2.  Identify a set of alternative risk management acts to be evaluated.

3.  Quantify the probable exposure and health consequences of each
    alternative act.

-   Estimate *population frequency distribution of exposures* (microbial
    loads ingested)

-   Estimate *population frequency distribution of illness-days* for
    excess illnesses caused by the exposures

-   Discrete-event simulation framework supports both.

4.  Characterize *uncertainties* about the probable health consequences
    of each act.

-   Uncertainty and sensitivity analyses

-   Data, model, and construct uncertainties

5.  Choose the risk management act (from among those evaluated) having
    the *most preferred probable frequency distribution* of health
    impacts (and uncertainties).

Four Principles for Useful Risk Modeling

*[Principle 1]{.underline}: Include [all]{.underline} known important
risk factors in the risk model.*

*[Principle 2]{.underline}: Quantify probable [causal]{.underline}
relations between (a) Actions and exposures; and (b) Exposures and their
adverse health consequences (given other risk factors).*

> Diet, cats, kids, food safety habits, etc.

Act Exposure Risk Age

*[Principle 3:]{.underline} Model the [physical process]{.underline}
leading from acts to health effects (as far as practical with known
information).*

*[Principle 4]{.underline}: [Validate]{.underline} the model (whole and
assumptions) using [individual-level data]{.underline}.*

-   *Do other models fit the facts better?*

-   *Remaining uncertainties (model, data, construct)?*

*[Principle 1]{.underline}: Include [all]{.underline} known causally
important risk factors in the risk model.*

Use *variable-selection techniques* to find important risk factors in
CDC data. (*Example:* Classification tree output.)

![](media/image1.wmf){width="7.572916666666667in"
height="7.563194444444444in"}

Other techniques for identifying variables that matter for predicting
risk:

1.  Correlations

2.  Regression

3.  Regression with variable selection

4.  Bootstrap, cross-validation, Bayesian...

**CORRELATIONS OF RISK FACTORS WITH ILLNESS STATUS**

Bold correlations are significant at p \< .05000

  -------------- -------------------------------------------------- ---------------
  **Factor**     **Description**                                    **Correlation
                                                                    with Cases**

  Q13C_PRE       how many **meals did you prepare**, 1-15=number    **-.07**

  **Q19C_RES**   subject eat **restaurant lettuce** on a sandwich   **.08**
                 in last 5 days, 1=y, 0 = n                         

  **CAT1**       **cats** in the house in last 5 days, 1=y, 0 = n   **.05**

  CHICK1         **raw chicken** in your home in last 5 days, 1=y,  .01
                 0 = n                                              

  CHICK2         **raw chicken** in the refrigerator in last 5      -.02
                 days, 1=y, 0 = n                                   

  CHICK3         **meals prepared with chicken** in last 5 days,    -.01
                 1=y, 0 = n                                         

  CHICK4         subject **touch any raw chicken** in last 5 days,  -.01
                 1=y, 0 = n                                         

  **CHICK5**     parent **wash hands** after handle chick last 5    **-.03**
                 days, 1=no, 2=sometimes, 3=usually, 4=always,      
                 5=never handle                                     

  **Q64WASHB**   do you **wash hands after handling ground beef,**  **-.03**
                 1=no, 2=sometimes, 3=usually, 4=always, 5=never    
                 handle                                             

  **CHICK6**     **wash board** after cut raw chick in last 5 days, **-.04**
                 1=no, 2=sometimes, 3=usually, 4=always, 5=never    
                 handle                                             

  **Q9RESTAU**   subject **eat at restaurant in last 5 days**,      **.03**
                 1-15=number of times, 77=DK/NS, 88=none,           
                 99=refused                                         

  **Q36PNKHB**   subject eat **pink hamburgers** (home) in last 5   **.06**
                 days, 1=y, 0 = n                                   

  **Q37RSTHB**   subject eat **hamburgers at a restaurant** in last **.08**
                 5 days, 1=y, 0 = n                                 

  **Q38PRSHB**   subject eat **pink hamburgers (restaurant)** in    **.05**
                 last 5 days, 1=y, 0 = n                            

  **Q42PNKST**   subject eat **pink steak** in last 5 days, 1=y, 0  **.06**
                 = n                                                

  **Q44PNKRB**   subject eat **pink roast beef in** last 5 days,    **.06**
                 1=y, 0 = n                                         

  **Q60_BURG**   **pink burger** is considered cooked, 1=y, 0 = n   **.04**

  **Q61_BRG2**   if subject received a **pink burger,** would you   **-.05**
                 **send it back**, 1=y, 0 = n                       

  **Q65LABEL**   ever **noticed handling label on raw meat**, 1=y,  **-.03**
                 0 = n, 3=never buy                                 

  **Q67CHANG**   **changed behavior** because of handling label on  **.05**
                 raw meat, 1=y, 0 = n                               

  **Q112_AGE**   **age in years,** 0-99=age in years                **-.09**

  **Q115_EDU**   INTERVIEWEE\'s **education** (not necessarily      **.04**
                 subject), 1=no school, 2=elemen, 3=some HS, 4=GED, 
                 5=some college, 6=college grad, 7, 9               

  **Q118INCM**   1995 **gross** **income,** 1 = \<15k, 2=15-30k,    **-.02**
                 3=30k-60k, 4=60-100k, 5=GT 100k                    
  -------------- -------------------------------------------------- ---------------

\**Note:* All data and analyses in these notes are illustrative only.
With CDC's assistance, similar analyses can be conducted using real data
from both random surveys and case-control data.

Multiple Linear Regression Outputs

Regression Summary for Dependent Variable: RISKRATE

  ------------- ------- ----------- -------- ---------- ---------- ----------
                BETA    Std. Err.   B        St. Err of t(7473)    p-level
                        of BETA              B                     

  Intercpt                          .17      .04        4.4        .0000

  Q13C_PRE      -.031   .01         -.00     .00        -2.5       .0128

  Q19C_RES      .047    .01         .05      .01        4.1        .0000

  Q3_DAYCA      .000    .01         .00      .02        .0         .9846

  Q4TRAVEL      .006    .01         .01      .03        .5         .6344

  Q9RESTAU      .005    .01         .00      .00        .4         .6649

  Q18RESTL      .036    .01         .03      .01        3.0        .0030

  Q36PNKHB      .007    .01         .01      .02        .6         .5319

  Q37RSTHB      .023    .01         .03      .01        1.9        .0548

  Q38PRSHB      .022    .01         .05      .03        1.9        .0625

  Q42PNKST      .018    .01         .02      .01        1.5        .1288

  Q44PNKRB      .014    .01         .03      .02        1.2        .2241

  *CHICKEN*     .003    .01         .00      .01        .3         .7806

  Q60_BURG      .020    .01         .02      .01        1.6        .1090

  Q61_BRG2      -.028   .01         -.03     .01        -2.4       .0166

  Q64WASHB      -.006   .01         -.00     .00        -.5        .6144

  Q65LABEL      -.035   .01         -.02     .01        -2.9       .0032

  Q112_AGE      -.054   .01         -.00     .00        -4.4       .0000

  Q115_EDU      .018    .01         .01      .00        1.5        .1408

  Q118INCM      -.031   .01         -.01     .00        -2.7       .0074
  ------------- ------- ----------- -------- ---------- ---------- ----------

Multiple Regression with Variable Selection

Regression Summary for Dependent Variable: RISKRATE

(Using backward stepwise variable selection)

  ---------------- ------------ --------------- ------------- --------------
                   B            St. Err. of B   t(7489)       p-level

  Intercpt         .171         .014            12.6          .000000

  Q112_AGE         -.001        .000            -6.3          .000000

  Q18RESTL         .041         .009            4.5           .000008

  CAT1             .031         .009            3.5           .000433
  ---------------- ------------ --------------- ------------- --------------

**Example of Confounding**

Chicken Consumption, Restaurant Lettuce Consumption, and Risk of CP all
decrease with age. But this does not mean that reduced Chicken
Consumption *causes* reduced Risk of CP.

![](media/image2.wmf)

Multivariate analysis shows that Chicken Consumption is *correlated*
with Risk of CP, but does not appear to causally *affect* it.

-   *[Not:]{.underline} Age Chicken Consumption CP Risk*

```{=html}
<!-- -->
```
-   But rather: *Chicken Consumption Age CP Risk*

**How to Include Causally Relevant Variables, 1: *Correct for
Confounders***

> ***less raw chicken* in house**

######## *fewer chicken meals* ( lower Q?)

> **+**

#######  Older Age (Confounder)

> **+**

-   **eat out less**

-   **no kids in day care**

-   **less cat ownership**

-   **prepare meals at home more**

-   **less pink roast beef consumed**

-   **less acceptance of undercooked burgers**

-   **more often wash hands after ground beef**

> **+**

######### *Reduced Risk* of CP illness ( lower H?)

***Older Age* is positively correlated with *Less Chicken* consumption
*and* with *Reduced Risk* of CP illness per person per year -- but this
does not mean that *Less Chicken* consumption *causes* less CP risk.
Other correlates of *Older Age* may explain the lower risk of CP
illness.**

Correlations based on individual data ('96-'97 CDC Survey) Show Several
Confounders for the Chicken-Risk Relation.

  -------------------------------------- ------------ ------------- ----------
                                         **RISKRATE   **CHICKEN**   **AGE**
                                         (Case)**                   

  RISKRATE = Pr(Case instead of control) 1.00         .01           **-.09**

  CHICK3 = Meals prepared with chicken   -.01         .36           -.02

  Q3_DAYCA = Children in day care        .01          .01           -.14

  Q9RESTAU = Ate in restaurant           **.03**      **.05**       -.07

  Q18RESTAURANT LETTUCE                  **.07**      **.06**       -.20

  Q44PNKRB = Pink roast beef             **.06**      **.06**       -.06

  CAT1 = Cat ownership                   .05          -.01          -.08

  CHICKEN = Ate chicken meal in last 5   .01          1.00          -.03
  days                                                              
  -------------------------------------- ------------ ------------- ----------

\**Note:* All data and analyses in these notes are illustrative only.
With CDC's assistance, similar analyses can be conducted using real data
(for both random surveys and case-control data).

**Comments on Confounders**

A risk model that does not control for confounders is not a correct
epidemiological model. It should not be used to draw inferences or to
make decisions.

Controlling for confounding can clarify causal paths and help create
useful predictive risk models.

Example: Competing Causal Models

*[Model 1]{.underline}: chicken risk undercooking*

vs.

*[Model 2]{.underline}: beef risk poor preparation chicken risk*

*lettuce risk, etc.*

"Among persons with no foreign travel the following exposures were
significant risk factors for infection: eating **undercooked poultry**
(mOR 1.9), **chicken or turkey cooked outside the home** (mOR 2.4),
**non-poultry meat cooked outside the home** (mOR 2.2)... Eating chicken
or turkey cooked in the home was a protective factor (mOR 0.5)."
(Friedman et al., 2000)

*[Principle 1]{.underline}: Include [all]{.underline} known important
risk factors in the risk model (Cont.)*

*Q: Why bother?*

A: Omitting relevant variables creates *spurious* *statistical
relations* (no causal interpretation). Such relations (e.g., H = kQ,
with only one explanatory variable) do *not* support good decisions.

*A Simple Hypothetical Example:*

*H =* ILLNESS RISK

-1 1

*Q =* EXPOSURE AGE (Omitted)

+1/3

-   ILLNESS RISK = AGE -- EXPOSURE (Causal)

-   [EXPOSURE = (1/3)\*AGE AGE = 3\*EXPOSURE]{.underline}

```{=html}
<!-- -->
```
-   ILLNESS RISK = 2\*EXPOSURE (Statistical Relation)

*[Lesson]{.underline}:* Even when exposure *reduces* risk, the
statistical model H = kQ can show a *positive* relation between them.

*[Principle 1]{.underline}: Include [all]{.underline} known important
risk factors in the risk model (Cont.)*

When there are multiple risk factors, the model H = kQ must be replaced
by a *multivariate* risk model.

> [Example]{.underline}*:*
>
> *Food safety habits at home*
>
> *Age* *Risk* *Restaurant dining*
>
> *Cat ownership*

Multiple-variable model:

*H = (k~1~Q~1~ + k~2~Q~2~ + ...+ K~n~Q~n~)\*f(Age)*

-   Q~1~ = consumption of frozen chicken

-   Q~2~ = consumption of cooked chicken at home

-   Q~3~ = consumption of chicken salad out of home

-   Q~4~ = consumption of rare roast beef at home....

Effect of changing "Q" depends on *which component* is changed (and on
interactions).

*A simple example of factor interaction:* Age and Cat ownership.

![](media/image3.wmf){width="6.4375in" height="5.844444444444444in"}

Such risk *factor interactions* require a *non-linear* (and
multivariate) risk model:

H = f(*Q~1~, Q~2~, ..., Q~n~*), Q's are the risk factors.

*[Principle 2]{.underline}: Quantify probable [causal]{.underline}
relations between actions, exposures, and adverse health consequences
(given other risk factors).*

1.  [Draw the hypothesized causal path diagram]{.underline} based on
    individual-level data.

2.  [Test the causal hypotheses]{.underline} implied by the diagram.
    (Conditional independence, composition relations are implied).

3.  [Quantify]{.underline} the conditional probability relations in the
    diagram.

Example:

> Food safety habits lettuce, chicken, beef at home
>
> Age *Risk* Restaurant dining
>
> Cat ownership chicken out of home
>
> non-chicken out of home

Basic Ideas of the Simulation Model

Farming practices Season

Initial microbial load (CFU/chicken) leaving farm

Microbial load after transportation

Microbial load after processing

Microbial load after storage/preparation

Microbial load ingested

(CFUs/meal) x (meals/yr.) demographics

Dose-response model age

FQ resistance? Illness-days FQ prescription?

*final microbial load = initial load \* transportFactor \* processFactor
\* storageAndPrepFactor,*

**Comments on Causal Modeling**

1.  Attributable risk calculations do *not* address issues of causation.

2.  Non-causal models and causal models that have not been
    tested/validated (including *H = kQ*) *should not be used* for risk
    management decision-making.

3.  When multiple risk factors (e.g., both resistant and non-resistant
    CP strains) contribute to an illness, the traditional "attributable
    risk" is not well-defined.

Attributable Risk 101:

  --------------------- ---------------------- --------------------------
                        Exposed                Unexposed

  Cases                 50                     10

  Controls              50                     90

  Risk Rate:            0.5                    0.1
  --------------------- ---------------------- --------------------------

*Relative Risk* = 0.5/0.1 = 5

*Attributable risk* = (50 -- 10)/60 = 2/3

Attributable Risk 201:

Suppose that people are exposed to factor A if and only of they are also
exposed to factors B, C, and D.

A B C D Risk

(e.g., raw chicken pink chicken chicken Risk)

Then *each factor* has the same attributable risk (2/3). The same is
true in this model:

A B C D Risk

Multivariate attributable risks are *required* to avoid errors and
paradoxes.

**APPLICATION TO CVM'S PROPOSED APPROACH**

"Additionally, exposure is **associated** **with** meat **containing**
drug-resistant bacteria whose resistance **was attributed to** the use
of antimicrobial drugs in the food animal species that produced the
meat.

One way to estimate the **prevalence of people impacted** (H(x)) is as
follows:

> > H(x) = estimate of total prevalence of cases of disease\
> > \
> > proportion of total cases **due to exposure** to animal-derived food
> > commodity\
> > *Not defined!*
>
> > proportion of cases with resistance **attributed to** animal-derived
> > food commodity\
> > \
> > proportion of cases **expected** to be treated with the
> > antimicrobial drug of interest

The estimated total prevalence of cases is determined annually for
foodborne pathogens in FoodNet, Foodborne Diseases Active Surveillance
Network, by the CDC. Periodically, the CDC also conducts case-control
studies and surveys that provide more detailed information."

**Application to CVM's Threshold Framework:**

**"Measurable human health impact (H(x)):** One way to estimate the
prevalence of people impacted (H(x)) is as follows:

> H(x) = estimate of the total prevalence of cases of disease
>
> remove confounders!
>
> proportion of total cases **due to exposure to animal-derived food
> commodity**\
> \
> proportion of cases with resistance **attributed to** animal-derived
> food commodity\
> \
> proportion of cases expected to be treated with the antimicrobial drug
> of interest

The estimated total prevalence of cases is determined annually for
foodborne pathogens in FoodNet, Foodborne Diseases Active Surveillance
Network, by the CDC. Periodically, the CDC also conducts case-control
studies and surveys that provide more detailed information. Such
information includes the prevalence of people impacted by infections
**caused by drug-resistant bacteria** ***remove effects of other
factors!*** for which they sought care and received an antibiotic to
which the bacteria were resistant."

***[Principle 3:]{.underline} Model the [physical process]{.underline}
leading from acts to health effects (as far as practical with known
information).***

*Q: The proportion of sampled meat containing bacteria depends on the
size of the sample units (e.g., pounds vs. kilograms). What sample unit
size should be used to calculate risk, and why?*

**Figure A:** Proportion of samples (boxes) containing bacteria (x) =
2/4 = 0.5

  ----------------- ----------------- ----------------- -----------------
                    x                 x                 

  ----------------- ----------------- ----------------- -----------------

**Figure B:** Proportion of samples containing bacteria = 2/8 = 0.25

  -------- -------- -------- -------- -------- -------- -------- --------
                    x                 x                          

  -------- -------- -------- -------- -------- -------- -------- --------

**Figure C:** Proportion of samples containing bacteria = 2/16 = 0.125

  ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
                      x                   x                                  

  ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

The *CFU of bacteria per pound* of meat remains constant. But the
*proportion of samples* with bacteria decreases with samples per pound.

**Application to CVM's Threshold Framework:**

**"Measurable level of exposure (Q):** This document describes a
measurable level of exposure as the number of **pounds of a particular
food animal commodity containing drug-resistant bacteria**. ... The
**portion of the total number of pounds containing any of the bacteria
of interest** is estimated from the USDA data. The prevalence of
antimicrobial resistance among bacterial isolates is determined through
the NARMS program.

Therefore, Q could be estimated as follows:

> Q = total pounds of product consumed
>
> *Why pounds?*\
> \
> **proportion of sampled pounds containing bacteria**\
> \
> proportion of samples from which resistant bacteria are isolated

(Grams gives a different answer than pounds.)

To avoid ambiguous risk estimates (i.e., different if consumption is
measured in grams or tons or pounds), one should use [*CFU load* *per
pound*]{.underline} (or gram or ton) of meat, not proportion of samples
with contamination.

Then, the risk estimate will no longer depend on the sample unit size.

Simulating the *frequency distribution of microbial load per meal*
consumed solves the problem.

Simulation also allows the frequency of *high-contamination events*
(e.g., above minimum infective dose) to be quantified.

**Example:**

  -------- -------- -------- -------- -------- -------- -------- --------
                    x                                   x        

  -------- -------- -------- -------- -------- -------- -------- --------

vs.

  -------- -------- -------- -------- -------- -------- -------- --------
                    x x                                          

  -------- -------- -------- -------- -------- -------- -------- --------

***[Principle 4]{.underline}: [Validate]{.underline} the model (whole
and assumptions) using [individual-level data]{.underline}.***

*Example:* Don't assume that *H = kQ* if it is possible that *H = kQ +
background.* In any case, test the assumption that *H = kQ!*

![](media/image4.wmf)

Assuming a model without validating it can give useless results. (E.g.,
it can turn a zero or negative relation into a positive estimated one.)

*[Example]{.underline}:* Fitting CVM's model, *H = kQ,* to available
FoodNet catchment area data
(<http://www.cdc.gov/ncidod/dbmd/foodnet/SURVEYS/Pop_surv.htm>) shows a
significant *positive* exposure-risk relation. But fitting a linear
regression line to the same data shows a non-significant *negative*
relation. Which is more valid?

![](media/image5.wmf)

Regression Summary for Dependent Variable: INCIDENC: R= .94 R²= .88
Adjusted R²= .87

+-------------------------------+--------+-----------+-----+----------+
|                               | B      | St. Err.  | t   | p-level  |
|                               |        | of B      | (8) |          |
+-------------------------------+--------+-----------+-----+----------+
| #### CHICKEN Consumption      | #      | #### .05  | ##  | ####     |
|                               | ### .4 |           | ##  |  .000051 |
| #### (No intercept)           |        |           | 7.8 |          |
+-------------------------------+--------+-----------+-----+----------+
| CHICKEN Consumption           | ###    | #### .70  | ##  | #### .91 |
| (intercept allowed)           | # -.07 |           | ##  |          |
|                               |        |           | -.1 |          |
+-------------------------------+--------+-----------+-----+----------+

A: The more general model (H = kQ + b) includes H = kQ as a special
case. It should be used if it is *possible* that the background risk is
non-zero (always, in practice).

In this example, the negative regression relation, rather than CVM's
positive relation, is more valid for use in risk management.

CVM's model can show a significant positive relation even when the data
(and a regression line) show a negative relation because H = kQ must go
through the origin, even if measured values for H and Q do not.

![](media/image6.wmf)

In practice, non-zero background (intercept) terms reflect *omitted
explanatory variables* (and imperfectly defined attributable risks).

*Why it matters: Unvalidated modeling asumptions can lead to
inappropriate resource allocations.*

![](media/image7.wmf)

Straight lines through the origin (H = kQ) would mis-identify Fried
Chicken as a more serious problem than non-fast food restaurants
(NFAST6_7), because it has a higher intercept.

Model population *frequency distributions* of exposures, not average
exposures. (Most dose-response models show that high concentrations of
CFUs are dis-proportionately likely to cause illness.)

![](media/image8.wmf){width="5.996527777777778in"
height="4.497222222222222in"}

Reducing *average* microbial load reduces the *right-tail* of the
microbial load frequency distribution (the part that causes illness)
much more than proportionally.

**Summary and Conclusions**

A satisfactory framework for protecting public health must:

1.  Quantify risks of alternative actions.

2.  Model the effects of multiple risk factors and explanatory
    variables, including confounders.

3.  Account for factor interactions.

4.  Avoid untrue and untested assumptions about model form.

5.  Quantify frequency distributions of exposures and resulting risks.

Be consistent with the underlying physical process leading from acts to
probable consequences.