**THREE NONPROBLEMS IN THE FREQUENTIST APPROACH TO CLINICAL TRIALS**

**Thomas Permutt, U.S. Food and Drug Administration**

**Division of Biometrics II (HFD-715), Rockville, Maryland 20857**

**Key Words: Multiple endpoints, Multiple looks, Sequential analysis,
Likelihood**

Opinions are those of the author and not of the Food and Drug
Administration.

# introduction

When Dr. Berger told me he wanted to have a Bayesian, a likelihoodist
and a frequentist discuss clinical trials and asked me to participate, I
asked, "Which one am I?" He said, "The frequentist, of course." It was a
bit of a surprise to me, and it may be a surprise to some of my old
friends and teachers. But Vance and I work together at the FDA, and in
the job Vance has known me in, I have been a frequentist; and I think
the frequentist approach has served me well in that job. So I'd like to
talk here not so much about the deep, philosophical issues, as about my
job, and about how the frequentist approach is useful in my job.

I review clinical trials for the Food and Drug Administration. The
review of trials is in most ways remarkably close to the two-decision
problem in the classical von Neumann--Wald--Neyman formulation (although
there are some important differences that I'll discuss later). For
approval of a New Drug Application, the Food, Drug and Cosmetic Act
requires "substantial evidence that the drug will have the effect it
purports or is represented to have." I am asked whether there is, within
a study report, such "substantial evidence" from a statistical point of
view. The decision is pretty nearly binary: there are some less discrete
questions about how the effect should be represented, but for the most
part I am being asked a Yes or No question. This is because the people
asking me are in their turn being asked a Yes or No question: May this
drug be marketed?

Now, in the simplest cases, for all our philosophic differences, we are
pretty well agreed on what to do. Say we have an estimated treatment
effect which is a normal random variable. We all say "Yes" if the
estimated treatment effect is far enough away from zero, in the right
direction. We all think that the likelihood has something to do with how
far away is enough. The frequentist calculates tail probabilities:

$$X > c\ \text{where}\int_{\frac{\varphi(u - u)}{\varphi(u - 0)} > c}^{}{\varphi(u - 0)\ \text{du}} = \alpha\text{.}$$

This calculation involves the likelihood ratio, in the range of
integration, but it is an integral of the density rather than the
likelihood; that is, an integral over a region of the sample space. The
Bayesian does another calculation, quite different in principle,
involving an integral in the parameter space rather than the sample
space:

$$\int_{}^{}{U\left( t,\ d_{1} \right)}\ \varphi(X - t)\pi(t)\ \text{dt} > \int_{}^{}{U\left( t,\ d_{2} \right)}\ \varphi(X - t)\pi(t)\ \text{dt}$$

with utility functions (*U*), prior probabilities (**) and possible
decisions (*d*~1~ and *d*~2~). The likelihoodist says, "You guys always
do these calculations, but the answer always turns out to be 8. Why
don't we just postulate a critical value for the likelihood ratio of 8,
and skip the calculation, which I don't believe anyway?"

It doesn't really matter much. Partly because it doesn't matter, we get
sucked into debates about which is the most logical or elegant way to
view the answer that we all agree on anyway. But the extensions of these
three methods to more complicated questions lead to answers that we
don't all agree on. I want to discuss three kinds of complication:
multiple endpoints, multiple looks and multiple studies. These three
kinds of multiplicity have been dealt with more or less satisfactorily
in the frequentist framework, so I call them three nonproblems.

# Multiple Endpoints

Large, simple trials are called that for a reason: most drug trials are
small, complicated trials. There are hundreds of observations on each
patient; many of them are lab tests looking for side effects, but
typically several different variables could be thought of as measures of
therapeutic effect. So now I have a vector of outcomes, still normally
distributed, say with mean zero under the null hypothesis and variance 1
for each component, and some correlations. For simplicity I'll let the
dimension of this vector be 2, even though the difficulty mainly
concerns higher dimensions.

Suppose I observe the vector (2.2, 0). This is pretty far away from
zero, in the sense that 2.2 is a
![](media/image1.wmf){width="5.995138888888889in"
height="3.0618055555555554in"}pretty big number for a standard normal
variate. The ratio of the likelihood at (2.2, 0) to that at (0, 0) is 11
or more (depending on the correlation), which we all agree is big enough
for the simple problem.

Now, the frequentist says, This point is pretty far away from zero; but
there are a lot of other points far away, too. Which ones? Well, some of
us like to draw squares in the sample space, and say that everything
outside the square is farther away; that's what the Bonferroni approach
amounts to (which I see a lot of in my work, I'm not sure why). Some of
us draw ellipses, that's the Hotelling approach; or you could draw
straight lines of various slopes. But all of us frequentists think there
are a lot of other points equally far away from zero, or farther, than
(2.2, 0). And we also think that as the dimension of the problem gets
bigger, the other points equally far away are more and more a thing to
worry about.

While the frequentist is worrying about the dimension of the sample
space, the Bayesian is worrying about the dimension of the parameter
space. The Bayesian says, Yes, the likelihood of the parameter being
around (2.2, 0) compared to (0, 0) is pretty big; but the prior
probability of a region around (2.2, 0) has to be pretty small: after
all, there are a lot of other regions. And again, as the dimension of
the problem gets bigger, the parameter space gets bigger, so the prior
probability gets spread pretty thin. As a consequence, the Bayesian,
very much like the frequentist, is skeptical of believing what he thinks
he sees in high-dimensional problems.

What about the likelihoodists? Well, some have said that dimension
doesn't matter, because a likelihood ratio of 11 is just a plain, big
likelihood ratio. Maybe not all likelihoodists think that, but I don't
think any has a real theory to deal with the problem, along the lines
that both frequentists and Bayesians do. And, oddly enough, the
frequentists and Bayesians end up with not very different answers to the
problem of dimension, with the likelihoodists far off to one side. So,
the claim of some likelihoodists to hold the common ground between
frequentists and Bayesians doesn't stand up here. Sure, we all believe
in likelihood. What we don't all believe in so much is the importance of
the *maximum* likelihood, which is the likelihood at the point ** =
*X*, compared only to that at ** = 0. Both Bayesians and frequentists
worry much more than likelihoodists about other points besides those
two: frequentists in the sample space and Bayesians in the parameter
space.

# Multiple looks

Because drug studies are complicated, they are also expensive, and we
know more about what the sample size should have been after we do the
trial than before, so there is often some interest in sequential
methods, especially group-sequential trials. The most telling criticism
of frequentism in Dr. Royall's book is that frequentists have botched
the problem of sequential analysis, and that likelihoodism fixes it, or
anyway makes it go away. The issue is important in my work evaluating
clinical trials, and Royall is eloquent on the subject. I also think he
is wrong. Well, no, I shouldn't say that; I promised not to talk about
philosophy but about my job. In fact, the problem may be that Royall's
job and mine are different enough that what works for him doesn't work
for me.

I said earlier that reviewing trials in New Drug Applications is a lot
like, but not quite like, the two-decision problem of classical decision
theory. Well, classical decision theory is a special case of game
theory. A decision problem is a game with only one player and Nature, a
random but disinterested element. A drug review is not a one-player
game. In some ways it is more like a casino game, where the house
announces the rules in advance and other players can place their bets as
they see fit. I am expected not only to make a sound decision about an
individual drug but to do so in a predictable way: to post rules, so to
speak, for a game that drug makers can choose when and how to play. The
drug makers, my brothers in science, my partners in this game, I dare
not say adversaries (for it is certainly not a zero-sum game): anyway,
they are definitely not disinterested elements, and their interests are
similar but not identical to mine.

The essence of the sequential game is this: of all the data that might
be collected in a trial, what subsets are you allowed to submit for
review? To a good approximation, the house rule is this. You can choose
any subset you want, as long as, first, it's all the data you actually
have (i.e., it's the first *n* patients, not the last *n* or some other
set of size *n*, and not even the first *n* after you have seen 2*n*);
and, also, the probability of Type I error is controlled, considering
the whole process including the random stopping time. Royall proposes
dropping the last requirement. All the data are, after all, all the
data, regardless of what was in the mind of the collector.

You can do that in a casino. You can use any stopping rule you like, and
walk away with whatever fortune that leaves you. But why can you do
that? Because you pay by the hand, so your fortune is a martingale, so
the optional sampling theorem applies, so the house never gets hurt, on
average, whatever your stopping rule. In a New Drug Application, you
don't pay by the patient, and there is no automatic assurance that no
one gets hurt by optional stopping rules, so we need other assurances.

Now, some readers may find this to be a rather specialized argument:
this kind of game-playing may be my job, but it's not yours. Lloyd
Fisher, in fact, opines rather sententiously that it's not even my job:
"The problem is that the agency and advisory committee must approve
efficacious drugs and not play scientific games." Later in the same
paper, though, he is clearly arguing not against treating the process as
a game in the mathematical sense but only against playing it in a
particular, silly way. Besides, while you may not all play the drug
approval game, you do play other games: the publication game, the JSM
game, and perhaps what Senn has called "Fisher's Game with the Devil." I
think there are worse ways of looking at science than as a game in which
you may not ever know the true state of nature but you do at least
understand the rules and the consequences your own behavior as a
scientist.

By the way, I would think a Bayesian would tread very carefully here.
The Axioms of Rational Behavior only lead to a Bayes solution for
one-player games. You can go spectacularly wrong trying to apply Bayes
theory, rather than minimax theory, to two-player games:

> "Some day, somewhere," he says, "a guy is going to come to you and
> show you a nice brand-new deck of cards on which the seal is never
> broken, and this guy is going to offer to bet you that the jack of
> spades will jump out of this deck and squirt cider in your ear. But,
> son," the old guy says, "do not bet him, for as sure as you do you are
> going to get an ear full of cider."
>
> ---Damon Runyon, "The Idyll of Miss Sarah Brown"

# Multiple Studies

Of course, a New Drug Application depends on more than one clinical
trial, and the real binary decision is on the application as a whole,
not necessarily on the individual studies. I haven't left much time to
talk about multiple studies. That's just as well, because I'm going to
let the frequentist team down here. I don't think frequentist
meta-analysis, whether with fixed or random study effects, is a very
elegant way to look at the combined evidence from several studies.
Meta-analysis was certainly a necessary thing in a frequentist-dominated
world. Before, you used to have writers of review articles counting up
how many p-values were less than 0.05 and how many were more than 0.05
and "balancing" the "positive" against the "negative" studies in a way
that made no sense at all. Meta-analysis is the only purely frequentist
way out of that trap, but it's a trap that Bayesians and likelihoodists
were unlikely to fall into in the first place. If you plot the
log-likelihood against the parameter value, what Goodman calls "support
curves," you can see clearly whether the studies tend to support or to
contradict each other, and what the common parameter could be if there
could be one. So, I can end on a note of agreement. When the decision,
even a binary decision, depends on information from several studies, it
does not depend only on a p-value for each study. Die-hard frequentists
can still do a significance test at the end of the process, but we need
to analyze the studies together in a rational way. The rational way, we
are all agreed, has to involve the likelihood.

# summary---three frequentist nonproblems

1.  Multiple endpoints, the dimensional problem:

    1.  Frequentists and Bayesians do different integrals which turn out
        practically the same. Likelihoodists just look at the maximum,
        which is very different.

2.  Multiple looks, the game-theory problem:

    1.  Frequentists play the game. Likelihoodists declare victory and
        go home. Bayesians stay on the sidelines.

3.  Multiple studies:

    1.  Everybody looks at the likelihood, or should, anyway.

# references

Fisher, L. *Controlled Clinical Trials* 20:16 (1999)

Senn, S. *Statistics in Medicine* 13:217 (1994)

Goodman, S. *Controlled Clinical Trials* 10:188 (1989)

Royall, R. M. *Statistical Evidence*. Chapman & Hall (1997)