UNITED STATES FOOD AND DRUG ADMINISTRATION













                   DRAFT GUIDANCE FOR INDUSTRY AND FOOD AND DRUG

                   ADMINISTRATION STAFF; GUIDANCE FOR THE USE OF

                      BAYESIAN STATISTICS IN MEDICAL DEVICES















                                Rockville, Maryland

                              Thursday, July 27,2006






                                                              2


           1    PARTICIPANTS:

           2    FDA

           3       DANIEL SCHULTZ

           4       DONNA-BEA TILLMAN

           5       SUSAN GARDNER

           6       GARY GRAY

           7       TELBA IRONY

           8       GENE PENNELLO

           9       LAURA THOMPSON

          10    Panel I

          11       LARRY KESSLER

          12       SUSAN ALPERT

          13       DONALD BERRY

          14       BARBARA BUCH

          15       BAILEY LIPSCOMB

          16       JANET TRUNZO

          17    Panel II

          18       GREG CAMPBELL

          19       BRADLEY P. CARLIN

          20       RALPH D'AGOSTINO

          21       STEVEN GOODMAN

          22       ROSEANNE WHITE




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                              3


           1    PARTICIPANTS (CONT'D):

           2    BRAM ZUCKERMAN

           3    Public Speakers

           4       PETER 'TONY' LACHENBRUCH

           5       PATIENCE AJONGWEN

           6       MICHAEL LU

           7       BARATHI SETHURAMAN

           8       JERRY ZHANG

           9       ANDY MUGGLIN

          10       MARK VON TRESS

          11       JANICE HOGAN

          12       LOU JACQUES

          13       XIAOZHENG ZHANG

          14       ROGER LEWIS

          15       HONGJIE ZHENG

          16       DAVID BREITER

          17    Other Participants

          18       MARY FOULKES

          19       DORTHEA UGI

          20       TERESA ZYCZYNSKI

          21       WEISHI YUAN

          22       KYUNG NAM LEE




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                              4


           1    PARTICIPANTS (CONT'D):

           2       NARINDER SHARGIU

           3       LIANG LI

           4       OWEN FARIS

           5       PATRICIA LOVE

           6       ANN MELRERSON

           7       DORA VEGA

           8       RANDALL BRAKMA

           9       MATHEW TRIPTHI

          10       JOANN MINOR

          11       ANN HARDY

          12       JEANIE KENNEDY

          13       STEVE THOMSON

          14       LAURENCE LANDOW

          15       PETER LACHENBRUCH

          16       LEV SIROTA

          17       ROSS BRECHNER

          18       JANET JOHNSON

          19       ROBERT WAGNER

          20       GHANSHYAM GUPTA

          21

          22                       *  *  *  *  *




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                              5


           1                   P R O C E E D I N G S

           2                                             (8:34 a.m.)

           3              MR. CAMPBELL:  I would like you --

           4    to ask you to take your seats if you would.

           5    We are about to get started.  Welcome, my

           6    name is Greg Campbell.  I am the Director of

           7    the Division of Biostatistics in the Office

           8    of Surveillance and Biometrics at the Center

           9    for Devices and Radiological Health at the

          10    FDA, and I am welcoming you to this Public

          11    Meeting on the document, the draft guidance

          12    document entitled "The Use of Bayesian

          13    Statistics in Medical Device Clinical Trials.

          14              I need to make a few short

          15    announcements about, sort of, the logistics

          16    part.  You will find in your registration

          17    folder, a one-pager about lunch, lunch is on

          18    your own today.  And those are some of the

          19    places that you can find that are within a

          20    mile or so of the auditorium here.  There are

          21    also other restaurants a little further down

          22    Shady Grove Road, if you so choose.  Coffee




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                              6


           1    break is actually on your own as well.  Since

           2    this is an FDA meeting.  There is a coffee

           3    place upstairs on the second floor.

           4              And, in addition there -- this

           5    meeting will be transcribed, and I need to

           6    make a few remarks about that.  It is a

           7    transcription, but it is not going to help in

           8    terms of people who are formulating comments,

           9    official written comments for the docket.

          10    Because the transcription won't be completed

          11    until after the comment period has passed.

          12    So if you are thinking about making written

          13    comments to the docket and I hope you do.

          14    Please don't wait for the transcription.

          15              Thanks are in order for Cindy

          16    Garris and Bridgette Stein, who help man the

          17    table out-front.  A reminder, there is no

          18    food or drink in the auditorium, so I would

          19    like to ask you to respect that.  In the

          20    morning, we will have some introductory

          21    remarks, and then a number of statisticians

          22    from the Division of Biostatistics will talk




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                              7


           1    in depth about the guidance document.

           2              And there are two panels in the

           3    afternoon, one on planning considerations,

           4    and one on practical issues.  What I'd like

           5    to do is, to call your attention to the notes

           6    in your folder.  At the bottom of each page

           7    of the notes, you will see in very little

           8    print instructions on how to make official

           9    comments to the docket.  What is not there is

          10    the date by which you need to do that, which

          11    is August the 21st, so the comment period,

          12    which began May the 23rd, goes until August

          13    the 21st.

          14              It is my pleasure at this time to

          15    welcome the Director of the Center for

          16    Devices and Radiological Health, Dr. Daniel

          17    Schultz who will make a few introductory

          18    remarks.

          19              MR. SCHULTZ:  I just want to add to

          20    what Greg said about making comments to the

          21    docket.  There is a specified time for making

          22    comments on draft guidances, but the docket




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                              8


           1    is always open for comment, is that correct

           2    -- is that a correct statement?

           3              SPEAKER:  I am not sure it is.

           4              MR. SCHULTZ:  I think so, anyway.

           5    So we encourage you to make your comments,

           6    we'd like you to make them during the

           7    specified time, but we also encourage you to

           8    think about this document, not just for the

           9    next couple of weeks, but for a long time to

          10    come, and to communicate back to us in any

          11    way possible your thoughts and ideas.

          12              Any way, I am not here to explain

          13    Bayes' theorem.  I am not here to talk about

          14    posterior probabilities.  You are going to be

          15    hearing a lot about that later on, during

          16    this morning's talks, and this afternoon's

          17    discussion.  What I am here to talk about is,

          18    how this document, this idea, this type of

          19    meeting contributes to getting safe and

          20    effective medical devices and medical

          21    products to the market place.

          22              I think that this is one-step, in a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                              9


           1    long evolution in terms our interest and

           2    CDRH's leadership in bringing the idea of

           3    Bayesian Statistics into the regulatory

           4    process.  This did not happen by accident.

           5    We have been having discussions and thinking

           6    about this issue for a long time.  We

           7    actually held a workshop back in 1998,

           8    entitled Bayesian Methods in Medical Device

           9    Clinical Trials.

          10              In 2004, FDA and Johns Hopkins

          11    co-sponsored a workshop, "Can Bayesian

          12    approaches to studying new treatments improve

          13    regulatory decision making."  In the interim,

          14    we have seen a progression of the use of

          15    Bayesian Statistics in real-life situations,

          16    in real-life submissions, which have in fact

          17    contributed to the review process, and again

          18    the ultimate goal, which is making it

          19    possible to get safe and effective products

          20    to market using good science.

          21              Based on our experience we've

          22    written this guidance document, "we" -- I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             10


           1    guess that's the very much the royal "we"

           2    have written this guidance document that

           3    reflects our current thinking on how Bayesian

           4    Statistics might be applied to medical

           5    devices.  And there is a reason why CRH has

           6    taken the lead in this effort.

           7              And part of it has to do with the

           8    fact that we've hired some of the best, and

           9    brightest, and most innovative statisticians

          10    both Bayesian and otherwise in the world, and

          11    I think we have a group that is very, very

          12    interested in -- in pursuing this area and

          13    making it a reality with respect to the

          14    regulatory process.

          15              And in addition to that we

          16    obviously have a director of our statistical

          17    group who is always thinking about new ways

          18    and better ways to help us do our job in

          19    terms of making sure that we have good

          20    evidence that allows us to bring new products

          21    to market.

          22              The other reason is that medical




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             11


           1    devices are particularly applicable to the

           2    use of Bayesian Statistics.  The way that

           3    medical devices are developed, the fact that

           4    there tends to be an evolutionary rather than

           5    a single, sort of, date of discovery followed

           6    by development in medical devices makes

           7    Bayesian Statistics a very, very attractive

           8    option in terms of how we look at and how we

           9    study these products.

          10              We are holding this meeting because

          11    we want to hear from you.  It's one thing to

          12    put out a guidance document and say you know,

          13    "this is what we think."  But the reality is

          14    that the only reason for doing guidance, and

          15    the only reason for putting out these

          16    documents is if they are going to be used.

          17    And the only way they are going to be used is

          18    if people read them, comment on them, think

          19    about them, give us feedback, and figure out

          20    ways that we can work together to try to make

          21    this not just a piece of paper, but actually

          22    a reality in terms of our daily lives.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             12


           1              And again, sort of, getting back to

           2    what I am interested in is why is this really

           3    important?  There are a lot of reasons why

           4    there are delays in getting new technologies

           5    to market.  And I am sure you've heard about

           6    lots of them, people talk to us about our

           7    regulatory process, and about efficiencies,

           8    and about resources, and about a whole

           9    variety of different things.  And yes, they

          10    all certainly matter.

          11              But from where I sit, the biggest

          12    enemy of innovation is uncertainty.  And to

          13    the degree that we can increase the amount of

          14    certainty that we have, regarding how these

          15    products function, how they will work, how

          16    they will impact patients, that's the thing

          17    that we can -- we can do I think, and that

          18    this guidance and some of the other aspects

          19    of our innovations programs are really trying

          20    to address.  And I think that that's really

          21    something that we are hoping to get out of

          22    this document this day and the discussion




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             13


           1    that will occur.

           2              So I could put up some slides and

           3    start getting into the theorem and some of

           4    these other things, but I am going to leave

           5    that to the people that really understand it

           6    and can explain that to you and discuss it

           7    with some degree of intelligence.  So I am

           8    going to stop here and I am just going to

           9    say, thank you very, very much.

          10              I know that there are people who

          11    have traveled long distances, even from the

          12    great State of New Mexico, to be here today,

          13    and that's great.  And we certainly again

          14    value your interest, we value your

          15    cooperation, and we value your input into

          16    making this day, and making this innovation

          17    process using Bayesian Statistics a success.

          18    So thank you very much.

          19                   (Applause)

          20              MR. CAMPBELL:  Thank you.  Just to

          21    follow up, if you have comments and the

          22    docket is no longer open, because it is after




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             14


           1    August 21st, feel free to send them to the

           2    FDA nonetheless, and if you can't do that

           3    through the docket, you can send them

           4    directly to my e-mail, which is on the first

           5    page of the guidance document.

           6              It is my pleasure now to introduce

           7    Dr. Donna-Bea Tillman, who is the Director of

           8    the Office of Device Evaluation, the main

           9    therapeutic review, pre-market review office

          10    in CDRH.  Donna-Bea.

          11              MS. TILLMAN:  Thanks Greg, and good

          12    morning to you all.  I was saying to Greg

          13    this morning it's a little amusing for me

          14    also to be thinking about addressing a

          15    meeting on Bayesian Statistics, because I am

          16    an engineer and statistics is certainly

          17    something that I have come to later in life.

          18              Actually thanks to Greg, and Garry,

          19    and my colleagues in Biostatistics, who have

          20    taken me from having not a clue about what

          21    the difference between alpha and beta is, to

          22    I think fairly conversant in some of the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             15


           1    fundamental issues.  So I am really excited

           2    about this meeting today too.

           3              There are a lot of challenges that

           4    we are facing in medical devices, and in the

           5    pre-market evaluation of medical devices.  I

           6    don't have to tell you all that medical

           7    device technology is progressing at an ever-

           8    increasing pace.  Devices are getting more

           9    complicated.  The kinds of questions we are

          10    asking medical devices to answer are getting

          11    more complicated.

          12              Patients are demanding more control

          13    over their healthcare decisions.  Clinical

          14    trials are getting more difficult to conduct,

          15    because of issues relating to informed

          16    consent.  And so for all of these reasons we

          17    need to make sure that we get the biggest

          18    bang for the buck out of the clinical trials

          19    that we conduct.

          20              And that they are -- as much

          21    certainty as Dr.  Schultz said that we can

          22    build into them, that they are going to lead




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             16


           1    us to the ultimate aim of getting safe and

           2    effective devices to the market more quickly.

           3    You know, we'd like to be able to do that.

           4              One of the other things that I

           5    think it's important to consider when talking

           6    about Bayesian Statistics at CDRH is the

           7    notion of valid scientific evidence.  Our

           8    definition of valid scientific evidence for

           9    medical devices is fairly broad.  In fact it

          10    enables us to consider many different ways to

          11    reaching the end.

          12              We can do randomized controlled

          13    trials certainly, which are the gold

          14    standards, and which are frankly the only

          15    standard used generally for pharmaceuticals,

          16    but we have more quivers than our arrows, if

          17    you will.  We can consider well-controlled

          18    trials.  We can consider trials with

          19    historical controls; we can consider trails

          20    with OPCs, and even in some cases well-

          21    matched case histories.

          22              And, so our flexible approach to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             17


           1    what constitutes valid scientific evidence,

           2    incorporates also the Frequentist approach to

           3    statistical design and also the Bayesian

           4    approach.  And so we are very excited about

           5    the ways in which we can use Bayesian

           6    Statistics and Bayesian approaches to develop

           7    good scientific evidence, and to, sort of,

           8    allow us to follow our mandate of insuring

           9    the least burdensome path to market for

          10    medical devices.

          11              As we do this, my office, the

          12    Office of Device Evaluation, and my

          13    colleagues at the Office of In-Vitro

          14    Diagnosis Devices rely heavily on our

          15    colleagues in the Office of Surveillance and

          16    Biometrics, our statistical colleagues.  They

          17    are an integral part of the pre-market review

          18    process.

          19              They work with us in the companies

          20    in designing the clinical trials and in

          21    evaluating the clinical trials.  And we

          22    really work as a team.  When it comes to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             18


           1    Bayesian Statistics, in fact we even work

           2    even more like a team.  Because when we use

           3    Bayesian Statistical methods, our clinicians

           4    and engineers play an even larger role.

           5              Because they help us to figure out

           6    when the -- when we are considering whether

           7    something constitutes a valid prior, they

           8    help us to interpret what that information

           9    means in that context of the device being

          10    studied and how it is intended to be used.

          11    And whether in fact that that prior

          12    information is valid.

          13              So Bayesian Statistics has a big

          14    impact on clinical trial design.  I think one

          15    thing that is also important to note though

          16    is that Bayesian Statistics are not a panacea

          17    for poor trial design or for failed studies.

          18    And I know we all don't like to talk about

          19    failed studies, we hope that they don't

          20    happen; you all certainly hope that they

          21    don't happen.

          22              But I think it's really important




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             19


           1    to recognize that the appropriate time in

           2    which to integrate Bayesian Statistical

           3    methods into a trial is at the beginning.

           4    When you are sitting down and thinking about

           5    how to design your trial; that is the time to

           6    think about Bayesian statistics, not at the

           7    end.

           8              And it is also important to note

           9    that Bayesian statistics doesn't substitute

          10    for a good scientific clinical trial design

          11    issues, and so it's not the be-all and

          12    end-all, but when used appropriately, and

          13    when we leverage the scientific expertise

          14    that we have in our own statistical group,

          15    and that exists out there in the statistical

          16    and clinical communities, I believe that

          17    Bayesian statistical approaches are for a

          18    robust way to answer a lot of the challenging

          19    questions that we have to answer in the

          20    pre-market evaluation of medical devices.

          21              And so I'm really excited to be

          22    here today.  I see a number of folks from the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             20


           1    Office of Device Evaluation out there in the

           2    audience, and we are all interested to learn

           3    more about Bayesian statistics and to hear

           4    your comments.  And I thank Greg for inviting

           5    me.

           6                   (Applause)

           7              MR. CAMPBELL:  It is my pleasure

           8    now to introduce Dr. Susan Gardner, who is

           9    the Director of the Office of Surveillance

          10    and Biometrics, in CDRH.  Susan.

          11              MS. GARDNER:  Thank you Greg and

          12    this is now your third welcome to the

          13    meeting, and the third person to speak to you

          14    who is not a statistician.  So I'm going to

          15    make it really quick, so we can get on with

          16    the -- the more statistical part.  First of

          17    all, I would like to congratulate the

          18    Division of Biostatistics for working so hard

          19    on this effort, for getting the guidance

          20    documented, and certainly for going the extra

          21    step, and holding this public meeting.  It is

          22    clearly being well attended and I hear we




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             21


           1    have people also on the phone as well as

           2    people in the audience.

           3              I would also like to congratulate

           4    them for continuing over a long period of

           5    time, to work so hard on this effort.  As Dan

           6    mentioned, we started back in the late '90s,

           7    to really think about Bayesian statistics and

           8    how we could use them in the center.  They

           9    have brought in speakers for short courses,

          10    they have looked hard to hire the right

          11    people with Bayesian backgrounds.  They have

          12    trained their own staff, they've worked hard

          13    to train staff in the center, and they

          14    certainly have taken Bayesian statistics

          15    outside of the center and worked with the

          16    agency and folks outside, so it's really been

          17    a magnificent effort.

          18              Near and dear to my heart, of

          19    course, is we want to think about how we can

          20    use Bayesian statistics in the post-market

          21    arena also.  This ties in perfectly with our

          22    emphasis on the total product lifecycle, so a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             22


           1    couple of areas that we are thinking about is

           2    how we use Bayesian statistics in our

           3    condition of approval studies, studies that

           4    are done after products are approved to go to

           5    market.

           6              We are also looking at Bayesian

           7    data-mining techniques and we have a contract

           8    with a company called Lincoln Technologies

           9    that do Bayesian software development.  We

          10    currently have a database of, close probably

          11    to 2 million adverse event reports, and so

          12    it's a potential rich environment for us to

          13    look for additional signals using Bayesian

          14    software.

          15              And thirdly, we would like to see

          16    if there could be additional coupling again,

          17    sort of, between the pre and post-market

          18    signals and how we tie this together using

          19    Bayesian techniques.  So there is a lot of

          20    exciting work to be done.  We look forward to

          21    the meeting today.  We thank you very much

          22    for coming and I can only echo what Dan and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             23


           1    Donna-Bea said, is that we really look

           2    forward and need your comments to move this

           3    effort forward.  Thank you.

           4                   (Applause)

           5              MR. CAMPBELL:  Thank you, Susan.

           6    Okay.  So what we're going to do now is move

           7    to the part of the agenda where there will be

           8    five presentations -- that's not me.  Oh,

           9    yeah, okay.  That will introduce you

          10    hopefully to the guidance document.  What I

          11    need to do in the way of preface though is

          12    point out that this is a guidance document

          13    for industry and FDA staff.  It's not a

          14    guidance document for statisticians alone,

          15    and that it should be accessible to people

          16    who are not statisticians.

          17              So in this morning's presentations,

          18    you should feel that -- you should understand

          19    what the issues are even though you don't

          20    have formal training in Bayes' Theorem as Dr.

          21    Shultz has just pointed out, so I want to

          22    make sure to emphasize that.  Okay, so for




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             24


           1    the brief presentation that I'm going to

           2    mention, I'm going to point the focus on

           3    exactly what we hope to accomplish in today's

           4    meeting.

           5              Talk a little bit about the history

           6    of Bayesian statistics and CDRH, and then

           7    begin to delve into the document in section 3

           8    and talk about what is Bayesian statistics,

           9    briefly, and why would we want to use it in

          10    the medical device arena.  So the document

          11    has been issued in May 23rd, of this year.

          12    The link there in the second bullet indicates

          13    how you can find it.  You can also find it --

          14    all of you presumably registered on the

          15    website fda.gov/CDRH and you can then find

          16    the document through the page that you

          17    registered on.

          18              The 90-day comment period, as I

          19    mentioned earlier, does end on the 21st of

          20    August, but you're free to send comments

          21    after that as well.  When you do make

          22    comments, you need to know the docket number,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             25


           1    which is Number 2006 D-0191, and the -- if

           2    you just choose to make comments

           3    electronically you need to go to the docket's

           4    website, which is the final bullet on this

           5    slide.

           6              What I need to do is, to remind

           7    people what FDA guidance is and is not.  It

           8    is nonbinding recommendations, it is when

           9    finalized it will represent FDA's current

          10    thinking on this topic, but it doesn't convey

          11    or confer any rights for or on any person and

          12    doesn't bind the FDA nor does it bind the

          13    public.  Furthermore, you can use an

          14    alternative approach, if it satisfies the

          15    requirements with applicable statutes and

          16    regulations, and if you wish to discuss such

          17    an approach, contact us at the FDA.

          18              In addition, this is not a guidance

          19    document, yet, this is a draft, and so it's

          20    even not for implementation, and you will

          21    find that written on the pages of the

          22    document.  It's out for public comment.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             26


           1    After the public comment period ends, we will

           2    get together, a group of us and try to

           3    incorporate all of the comments that people

           4    have made, address them all, and hopefully

           5    come out with a finalized document.

           6              Okay, how do you actually provide

           7    comments to the draft -- you need to submit

           8    them to the docket.  On your notes page you

           9    will find that at the very bottom, in very

          10    small type, the docket number and exactly how

          11    to do that.  You can either send them in

          12    written form to the address indicated there,

          13    "Division of Dockets Management," in

          14    Rockville, or you can submit them

          15    electronically.

          16              Okay, so this draft provides

          17    guidance on statistical aspects of the design

          18    and analysis of clinical trials for medical

          19    devices that use Bayesian statistical

          20    methods.  It is not meant to describe the

          21    content of a Bayesian submission to CDRH, and

          22    it's not all inclusive.  There are many books




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             27


           1    and many papers, which I have not -- which

           2    have not been referenced in the document and

           3    several people have reminded me of that

           4    recently.

           5              Okay, the -- this document reflects

           6    FDA's careful review of what we believe are

           7    the issues related to the use of Bayesian

           8    statistics in medical device clinical trials,

           9    and what we believe would be the least

          10    burdensome way to -- of addressing them.  But

          11    if you have a different view of this, if you

          12    would like to comment on what you perceive to

          13    be a less burdensome approach, you are

          14    certainly welcome to submit your comments.

          15              Okay, I need to talk a little bit

          16    about the history of the FDA, and in

          17    particular the Bayesian initiative.  It was

          18    begun in 1997, and at that time Dr.  Bruce

          19    Burlington who was the Director of the Center

          20    for Devices and Radiological Health, and

          21    Larry Kessler who was then the Director of

          22    the Office Surveillance and Biometrics, and I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             28


           1    together launched this initiative with the

           2    full support of the other office directors,

           3    with support of OD Division Directors and the

           4    branch chiefs.

           5              In 1998, as Dr. Shultz mentioned we

           6    had a medical device -- a Bayesian Medical

           7    Device Workshop in Washington.  Janet Trunzo,

           8    who is sitting in the front row and I jointly

           9    planned that effort.  Janet is with AdvaMed,

          10    which used to be called Health Industry

          11    Manufacturers Association, HIMA.  That

          12    meeting drew a couple of 100 people, was very

          13    successful, and signaled to the medical

          14    device industry that we were amenable to

          15    engaging people who wanted to conduct

          16    Bayesian clinical trials in design and

          17    analysis.

          18              More recently, there was a workshop

          19    that was held between the -- that was

          20    sponsored by the FDA and Johns Hopkins.  This

          21    was a much more general workshop.  The first

          22    workshop in 1998, was focused on medical




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             29


           1    devices.  The second workshop was an effort

           2    for the FDA to think in a public forum about

           3    the use of Bayesian statistics in a more

           4    general setting in that, at that time -- by

           5    that time, CDRH had been doing Bayesian

           6    statistics for quite a while, and there were

           7    questions about how transferable it might be

           8    to biologics or pharmaceuticals.

           9              So that was held at NIH a couple of

          10    years ago.  There were presentations by a lot

          11    of people and I realized after I put this

          12    slide together that I left out a lot of

          13    people whom I should have included, but Janet

          14    Woodcock and Norris Alderson, the associate

          15    commissioners who were involved in planning

          16    and speaking, Bob Temple from CDER, Steven

          17    Goodman from -- and John -- and Tom Louis

          18    from Johns Hopkins, Don Berry and Bailey

          19    Lipscomb and Dan Schultz, and the other

          20    center directors who were there, Scott Zieger

          21    and Susan Ellenburg, Ralph D'Agostino who is

          22    here today, Telba Irony, all played a role.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             30


           1              The second bullet indicates that

           2    there was the proceedings of this meeting

           3    were published in the journal Clinical

           4    Trials, thanks to the editorship of Steve

           5    Goodman, and if you would like to see the web

           6    cast of this workshop, it is still available

           7    at the web address that is there on -- in the

           8    bottom bullet.

           9              So we've been doing Bayesian

          10    statistics and reviewing submissions of a

          11    Bayesian nature for quite awhile now, and so

          12    far it's fair to say that we have a pretty

          13    good track record in the terms of original

          14    PMAs, in terms of PMA supplements that have

          15    used as their primary analysis, a Bayesian

          16    analysis, and this includes stent systems,

          17    heart valves, spinal cage systems and you

          18    will hear about, I think, some of those

          19    examples this afternoon.  We've also had a

          20    number of investigational device exemptions

          21    that have been approved using a Bayesian

          22    design and there are more reviews in process.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             31


           1              Okay, now, let me now turn

           2    attention to section of the document and

           3    begin to try to explain to non-

           4              Statisticians exactly what we are

           5    talking about when we say the words Bayesian

           6    statistics.  A Bayesian approach is a

           7    statistical theory and approach to data

           8    analysis that provides a coherent method for

           9    learning from evidence as it accumulates.

          10              It uses something that a

          11    mathematician who is a minister in England,

          12    over 200 years ago, approved as a theorem.

          13    The man was Thomas Bayes.  The theorem was

          14    published after he died, but it described a

          15    way of combining information, and in

          16    particular, in the Bayesian approach you talk

          17    about information that you have before study

          18    begins and that's captured in what's called

          19    the prior information or the prior

          20    distribution.  You combine it with

          21    information that you have from the current

          22    study and what you end up with is then called




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             32


           1    the posterior distribution, or prior for

           2    before, and posterior for after.

           3              And so you will hear some different

           4    terminology today, and the terminology, if

           5    you're not familiar will sound a little

           6    unusual, but Bayesians refer to two kinds of

           7    statistics, the Bayesian statistics and the

           8    other kind of statistics is called

           9    Frequentist or if you are British you say

          10    Frequentist statistics.  And the -- most of

          11    the statistics that people use who are not

          12    Bayesians, falls into this second bullet of

          13    calling it Frequentist Statistics.

          14              On the idea of being that you can

          15    capture things by observing things over a

          16    long period of time and counting the

          17    frequency by which a particular event occurs.

          18    So the prior distribution is the way you

          19    capture prior information that you have,

          20    before the study begins, you combine that

          21    with the current data, and the way you

          22    combine it with the current data, is through




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             33


           1    something called the likelihood.

           2              So the likelihood and this is the

           3    more statistical thing, if you want, captures

           4    the information that you have in the current

           5    study by itself.  So if you have no prior

           6    information, then everything is captured in

           7    the likelihood and your posterior

           8    distribution, what you learned after you've

           9    done the experiment or the clinical trial is

          10    then entirely represented by the likelihood.

          11              In a Bayesian approach, you have

          12    prior information before you begin and you

          13    then modify that through the likelihood to

          14    come up with what's called the posterior

          15    distribution.  Now, Telba Irony, who will

          16    follow me in a little while, will get into

          17    this in a lot more detail.  So if what I said

          18    made absolutely no sense to you, just capture

          19    that thought and we will come back to it

          20    later.

          21              Why medical devices is the place

          22    for Bayesian methods?  Well, if there is good




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             34


           1    prior information, this could enable the FDA

           2    to reach the same decision for a medical

           3    device submission with a shorter or smaller

           4    trial.  What is implicit in that statement

           5    though is if you don't have good prior

           6    information, you may end up doing a longer or

           7    a larger trial, so it's -- if you want a

           8    wager, it's a bet about how good you might

           9    think your prior information is.  But if

          10    there is good prior information, then this

          11    approach allows for rewarding that.

          12              So the second bullet then -- even

          13    in the case where you have no prior

          14    information, a Bayesian approach can be a

          15    flexible way to look at things like interim

          16    analysis and other modifications.  And you

          17    can use a Bayesian approach, actually, to do

          18    very complicated non- Bayesian calculations

          19    and modelings.

          20              What I should probably say is that

          21    my friends who are not Bayesians would hasten

          22    to point out when they see this slide that




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             35


           1    well, you know, if you're not a Bayesian, you

           2    can do some of this stuff too.  And that's

           3    true, but you have to work a lot harder to do

           4    it.  It's much harder to try to figure out

           5    how to combine information that you knew

           6    about before the trial.

           7              Some people do that informally.

           8    They say "Well, you know, I'm an informal

           9    Bayesian, and I look at what the evidence is

          10    from the trial and then I combine with what I

          11    knew from beforehand and then I arrive at my

          12    conclusion."  The beauty of the Bayesian

          13    approach is, it provides a -- a coherent way

          14    of doing that, and that -- and you start at

          15    the beginning by identifying what your prior

          16    information is, you prescribe exactly what

          17    that is and then you combine it with the

          18    current information to arrive at the

          19    posterior distribution and then make

          20    decisions accordingly.

          21              Why Bayesian methods for medical

          22    devices -- often there is very good prior




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             36


           1    information on medical devices and the reason

           2    is partly, because unlike, for example,

           3    pharmaceutical products, we have a very good

           4    idea of why some medical devices work.  We

           5    understand the mechanism of action.  The

           6    mechanism of action is often physical, and

           7    not pharmacokinetic or pharmacodynamic, and

           8    it's usually local and not systemic.  All of

           9    that suggests that we have an understanding

          10    in many cases about why a medical device

          11    might be effective.

          12              It's important to know note the

          13    second bullet that, if one were to base the

          14    prior distribution mainly on personal

          15    opinions that it may be controversial, and

          16    that if instead, we rely on empirical

          17    evidence, if we rely on quantitative data

          18    from other studies that that will become less

          19    contentious.  And so that is -- is actually

          20    the approach that we've taken in our guidance

          21    document and there will be more about that

          22    later.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             37


           1              Why are we worrying about Bayesian

           2    methods in medical device studies -- well, I

           3    want to return to the least burdensome

           4    provision which the -- the FDA Modernization

           5    Act of 1997, mandated for devices

           6    demonstrating effectiveness and safety.  It

           7    could be that a Bayesian approach, when

           8    correctly employed may be less burdensome

           9    than a Frequentist approach, and so an

          10    additional reason to think about encouraging

          11    companies who might want us to -- to do a

          12    Bayesian trial and design and analysis is

          13    that it may be the -- a less burdensome

          14    approach than a non-Bayesian one.

          15              A good question is, how is it that

          16    we're doing this now, and I think if you look

          17    at where Bayesian statistics was 20 years

          18    ago, people were able to do only very acute

          19    mathematically simple kinds of calculations,

          20    and what's happened in the last 20 or 25

          21    years is the computer revolution, the speed

          22    with which you can do these calculation has




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             38


           1    combined with an algorithm called Markov

           2    Chain Monte Carlo, MCMC, which will come up

           3    later, I think Laura is going to talk about

           4    that, enables people to calculate posterior

           5    distributions for any prior they want.

           6              In 1970, or the '70s, this could

           7    not be done.  People didn't know how to do

           8    it, and even if they did know how to do it,

           9    there wasn't the computing power to do it.

          10    And now the computing power is not an issue,

          11    the algorithm works well and so what this

          12    enables one to do is for any prior

          13    distribution that one could specify, and for

          14    any likelihood that you can write for the

          15    current trial, you can combine that

          16    information and come up with the posterior

          17    distribution and that's a real breakthrough.

          18              Okay, when do you need to come and

          19    talk to CDRH, and the advice in the document

          20    is, come before you start the study.  Come at

          21    the planning stage, we will be glad to talk

          22    to you about the design and the analysis and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             39


           1    if it is required that you're working with a

           2    device that requires an investigational

           3    device exemption, so there is some risk

           4    associated with that device, it is

           5    recommended that you meet with the FDA before

           6    you submit the IDE.

           7              For those people who are working

           8    with devices where you are not required to

           9    have an IDE, you might still choose and I

          10    would encourage you to think about having a

          11    pre-IDE meeting, nonetheless.  And this is

          12    done in many situations, certainly, in the

          13    office of In-Vitro Diagnostics that deals

          14    with diagnostic devices; this comes up quite

          15    a bit.

          16              Okay, with that I will -- it's now

          17    my pleasure to introduce our next speaker,

          18    who is Dr. Gary Gray.  Gary is the Associate

          19    Director in the Division of Biostatistics,

          20    and I will bring up his -- if I can find it.

          21    It's not it -- Gary.

          22              MR. GRAY:  Thank you Greg.  I want




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             40


           1    to just take a few minutes here to talk a

           2    little bit about some of the potential

           3    advantages and disadvantages to using the

           4    Bayesian approach in medical device clinical

           5    trials.  Now, before I get to that I'm going

           6    to talk a little about some of the other

           7    sections of -- the other parts of section 3

           8    of the document, software choices, resources,

           9    sound science, et cetera.

          10              Now, in terms of software, at this

          11    point, we really see that there is two

          12    alternatives that you can choose.  The first

          13    alternative is to use a software package

          14    called WinBUGS.  It's the only really widely

          15    available software that's specifically

          16    designed to be used in Bayesian analysis,

          17    especially hierarchical models.  WinBUGS is

          18    noncommercial.

          19              There are several variants

          20    available called Classic BUGS, WinBUGS.

          21    There is different versions, and there is

          22    other things in development.  Your second




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             41


           1    alternative is to write some custom programs

           2    in a more general language like Fortran or C,

           3    or SAS macros or whatever works, and this

           4    approach is a lot -- is more flexible.  You

           5    can do most anything, but it requires a

           6    little more program review, because it's

           7    custom programming for this specific

           8    application.

           9              Now, we often will because we

          10    --when we start out with software, it's not

          11    really as well tested.  We might -- we would

          12    use one method as a check on another, so

          13    sometimes we end up if the -- if a sponsor

          14    submits us an application that used WinBUGS,

          15    we will write Fortran programs or whatever.

          16              We use one method to check on

          17    another, and that's because independent --

          18    having independent programs increase your

          19    confidence.  We do however, expect that other

          20    specialized softwares will become available,

          21    and of course, we do recommend that you

          22    consult with us before you analyze the data.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             42


           1    Later talks by Telba Irony, Gene Pennello,

           2    Laura Thompson.  They are going to talk about

           3    particular computing issues in more detail.

           4    Not related to the choice of software, but

           5    just more specific computing issues.

           6              Now, as Greg said there are --

           7    there is extensive literature on Bayesian

           8    statistics, which could not possibly go into

           9    a guidance document.  It ranges from basic

          10    introductory type of material to highly

          11    technical, and in Section 3.6 we have a

          12    starting point.  You might use that as a

          13    starting point, if you want to learn more,

          14    and hopefully, this meeting here might

          15    provide a little bit of a starting point for

          16    people as well.

          17              Now, we always want to make clear

          18    that we believe that good clinical trial

          19    design is important regardless of whether you

          20    use a Bayesian or a Frequentist statistical

          21    approach.  Perspective planning of the trial,

          22    randomization, using concurrent controls,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             43


           1    blinding when possible, there is all kinds of

           2    things that go into a good clinical trial

           3    that really are independent of the

           4    statistical methods that you might use.  And

           5    again, there is a large literature on this

           6    kind of -- on this topic that if you want to

           7    explore, you can do that.

           8              Now, just a few comments on

           9    potential benefits and difficulties.  We are

          10    considering these benefits and difficulties

          11    from the point of view of a regulatory

          12    environment.  That is, we are talking about

          13    medical device clinical trials, where the

          14    results are going to be submitted to the FDA

          15    for device approval.  In some sense, our

          16    compared areas are, as what you might call a

          17    simple Frequentist trial.

          18              Now, if you start thinking about

          19    these benefits and difficulties, when you

          20    look at them, you realize that first of all,

          21    whether something is a benefit or not, it

          22    might be depending on the person's point of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             44


           1    view.  And second of all, if some of things

           2    that we, maybe, labeled difficulties might

           3    actually payoff later on actually become

           4    advantages in the long run.

           5              Now, in some sense, I think when we

           6    say difficult, we are talking about things

           7    that require a little more work from both the

           8    sponsor and from the FDA staff.  And a lot of

           9    these issues are going to be covered in the

          10    later talks in a bit more detail.  So the

          11    first potential benefit is sample size

          12    reduction or augmentation.  If you have data

          13    from good prior studies, it can often be

          14    incorporated into your analysis.

          15              Usually, we would use -- the most

          16    common way to do that is to use a

          17    hierarchical model, which is a very flexible

          18    method for incorporating the prior -- your

          19    prior data into your current -- with your --

          20    what you get from your current study and Gene

          21    Pennello is going to talk a fair a bit about

          22    that later on.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             45


           1              And the other thing is that you

           2    might use Bayesian techniques in an adaptive

           3    way to perhaps adjust the sample size for the

           4    trial as it's ongoing, and Telba Irony is

           5    going to talk a bit about that in her talk.

           6    Now, this is one -- this potential benefit is

           7    one that -- depending on your point of view,

           8    you might argue about whether it's a benefit

           9    or not, but that depends, I think, on your --

          10    how you want to use the information.

          11              And as we talked about earlier,

          12    there are in the Bayesian -- using Bayesian

          13    statistical techniques might make it easier

          14    in a lot of ways to incorporate changes to

          15    the trial design, to design in a -- the

          16    possibility of changing the trial as it goes.

          17    And one example of this is -- is a technique

          18    called adaptive randomization, where you

          19    actually change the probability of treatment

          20    assignment through the course of the trial,

          21    and there are some situations where that kind

          22    of a trial design can be very useful.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             46


           1              And as Greg mentioned exact -- this

           2    top -- the benefit called exact analysis, you

           3    can often compute in the Bayesian framework,

           4    the kind of -- the posterior distribution

           5    where as in if you us a frequent approach --

           6    Frequentist approach, it's very difficult to

           7    do the kind of analysis you want to do.

           8              Now, because we are using prior

           9    information and because often the models are

          10    somewhat complex, we feel that the study

          11    might need a lot more extensive preplanning

          12    than otherwise.  You need to have agreement

          13    on the priors, you need to have agreement on

          14    the general form of the model and in the

          15    regulatory environment these kind of issues

          16    are particularly important.

          17              The model building can be

          18    complicated.  The -- we have very flexible

          19    tools to design complicated studies and to do

          20    complicated modeling, so there is a lot of

          21    issues -- a lot of details that require

          22    attention, the prior distributions, how the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             47


           1    covariates are going to be incorporated, what

           2    are the relationships between the various

           3    previous studies, all that kind of stuff.

           4              And because of all this, there is

           5    someone of a specific expertise that is

           6    needed.  It's not off the shelf clinical

           7    trial design.  You need to have an

           8    understanding of Bayesian statistics, you

           9    need to have some expertise in model building

          10    and there is some specific computational

          11    skills.  For some companies that would mean

          12    -- that might mean they need to go out and

          13    hunt for someone who has that specific

          14    expertise.

          15              Now, because we are in the

          16    regulatory setting, especially, the choice of

          17    the priors needs to be justified.  At FDA

          18    that means that the commissions, the

          19    statisticians, the engineers all have to

          20    agree that whatever prior information we are

          21    using is appropriate to be incorporated in,

          22    with the data from the current study.  If the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             48


           1    device comes through an advisory panel, they

           2    might need to be -- to agree that it's

           3    appropriate prior information to be used.

           4              And then finally when you get out

           5    into the medical community, that's another

           6    audience who has to understand and agree that

           7    the choice of priors was reasonable.  The

           8    labeling might be -- the labeling will most

           9    likely be different from standard.  You might

          10    consider that a difficulty or an advantage,

          11    depending on your point of view, but the

          12    labeling might be different.

          13              In terms of checking the

          14    calculations, there is a lot more flexibility

          15    and that means there is a lot more potential

          16    for misunderstandings, and the software is

          17    complex and the development of the -- of a

          18    commercial software product certainly is not

          19    -- it's not mature.  Most of the -- many a

          20    times the calculations that we've used are

          21    based on simulations and there is -- you have

          22    to be sensitive of issues about convergence.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             49


           1              So because of all these reasons, we

           2    will carry out detailed reviews of Bayesian

           3    -- the submissions that uses Bayesian

           4    statistics.  And that is a lot easier if we

           5    see the data and the programs in the

           6    electronic format.  Okay, finally, our last

           7    potential difficulty, if you want to call it

           8    that is the differences -- there might be a

           9    difference between a Bayesian and a

          10    Frequentist analysis.

          11              The two methodologies, any two

          12    statistical analyses might give different

          13    results.  And from our point of view in the

          14    regulatory setting, that is just -- that's a

          15    good reason why we would like to see the

          16    analysis pre- specified before the study is

          17    completed or even it starts.  Okay, so thank

          18    you for your attention, for these few

          19    minutes, and next I'm going to introduce

          20    Telba Irony, who is going to talk about

          21    flexible trial design.

          22              MS. IRONY:  Okay, I appreciate you




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             50


           1    all being here and my talk is brief and I am

           2    going to -- let's see -- going, that's my

           3    outline.  First I will give you an idea about

           4    what's Bayesian statistics in a nutshell

           5    very, you know, quick idea.  Then in the

           6    second part of my presentation, I am going to

           7    switch to the regulatory setting and I will

           8    discuss how do you plan a Bayesian trial,

           9    highlighting the flexibility of the approach.

          10              I will also discuss appendix 9.4

          11    where the simulations to obtain -- operating

          12    characteristics of the design I described.

          13    The simulations are usually performed at the

          14    planning stage of the trial and that's I'm

          15    linking the appendix with the section 5.

          16    Finally, I will briefly talk about the

          17    analysis of Bayesian flexible design and you

          18    may notice that in -- the sections in the

          19    guidance correspond to the numbers in the

          20    slides, so the people that want to follow my

          21    presentation used in the guidance will have

          22    it easier that way.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             51


           1              Okay, what's Bayesian statistics

           2    and what's the statistics in general.  We'll

           3    all say that statistics is the discipline

           4    that allows one to learn from evidence in the

           5    presence of uncertainty.  In the Bayesian,

           6    we'll go further and we'll say that one wants

           7    to learn about a known state of nature in

           8    order to make decisions.  In other wards, the

           9    essence of the Bayesian approach is causally

          10    related to decision theory.

          11              Usually, one wants to learn about

          12    the known state of nature and we call our

          13    jargon is to say that it's a parameter.  And

          14    a Bayesian will say that probability measures

          15    one's uncertainty about the state of nature

          16    or about a parameter.  And they will say that

          17    that is the only valid way to measure

          18    uncertainty.  For instance, a p- value

          19    wouldn't be a valid way to measure one's

          20    uncertainty about -- in a known state of

          21    nature.  In any case for everybody, Bayesian

          22    or no Bayesian, a way to change probability




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             52


           1    is to use Bayes Theorem.

           2              There is no controversial issues

           3    about the use of Bayes Theorem, it's just a

           4    mathematical formula and there is no way to

           5    get around of it.  You might choose to use it

           6    or not, but if you choose to use it, that's

           7    the way of doing it.  So let me give you an

           8    example.  Let's say you have a clinical trial

           9    to evaluate a new treatment and let's say

          10    that we want to know what's the treatment

          11    effect.  We call it delta and that's

          12    basically our known state of nature.  We want

          13    to know about delta.  One will assign a prior

          14    probability to delta, why because we don't

          15    know.  It is an unknown state of nature, so

          16    we will assign a prior probability.  In other

          17    words, we'll tell "What is likely, what kind

          18    of values for delta are likely".

          19              We want to learn about delta; so we

          20    will perform a clinical trial and we will

          21    obtain data from this clinical trial.  As

          22    Greg mentioned before this data is described




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             53


           1    but what we call the likelihood function.  Is

           2    the function that relates the data from the

           3    trial with our unknown state of nature.  Our

           4    objective is to come up with a posterior

           5    probability of -- on delta.  In other words,

           6    we will change the likely values of delta and

           7    we will change based on the evidence provided

           8    by the clinical trial.

           9              The question is, what's the

          10    probability that the treatment effect is

          11    large enough to be clinically significant.

          12    That's the question we want to answer at

          13    least at the FDA and most of the time.  So I

          14    give you some examples of what are prior

          15    distributions and how it expresses one's

          16    uncertainty about delta.  For instance, I'm a

          17    statistician, so that's my uncertainty about

          18    a -- the values of delta.  I call it's a

          19    non-informative prior that means for me, that

          20    -- I am not a physician, any value between

          21    zero and one is equally likely.  So that's

          22    represented for -- by this flat line that I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             54


           1    am saying that value 0.2 is as likely as a

           2    value 0.4 as likely at value 0.69 and so on.

           3              Now, I can go to my friend Barbara,

           4    a clinician, and she knows much more about

           5    how the patients behave and how the

           6    treatments might be effective or not and she

           7    might come to me and say "Look, I am an

           8    expert and it's much more likely that this

           9    effect is between 0 and 0.5 -- then it's

          10    0.8."  So she might want to represent her

          11    prior information through these kind of

          12    function.  You see, that you have a bump

          13    between 0 and 0.5.  That means there is more

          14    probability that the effect is there, then

          15    it's close to 1.

          16              Now, I mentioned the likelihood

          17    function and I just want to reiterate that

          18    that's the way we represent the data and

          19    relate the data that we obtain in the trial

          20    to the -- our unknown state of nature or our

          21    parameter of interest.  And then we arrive at

          22    the posterior distribution which is the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             55


           1    probability of each possible value of the

           2    parameter conditional on the prior

           3    distribution that we used and on the evidence

           4    provided by the observed data in the clinical

           5    trial.

           6              So we like to use this picture.

           7    Let's say I use Barbara's prior and I, you

           8    know, conduct a clinical trial, get data.  So

           9    I put them both in my green box or green

          10    circle called Bayes Theorem and I come up

          11    with a posterior distribution and you see

          12    that, you know, given the data, this

          13    posterior becomes a little sharper around

          14    0.2.  I will say that, you know, have a lot

          15    of probability between 0.1 and 0.3.  Now I go

          16    to my clinician say "Oh, it's not enough

          17    information, you know, I am still uncertain

          18    and a range from 0.1 to 0.3 is not good

          19    enough.  I want to be more precise, I want to

          20    learn more".

          21              So what do you do?  Today's

          22    posterior becomes tomorrow's prior.  So you




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             56


           1    go to the next step and you use yesterday's

           2    posterior which is today's prior.  You

           3    combine with more data and you use against

           4    Bayes Theorem and you get to a sharper

           5    posterior.  The more information you gather

           6    from your trial, the sharper your posterior

           7    distribution may become.  It doesn't always

           8    become that way because you might have, you

           9    know, what we call not-so- good prior or may

          10    be not-so-good data.  So if enough data are

          11    collected, the prior information is washed

          12    away.  In other words, if you have a huge

          13    trial it doesn't matter what's your initial

          14    prior information was.

          15              Now I saw -- here we have a

          16    presentation of a posterior distribution and

          17    I am using this picture to show what that

          18    credible Interval is.  And later on Gene

          19    Pennello is going to talk a little bit more

          20    about the credible interval.  The credible

          21    interval is the interval that provides an

          22    area under the curve, in this case, of 95




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             57


           1    percent.  And with this interval we can say

           2    that the probability that the treatment

           3    effect is between 0.15 and 0.25 is 95

           4    percent.  Be aware that this is not a

           5    confidence interval, the usual confidence

           6    interval.  People mixed them up and people

           7    think that the confidence interval is

           8    actually the credible interval.  But the

           9    concepts are completely different.  They are

          10    not the same.

          11              Now, section 4.5 in the Guidance

          12    talk about predictive distribution and while

          13    bring in the predictive distributions here

          14    now because they are actually a special type

          15    of posterior distribution.  In other words,

          16    once we are, let's say, in the middle of the

          17    trial, the remaining results for the patients

          18    that are still going to be recruited or are

          19    in the trial, they are also unknown state of

          20    natures.  So we can compute the probability

          21    of the future or unobserved events, given the

          22    data that we already observed this far.  The




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             58


           1    predictive distribution can also give the

           2    probability of the outcomes of a future

           3    patient in mid-course of a trial and it can

           4    give the probability that a new patient will

           5    be successful when implanted with a devise

           6    once the device is on the market.

           7              What are the uses of the predictive

           8    distribution?  Well, everybody wants to

           9    predict things.  So you might think that

          10    there are lots of uses for that -- that kind

          11    of distribution.  For instance, the

          12    predictive distribution might help us to

          13    decide when to stop a trial.  In other words,

          14    if you can predict that the probability that

          15    the trial would be successful is high enough

          16    why would we continue.  We might be able to

          17    stop the trial at mid-course.

          18              The predictive distribution also

          19    helps physicians and patients to make

          20    decisions about the use of a device

          21    intervention.  And if we can place this

          22    predictive distributions at the labeling, you




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             59


           1    know, the physicians that will read the

           2    labeling will say, okay, based on this

           3    prediction, I can tell you that the

           4    probability that you will be successful, if

           5    you get this device, would be, let's say 80

           6    percent.

           7              That will help the patient to

           8    decide whether or not he will want to get

           9    that device implanted.  The predictive

          10    distribution can also predict a clinical

          11    outcome from a valid surrogate.  And it can

          12    adjust the trial results for missing data,

          13    again missing data are unknown states of

          14    nature.  So we can use predictive

          15    probabilities to impute this unknown state of

          16    nature.  Now, I am going to talk a little bit

          17    about exchangeability and -- I will like to

          18    tell you something.  This is a very hard

          19    concept and especially if you want to make it

          20    technical and mathematical.

          21              If you want to be precise, nobody

          22    gets it.  And if you want people to get it,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             60


           1    you might be incorrect.  So we particularly

           2    welcome comments to the docket with respect

           3    to this section 4.6.  If you have anything to

           4    help us with this respect, please said it to

           5    the docket, we will appreciate.

           6              Having said that, I am going to

           7    talk about exchangeabilities through some

           8    examples.  So I will say that patients in a

           9    trial are exchangeable if they belong to the

          10    same population of patients.  So if patients

          11    in a trial are exchangeable with the patients

          12    in the population for which the device is

          13    intended, then the trial can be used to

          14    estimate safety and effectiveness of the

          15    device for the intended population.

          16              If they aren't exchangeable, we

          17    shouldn't use that trial to make that

          18    estimate.  And we can go further and say that

          19    two trials or two samples are exchangeable

          20    and now we are talking about exchangeability

          21    of samples or of trials.  If they are

          22    elements of the same super- population of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             61


           1    trials and this concept is important because

           2    we are required this kind of exchangeability

           3    to use hierarchical models.  Gene Pennello is

           4    going to talk them next to me.

           5              Now, section 4.7 talks about the

           6    likelihood principle.  That's a central

           7    principle to the Bayesian approach and it

           8    says that all information about a known

           9    quantity of interest, of the parameter

          10    contained in the data from our clinical trial

          11    is in the likelihood function.  That means

          12    that if there are changes to experimental

          13    design that don't change the likelihood

          14    function.  For instance, increasing the

          15    sample size or reducing the sample size.

          16              Then the posterior distribution

          17    remains the same and the conclusions should

          18    not be modified.  So the reliance on the

          19    likelihood principle makes the Bayesian

          20    approach intrinsically flexible.  The idea is

          21    to constantly update the information in the

          22    light of evidence accumulated this far.  So




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             62


           1    that Bayes is in essence of the Bayesian

           2    approach.  Now because of the likelihood

           3    principle a trial may be altered in several

           4    ways without changing the likelihood

           5    function.

           6              For instance, if a modified sample

           7    size that shouldn't change the likelihood

           8    function if you perform interim looks for

           9    stopping early, that shouldn't change your

          10    likelihood function.  If you change the

          11    randomization rate in mid-course, that should

          12    not change the likelihood function and it

          13    applies to adaptive designs in general.

          14              I have to warn you however, there

          15    are some changes in design that will change

          16    the likelihood function and in that case we

          17    have to correct even the Bayesian approach.

          18    Now in a nutshell, what's the difference in

          19    approaches between the Bayesian and the

          20    Frequentists?  Well, for Bayesian approach,

          21    the inferences are based on the posterior.

          22    It follows the likelihood principle and I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             63


           1    will say that inferences are based on what we

           2    call the parameter space.  The unknown is the

           3    parameter, is not the data.  The data is

           4    observed, is not unknown.

           5              Because the inferences are based on

           6    the parameter space, pre specification of the

           7    sample size is not essential because if we

           8    change the sample size we don't change the

           9    parameter space.  Now the traditional

          10    Frequentists approach, the inferences are

          11    based on p- values.  It doesn't follow the

          12    likelihood principle and inferences are based

          13    on the sample space.

          14              The p-value is closely related with

          15    the sample space and I try to explain that in

          16    a nutshell, the traditional approach will

          17    pretend that they know the parameter and they

          18    will verify how likely would be to obtain the

          19    data that has been obtained, where at the

          20    parameter they wanted to be specified into

          21    the hypothesis.  So that's what they mean by

          22    the sample space.  The data is the unknown




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             64


           1    quantity and the parameter or the -- the real

           2    unknown quantity is thought of as it was not.

           3    A little complicated.

           4              Anyway, now I am going to section 5

           5    and I am going to talk to you about how to

           6    plan a Bayesian clinical trial.  And I will

           7    start to say that Bayesian trials require a

           8    sound clinical trial design.  One has to pre-

           9    specify the objectives of the trial, the

          10    endpoints to be evaluated the conditions

          11    under which the trial will be conducted, like

          12    inclusion, exclusion criteria, the population

          13    to be investigated, the plan statistical

          14    analysis that will be performed.  And one

          15    should worry about randomization,

          16    minimization of bias, masking, the type of

          17    analysis that will be employed and et cetera.

          18    And that's of course by -- valid for both

          19    Bayesian and Frequentist traditional trials.

          20              One should select the relevant

          21    endpoints, or the relevant parameter, or

          22    interest.  And they should be clinically




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             65


           1    relevant, they should be directly observable,

           2    if possible, and they should be of course

           3    related to the device claims.  One should

           4    also plan to collect the information on

           5    covariates that could influence the patient

           6    outcome, because later on we might have to

           7    adjust the results for these covariates.  We

           8    have to be careful by choosing a controllable

           9    for a comparison, and again the Gold standard

          10    is always a clinical trial with concurrent

          11    randomize controls -- randomize clinical

          12    trial concurrently controlled.  At last we

          13    goes, clinical trials are the ones that have

          14    self controls, or historical controls et

          15    cetera.  Active controls versus inactive

          16    controls.  Interestingly enough, Bayesian

          17    clinical trails are specially suited to

          18    demonstrate not only that a new device is not

          19    inferior to an active control, but also

          20    superior to an inactive control.  Of course,

          21    you can do that in the traditional trial as

          22    well.  But you know, you have special




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             66


           1    mechanisms in the Bayesian approach, that

           2    will make you be able to imbed both possible

           3    claims in the planning of the trial.

           4              I am not going to discuss how to

           5    use the prior informations, distributions,

           6    and hierarchy modus.  Dr.  Gene Pennello is

           7    going to do it right after my presentation.

           8    But I am going to talk about sample size

           9    determination.  And I am going to say that a

          10    person's size is a trial to gather enough

          11    information, to make a decision without

          12    wasting resources, or putting too many

          13    patients at risk.  The thing is that the

          14    sample size in a trial depends on several

          15    things, like a prior information, like the

          16    likelihood model that they are using.  That

          17    like the viability of this sample on the

          18    decision criteria, how certain do you need to

          19    be in order to make the decision.  Like for a

          20    very dangerous device, I would say, you might

          21    be more rigorous.  You might want to have

          22    more information for a device that's not as




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             67


           1    risky, you might be more lenient, and say,

           2    "Okay, I don't need to be that certain,

           3    because I don't see any risks with this

           4    device."  But the problem is that at the

           5    planning stage we are uncertain about several

           6    assumptions that we have to make in order to

           7    determine the idea of sample size for a

           8    trial.

           9              In particular, the viability of the

          10    sample is an unknown state of nature; we

          11    don't know it in the beginning of the trial,

          12    we might guess.  But if we are good Bayesians

          13    we will just guess as to we are going to

          14    assess a probability distribution, to that.

          15    It's an unknown state of nature.  So as a

          16    consequence, it is natural not to have a

          17    fixed sample size when using the Bayesian

          18    approach.  Of course, one might choose to use

          19    a fixed sample size, but, it's not required

          20    at all.  So how do you determine a sample

          21    size?  As I said, traditional Frequentists

          22    need to predetermine the sample size, because




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             68


           1    they work on the sample space.  The

           2    Bayesian's sample size is may be continuously

           3    updated as information is accrued.  So

           4    instead of the sample size, Bayesians tend to

           5    specify a stopping criteria based on the

           6    desired amount of information about the

           7    parameter of (off mike).  So you don't know

           8    when you're going to reach the desired amount

           9    of information, because of several unknowns

          10    in the trials, but you say, "I am going to

          11    stop the trial, when I have short enough

          12    credible interval, or when I'm certain enough

          13    about the parameters I need to make a

          14    decision."  At any point in a Bayesian trial

          15    the expected additional sample size needed to

          16    stop can be computed.

          17              However, you know, we make some

          18    recommendations for practical purposes, and

          19    for regulatory purposes.  One should decide

          20    in advance, what will be the minimal sample

          21    size considering both safety and effective

          22    endpoints to verify the mode of assumptions,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             69


           1    the prior distribution, and also to control

           2    Type I error.  One should also decide in

           3    advance the maximum sample size.  For ethical

           4    reasons, you don't want to put too many

           5    patients at risks.

           6              If the device doesn't work, for

           7    regulatory reasons, we need to limit the

           8    amount of patients that are used in a

           9    clinical trail, and also from the sponsor's

          10    point of view for economical point of

          11    reasons.  Financially, you might not have

          12    enough money to go for ever to find out the

          13    truth.  Now, due to the use of prior

          14    information, and flexible designs, a Bayesian

          15    trail will potentially end up with a smaller

          16    sample size, and I said potentially here.

          17    However, there is no guarantee that this is

          18    going to happen, and some Bayesian trials may

          19    have end up with a larger, being larger than

          20    a traditional trial.

          21              Interesting enough, this is a

          22    matter of probabilities again.  In our




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             70


           1    experience there is a higher chance that a

           2    Bayesian trial with a sound design will tend

           3    to be smaller.  So, how do we assess what we

           4    call the operating characteristics of a

           5    Bayesian design?  As I said before, the

           6    likelihood principle allows for flexibility

           7    in the Bayesian trial designs.  And often the

           8    Bayesian trials are designed with the minimal

           9    sample size, several interim looks in a

          10    maximum sample size.  At each interim look

          11    the trial may stop either for success or for

          12    futility.  I'll explain this better in the

          13    next slide that is referring to appendix,

          14    sorry, still going to talk a little bit about

          15    the operating characteristics.  For

          16    regulatory purposes, we need to evaluate the

          17    operating characteristics of the trial

          18    design.  What does it mean?  We need to get

          19    the probability of Type I error.  What is a

          20    Type I error?  Type I error is an error that

          21    you make when you approve a device that

          22    should not be approved.  So we want to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             71


           1    control that probability, want to make it

           2    small.

           3              Probability of Type II error is

           4    also important, particularly from the

           5    sponsor's point of view.  That's the

           6    probability of not approving a device that

           7    should be approved.  We want to balance them

           8    both.  But particularly, we as the FDA want

           9    to control the probability of Type I error.

          10    At each clinical trial run could also get the

          11    sample size distribution.  Again, simple size

          12    is a random quantity.  There are several

          13    values that are likely, so talk about sample

          14    size distribution, and what is the expected

          15    sample size for the trial.  Now, I am giving

          16    you more details that are in appendix 9.4 for

          17    people that are following -- using that

          18    guidance document, this is on page 34.

          19    Simulation results, to assess the operating

          20    characteristics of the design should be

          21    presented at the planning stage, at the IDE

          22    stage.  The simulated trials should mimic the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             72


           1    proposed trial by considering the same prior

           2    distribution that's going to be used, the

           3    same sample size, the same interim analysis,

           4    and any possible modifications of the trial

           5    in mid-course.  Now, these will help us -- by

           6    the simulation, will help us to assess Type I

           7    error.  Now, how large could the Type I error

           8    be?  Well, it depends on the submission, and

           9    it depends also on the confidence we'll have

          10    on the prior distribution.

          11              And I guess, you have to come to

          12    the FDA and discuss that with us as well.

          13    So, how do you assess Type I error?  You fix

          14    your parameters in the simulation, add the

          15    borderline values for which the device should

          16    not be approved.  It should almost be

          17    approved, but not quiet.  And then you

          18    simulate thousands of trials, using the same

          19    design that you use at the same success

          20    criteria that you are proposing.  The

          21    proportion of those simulated trials that are

          22    successful, that will lead to an approval




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             73


           1    will give us the Type I error rate.  So we

           2    want that to be small.  How do you assess

           3    Type II error rate, and consequently Power,

           4    you know, the power is 1 minus the Type II

           5    error rate.  So you fix the parameters at

           6    possible values for which the device should

           7    be approved.  In other words the parameters

           8    that the sponsors think are the reasonable

           9    ones, are the true ones.  And then you

          10    simulate thousand of trials, and you see how

          11    many of these trials that are simulated are

          12    successful, how many are not successful.  The

          13    proportion of unsuccessful trials will give

          14    you the Type II error rate.  It's easy to

          15    obtain the sample size distribution, and

          16    expect the sample size as you do the

          17    simulations.  You'll just estimate, you know,

          18    what are the sample size for each of the

          19    thousand of trials that are simulated.

          20              Severe age recommends that you

          21    simulate several likely scenarios.  This is

          22    an example of a table that you get when you




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             74


           1    perform simulations.  Let's say that you are

           2    simulating, otherwise for which you have a

           3    target success rate.  Okay, I put the target

           4    in the yellow line.  So, you simulated a 1000

           5    trials using that target rate, it's the

           6    fourth row in that table.  You simulated a

           7    1000 trials, therefore the target laid, you

           8    are successful in 9.2 percent of these 1000

           9    trials.  Your expected sample size, to reach

          10    that were 786 patients.  This is actually a

          11    large sample size, because you are right at

          12    the target.  But assume for instance that

          13    your adverse event rate is very low, is like

          14    the last row, is 0.06.  So it's a very good

          15    device, very small adverse event rate.  Okay,

          16    you simulate the 1000 trials with that rate,

          17    and you see that 99.7 percent of these trials

          18    are successful.

          19              Moreover, because the device is so

          20    good, and the adverse event rate is so small,

          21    you can reach this conclusion very quickly.

          22    So your expected sample size for this




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             75


           1    situation is only 327.  You perform the

           2    interim analysis, and then you quickly

           3    realize that the adverse event rate is very

           4    small.  Now, you go to the beginning of the

           5    table.  Let's say your device is really not

           6    good, and the adverse event rate is 0.2, is

           7    20 percent.  It's horrible device.  You

           8    simulate your trial, you simulate actually a

           9    1000 trials.  And in that case, only 0.6

          10    percent are successful.  So that's your Type

          11    I error rate, is a very small Type I error

          12    rate.  But see, it's uncertain, so you might

          13    commit an error even when your device is

          14    horrible.  As you go close to the target, for

          15    instance, 0.11, that's what we call the

          16    borderline situation.  If you simulate

          17    trials, in which the adverse event rate 0.11,

          18    you know, very close to the target, but

          19    you're still wasn't in the target.  So your

          20    percentage of successful trials will be about

          21    49 -- 0.49 percent.  That's what we call the

          22    Type I error rate, is that -- usual frequent




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             76


           1    is 5 percent.  And we'll be satisfied with

           2    such a trial.

           3              Now, how do you control Type I

           4    error rate?  Let's say you device a trial,

           5    you have an experimental design, and you

           6    perform the simulations, and your Type I

           7    error rate is too high.  How do you remedy

           8    that?  Well, you might increase the posterior

           9    or predictive probability threshold value, to

          10    define a successful trial.

          11              Instead of using, let's say 95

          12    percent, you use 96, 98 percent, et cetera.

          13    You might increase the number of patients

          14    before you perform your first interim look,

          15    or you might discount the prior information

          16    that you use.  You might reduce the number of

          17    interim analyses (off mike) increase the

          18    maximal sample size, or any combination of

          19    the above.  And you perform that by trial and

          20    error.  You know, you try to simulate until

          21    you reach an agreement with the FDA.  Now,

          22    how do you analyze a Bayesian trial?  Well,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             77


           1    as I said before, use sometimes predictive

           2    distributions, the predictive probability to

           3    stop -- to decide a trial, you can use it for

           4    success, or for futility.  Important here is

           5    the exchangeability issue to use predictive

           6    probability.  You have to assume that the

           7    future patients in a trial are exchangeable

           8    with the patients you observed thus far.

           9              You are going to predict outcomes

          10    for future patients.  Again you use the

          11    exchangeability assumption, and that's

          12    crucial, and that has to be discussed with

          13    the clinicians.  And you might use the

          14    predicted probabilities in the labeling; we

          15    have used that at the CDRH.  You might use

          16    predictive probabilities to predict, or

          17    impute missing data.  And if your assumption

          18    that the patients are missing at random is

          19    not valid, you have to perform what we call

          20    sensitivity analysis.  In other words we

          21    impute your worst-case scenarios, best-case

          22    scenarios, and you see how sensitive your




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             78


           1    conclusions will be to not -- to not being

           2    compliance with the assumptions of missing at

           3    random.  We have used a predictive

           4    probabilities also to predict the clinical

           5    outcome for a -- from a surrogate.  Let's

           6    give you an example.  Let's say that patients

           7    have measurements, a true follow up, if he is

           8    -- it's one, let's say a 12 months, and

           9    another one a 24 months.  Let's say that we

          10    want to make gradations for the second follow

          11    up visits of the patients, before all

          12    patients reach 24 months in the trial.

          13              For that some patients will have to

          14    have results for both follow up visits.  In

          15    other words we have to have a critical mass

          16    of patients that have results for both 12

          17    months and 24 months.  And there must be some

          18    kind of correlation between the early and

          19    later results.  If there are -- if there is

          20    correlation we can make these kinds of

          21    predictions.  In order to perform interim

          22    analysis in a Bayesian trial, they should




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             79


           1    always be pre-planned in a regulatory

           2    setting.  The simulations, as I said, should

           3    be performed in advance at the IDE, or

           4    planning stage.  Stopping may occur when the

           5    procedure probability of the hypothesis of

           6    interests is large enough, and stopping may

           7    occur by the use of predictive probability of

           8    a successful trial.  If the probability is

           9    large we'll have a success, if the

          10    probability is very small we should stop for

          11    futility.  Decision analysis should consider

          12    the cause of decision errors.  In our case we

          13    want to control Type I error, and possibly

          14    Type II error.

          15              That concludes my presentation.

          16    Now we have a break for 20 minutes, and

          17    following the break Dr. Gene Penello will

          18    continue with the presentations.  Thank you

          19    very much for your attention.

          20                   (Applause)

          21                   (Recess)

          22              MR. CAMPBELL:  Okay, I think we




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             80


           1    should get started in just a minute or so.

           2    Okay, thanks, I need to make one brief

           3    announcement and that's, that of course, in

           4    your packet is the copy of the Bayesian draft

           5    guidance document.  The overheads that are

           6    being presented this morning are not in your

           7    packet, but they are on the web.  If you go

           8    to the registration site, you can find them

           9    and print them off.  So if you were thinking

          10    that you had to furiously take notes on some

          11    of the presentations, you may not need to do

          12    that.

          13              What I would like to do is

          14    encourage people to think about comments and

          15    questions that they might have.  We hopefully

          16    will have a little time at the end of this

          17    morning and certainly a lot of time, this

          18    afternoon.  So as we -- as you hear these

          19    presentations and you have questions, you

          20    might make notations to yourself and prepare

          21    for questions and comments later today.

          22              And, so now, it is my pleasure to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             81


           1    introduce Gene Pennello, who is the team

           2    leader in the Diagnostic Devices Branch in

           3    the Division of Biostatistics, and he is

           4    going to tell us all about planning a

           5    Bayesian trial using prior distributions and

           6    hierarchical models.

           7              MR. PENNELLO:  Thanks, Greg.  So,

           8    in this part of the meeting, Laura Thompson

           9    and I are going to cover basically the rest

          10    of the guidance that hadn't just been covered

          11    yet, and if you could -- if you haven't

          12    realized it already, we are, kind of, going a

          13    little bit out of order.  So I hope, that is

          14    not too confusing to you, but our -- all of

          15    our slides, which you can get from the web,

          16    tell you which sections we are talking about.

          17              So the outline of my talk will be,

          18    I am going to cover some of the sections in

          19    -- subsections in section 5 and section 6,

          20    that weren't covered by Telba and subsection

          21    5.5 on the prior distribution.  And

          22    specifically, talk a little bit about




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             82


           1    informative prior distributions versus

           2    non-informative prior distributions.

           3              Subsection 5.6 on hierarchical

           4    models, which is a flexible way to combine

           5    prior information with current study data.

           6    Section 6 on analyzing the Bayesian trial,

           7    posterior distribution summaries, hypothesis

           8    testing, interval estimation, and a little

           9    bit about post-market surveillance using

          10    Bayesian methods.

          11              So, the initial information about

          12    the endpoints of your study is characterized

          13    in the Bayesian submission by the prior

          14    distribution.  So prior to any collection of

          15    the data in the study, initial uncertainty

          16    about the endpoints, which I'm going to be

          17    calling parameters in this talk, of interest

          18    is quantified by specification of

          19    probabilities for their possible values.  And

          20    this --as you've heard, this initial

          21    probability distribution is called the prior

          22    distribution.  The study designed for every




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             83


           1    Bayesian trial includes specification of the

           2    prior distribution.

           3              Now you can classify prior

           4    distributions by whether they are informative

           5    or non-informative, and let us talk about the

           6    informative prior distributions first.  These

           7    are formulated from prior information and

           8    they can -- potentially they can decrease the

           9    sample size required for a trial, because

          10    they inform you -- in addition to the trial

          11    results, they provide additional information

          12    to -- about study parameters.

          13              Possible sources of prior

          14    information include -- and these are just

          15    examples -- clinical trials conducted

          16    overseas, that may provide good prior

          17    information for a U.S. study, patient

          18    registries, clinical data on very similar

          19    products.  Perhaps you've designed a -- or

          20    you've changed your device slightly and it's

          21    a new model, but it is very similar to the

          22    previous model that has been approved, you




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             84


           1    might consider the data on the previous

           2    model, this prior information for the new

           3    model, pilot studies.  Also, historical

           4    controls could be considered as prior

           5    information for concurrent controls.

           6              The FDA should agree with your

           7    choice for an informative prior distribution.

           8    And to that end, you've heard Greg and others

           9    talk about this.  I think it's -- we

          10    encourage you very much to come in early to

          11    talk about your Bayesian analysis plan, and

          12    especially your proposed prior information.

          13    You need to make sure that the prior

          14    information is legal to use.

          15              There might be good prior

          16    information out there that is proprietary to

          17    another company, and there are some legal

          18    restrictions to using, for example,

          19    information in PMAs from other companies.  So

          20    that's something you need to think about.  We

          21    encourage you to hold a pre-IDE meeting with

          22    the FDA to come to an agreement on what prior




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             85


           1    information is scientifically valid for the

           2    Bayesian trial.

           3              And you need, you should submit

           4    your prior information as part of the formal

           5    IDE process when the -- when an IDE is

           6    required to be approved to conduct your

           7    trial.  Now, informative prior distributions

           8    can be based on data from previous studies

           9    and I am calling that a databased prior

          10    information or databased prior distribution.

          11    To be appropriate, the data from the prior

          12    studies, the prior studies on which -- for

          13    which the data are coming from usually need

          14    to be similar to the current study.

          15              So what we look at, when we look at

          16    proposed prior information from prior studies

          17    is how similar the prior studies are to the

          18    current study that you're going to conduct

          19    and we're looking at a number of factors and

          20    I've listed some here, the devices used

          21    should be similar or ideally the same.  The

          22    objectives of the two studies should be




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             86


           1    similar, the endpoints, the protocol, patient

           2    population, patient management,

           3    investigational sites.

           4              The prior studies shouldn't be, you

           5    know, 20 years ago.  It should be close to

           6    the -- in time to the current study that is

           7    going to be conducted, and also physician

           8    training experience and technique.  And it

           9    could be other factors I haven't listed here.

          10              Now, when we look at the similarity

          11    of the prior studies to the current study,

          12    that's really not a statistical evaluation,

          13    and we rely very heavily on our scientists,

          14    our clinicians and our engineers, to -- to

          15    make that determination to see whether the

          16    prior studies are appropriate and

          17    scientifically valid for considering as prior

          18    information going into a new study of a new

          19    device.  So they play a key role in

          20    determining if the prior studies are similar

          21    to the current study and therefore in

          22    determining if they are appropriate as prior




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             87


           1    information.

           2              Also, ideally, the studies used as

           3    prior information are representative of the

           4    possible sources of prior information that

           5    could have been selected.  You should avoid

           6    systematically excluding non-favorable

           7    studies to avoid introducing selection bias

           8    into the Bayesian statistical analysis.

           9              Now, you could have an informative

          10    prior distribution that is not based on data

          11    from prior studies, but are based merely on

          12    opinion from experts.  And it could be a very

          13    good opinion, but this is -- really creates a

          14    problem in the regulatory setting.  Approval

          15    of a device could be delayed or jeopardized,

          16    if an FDA advisory panel, for example, or

          17    other clinical evaluators, do not agree with

          18    the opinions used to generate that prior.  So

          19    it's much easier to -- for all different

          20    parties to agree that your prior information

          21    means something if it's based on some kind of

          22    empirical evidence, like data from prior




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             88


           1    studies and it is -- if it is based on

           2    opinion.

           3              Now, this slide is on prior

           4    information for studies that have a control,

           5    for example, randomized controlled trials.

           6    The prior information could be for the new

           7    device, or it could be for the control

           8    device, or it could be for both devices.

           9    Prior information for the concurrent control

          10    could include historical control data.

          11    That's actually a common source.

          12              And what that can do for you is it

          13    can --because that is increasing your

          14    precision on estimating the parameter for the

          15    control, that can allow you, in certain

          16    cases, to enable you to enroll fewer controls

          17    in your -- say, randomized controlled trial,

          18    and randomize more patients to the

          19    investigational device, thereby increasing

          20    the experience with it.  So there can be

          21    advantages to this, to using prior

          22    information for controls.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             89


           1              Now, we don't want to get carried

           2    away and there are certain situations where

           3    your prior information may be too

           4    informative.  For example, you might base --

           5    your prior information may be based on many

           6    more patients than you are even planning to

           7    enroll in the current study.  And that might

           8    be an indication that the prior information

           9    is more than what we would consider

          10    allowable.

          11              The prior distribution is too

          12    informative, and one way to measure it is if

          13    the prior probability -- that the pivotal

          14    study is a success, that is, if it

          15    demonstrates the proposed claims on the study

          16    parameters as excessively high, and if --

          17              So there's a lot of evaluation of

          18    the prior -- the proposed prior distribution

          19    based on these kinds of measures, like prior

          20    probability and study success.  So this is

          21    the probability before even going -- before

          22    you start collecting data in your pivotal




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             90


           1    trial.

           2              If it turns out that the -- this

           3    prior probability is too high, then there are

           4    remedies.  For example, you could make some

           5    kind of discounting of the prior

           6    distribution.  You can also increase the

           7    stringency, the success criterion.  For

           8    example, if the success criterion is that the

           9    posterior probability of the claim is 95

          10    percent, perhaps you could adjust that to 99

          11    percent or something.  Even higher, so that

          12    it makes it more difficult, and so -- because

          13    right now, with the prior information being

          14    too informative, it might be too easy to show

          15    safety and effectiveness beyond what we think

          16    is reasonable.

          17              Now, the second class of prior

          18    distributions is non-informative prior

          19    distributions.  You may have any device on

          20    which you really can't -- there aren't --

          21    there really isn't any good sources of prior

          22    information.  And if -- but there are some




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             91


           1    advantages to the Bayesian approach, in which

           2    you might want to use a Bayesian analysis,

           3    even though you don't have prior information

           4    and Telba alluded to some of those flexible

           5    study design.

           6              If the prior information on the

           7    study parameters is unavailable or does not

           8    exist, then the initial uncertainty about the

           9    parameters can be quantified with a

          10    non-informative prior distribution.  And for

          11    example, if you -- example of a possible

          12    non-informative prior is you could place

          13    uniform probabilities over the range from the

          14    minimum to the maximum possible of that

          15    parameter value, if they exist.

          16              So for example, an adverse event

          17    rate, say, you know, has to be between zero

          18    and one, you can place uniform probabilities

          19    over that range and that could be your prior

          20    distribution -- a non-informative prior

          21    distribution.

          22              Now, I want to go to subsection




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             92


           1    5.6, which has to do with hierarchical

           2    modeling, which is a flexible way of

           3    combining prior information with current

           4    study data.  So as you've heard in a Bayesian

           5    analysis, the prior distribution is combined

           6    with the current study data to produce the

           7    posterior information.  And all inferences on

           8    the study parameters are based on this

           9    posterior distribution.  But that posterior

          10    distribution is produced, according to some

          11    Bayesian statistical model, and hierarchical

          12    modeling is one way of producing a posterior

          13    distribution, which combines prior studies

          14    with the current study.

          15              Hierarchical models are often

          16    employed, because of their flexibility in

          17    combining prior information with current

          18    study data.  And it's called hierarchical

          19    because it assumes a hierarchical structure

          20    in the specification of the statistical

          21    distributions for the data and the

          22    parameters.  And I'll get to that in a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             93


           1    minute.

           2              But what happens in the

           3    hierarchical modeling is that when you are

           4    estimating the current -- the parameters in

           5    the current study, the hierarchical models

           6    borrow strength -- is a term that's used from

           7    the prior information, from the prior

           8    studies.  And that borrowing strength

           9    translates potentially to additional sample

          10    size, beyond the sample size in your current

          11    study.

          12              But the borrowing is adaptive.  And

          13    that's what's nice about it.  If the prior

          14    information is consistent with the current

          15    study results, then the hierarchical model

          16    will borrow a lot of strength from the prior

          17    information.  However, if the two different

          18    set -- two sets of results are very

          19    different, then the strength borrowed can be

          20    negligible.

          21              So this has -- this is very

          22    appealing to regulators.  Hierarchical models




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             94


           1    will reward sponsors for having prior

           2    information that's consistent with the actual

           3    device performance, but they protect against

           4    over reliance of the prior information that

           5    turns out to be overly optimistic or they can

           6    be even overly pessimistic for the current

           7    study parameters.  So it's -- it's a flexible

           8    way that protects the FDA from optimistic

           9    prior distributions, overly optimistic, and

          10    it actually is flexible for the sponsors in

          11    protecting against overly pessimistic prior

          12    information.

          13              So as an example of a hierarchical

          14    model and the kind of structure, that is

          15    assumed in a very non-technical way, suppose

          16    you have a registry for an approved device

          17    that is used as prior information for an

          18    investigational device.  Hierarchical model

          19    would assume -- there's two levels of the

          20    hierarchy.  And at the first level we look at

          21    patients within each of the studies and we

          22    consider the patients within the current




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             95


           1    study as exchangeable and Telba went through

           2    what exchangeability means a little bit, so

           3    that they can be used to estimate study

           4    parameters in that current study.

           5              Also we can think of patients

           6    within the registry as exchangeable, so that

           7    it can estimate -- these patients can be used

           8    to estimate -- study parameters in the

           9    registry.  But there's a second level of the

          10    hierarchy that's assumed and that's -- these

          11    patients across the two studies are not

          12    considered exchangeable.  If that were true,

          13    then we could just pool them to estimate

          14    parameters.  But instead of doing that, what

          15    we assume is that the two studies, the

          16    current study and the registry are

          17    exchangeable.  And this is at the study

          18    level.  So this is the second level of the

          19    hierarchy and it's what provides the

          20    flexibility in the analysis.

          21              So suppose we have a success rate

          22    and that's the primary endpoint for this




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             96


           1    study.  So under the second level of the

           2    hierarchy, the success rate, the two success

           3    rates are study specific.  So we are assuming

           4    that they are different in the registry and

           5    the current study.  And that's usually a

           6    prudent assumption, because after all no two

           7    studies are alike, especially when you are

           8    considering a registry versus perhaps a more

           9    well -- a well- characterized study.

          10              So -- but the two study-specific

          11    success rates, the one for the registry, and

          12    the one for the current study are related,

          13    because they are assumed exchangeable in the

          14    hierarchical model.  So you can't borrow

          15    strength from the registry to estimate the

          16    success rate in the current study.  So it

          17    provides the information -- the registry

          18    provides information about the success rate

          19    in the current study, but it's no as much as

          20    if the patients in the registry and the

          21    patients in the current study were considered

          22    together as exchangeable, because otherwise




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             97


           1    you could just pool them, so it doesn't

           2    provide that much information.

           3              Now sometimes, the exchangeability

           4    assumption may not be appropriate without

           5    having to make some kind of adjustments to

           6    the modeling.  In other words, sometimes we

           7    need to calibrate the studies so that they

           8    are similar enough after calibration such

           9    that you can make this exchangeability of

          10    studies assumption tenable.

          11              So if the prior and current studies

          12    are known to differ in important covariates

          13    and in the distribution of those covariates

          14    it might have effects on the outcomes, for

          15    example, prognostic variables.  And the

          16    exchangeability assumption is not likely to

          17    hold.  So the prior and current studies,

          18    however, might still be regarded as

          19    exchangeable after some statistical

          20    adjustment for all of the known important

          21    covariates.

          22              The statistical adjustment for the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             98


           1    covariates in effect calibrates the prior

           2    studies to the current studies so that the

           3    exchangeability assumption is more tenable.

           4    And an important point though is for proper

           5    calibration.  In this way, the covariate

           6    information at the patient level is

           7    ordinarily needed, in both the prior studies

           8    and the current study.

           9              For some of the prior studies that

          10    have been proposed in that -- and we have

          11    seen the number of studies that have been

          12    proposed -- that were actually summaries in

          13    the literature in some paper.  You may not

          14    have covariate information at the patient

          15    level, and so you may not be able to make

          16    these adjustments to make the assumption of

          17    study exchangeability tenable.

          18              So usually, you need that kind of

          19    patient level information in not only the

          20    current study, which of course, you can --

          21    you can get that information, but also in the

          22    prior studies.  So applications of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                             99


           1    hierarchical models are not restricted to

           2    just combining studies.  You can also look at

           3    combining data across prior exchangeable

           4    centers in a multi-center trial, and this has

           5    been done in at least one example through a

           6    Bayesian analysis.  And because -- at least

           7    -- and specially for medical devices,

           8    oftentimes, centers behave differently and

           9    the reason is -- some of the -- sometimes the

          10    reasons are that the physician skill level

          11    varies from center to center and so you might

          12    see quite different devise effects.

          13              So we worry about that and a way of

          14    accounting for that heterogeneity across

          15    centers might be through a hierarchical model

          16    over centers.  And it also can be used as a

          17    way of adjusting for multiple testing of --

          18    what you might consider exchangeable

          19    subgroups.  So it is a -- you can use

          20    hierarchical modeling to adjust for

          21    multiplicity.

          22              So now, onto section 6, which is




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            100


           1    analyzing the Bayesian trial, and I'll cover

           2    the sections Telba didn't cover.  Now, all

           3    inferences are based on the posterior

           4    distribution for the study parameters and the

           5    posterior distribution is an update of the

           6    prior distribution with the current study

           7    data.  But with -- to facilitate this

           8    statistical inference, we often would like to

           9    see summaries of that posterior distribution.

          10              So here are a number of possible

          11    summaries, you can look at a graphic

          12    representation of the posterior distribution

          13    for a parameter.  You could look at the

          14    posterior mean for that parameter, which is

          15    often the estimate of the parameter.  The

          16    posterior -- the standard deviation gives you

          17    -- measures the uncertainty of the -- about

          18    which the -- the uncertainty you have for

          19    that parameter value.

          20              I mean, you can look at the

          21    posterior probability of a claim that may be

          22    used to define study success, and you could




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            101


           1    also think about summaries based on Bayesian

           2    hypothesis testing, and in credible

           3    intervals.  The FDA bases approval often on

           4    demonstrating claims via hypothesis tests.

           5    So you could think about a Bayesian

           6    hypothesis test that is based on the

           7    posterior probability for the hypothesis.

           8              And so, what all I mean -- all I

           9    mean by Bayesian hypothesis test, all I mean

          10    by that is, there is some kind of decision

          11    rule that defines study success, which could

          12    be, say, the posterior probability of the

          13    primary endpoint claim being met at say, 95

          14    percent, some kind of decision rule.

          15              Now, Frequentist hypothesis tests

          16    look at operating characteristics like

          17    controlling the type I and type II error

          18    rates that Telba defined.  There is no such

          19    notion really in the Bayesian paradigm, but,

          20    however, regardless of whether a trial is

          21    Bayesian or Frequentist, the FDA is keenly

          22    interested in the type I and type II error




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            102


           1    rates associated with its decision to approve

           2    or not approve a device under review.

           3              So, even though, you may have --

           4    proposing a Bayesian trial, we do have to

           5    look at the -- that the type I error rate is

           6    not completely out of control.  Therefore,

           7    the FDA recommends that you investigate the

           8    type I and type II error rates of your

           9    proposed Bayesian hypothesis test or decision

          10    rule, and for complex modeling, especially

          11    simulations used to compute -- often used to

          12    compute these error rates.

          13              You can also summarize the

          14    posterior distribution using an interval

          15    estimate.  In Frequentist statistics, that is

          16    done with a confidence interval, which is

          17    basically an interval estimate for a

          18    parameter that indicates the likely values of

          19    it.  The Bayesian counterpart, as Telba

          20    mentioned, is the credible interval.  If the

          21    posterior probability of the parameters lies

          22    in an interval that is 0.95, then you would




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            103


           1    call that a 95 percent credible interval.

           2              There are different types of

           3    credible intervals in Bayesian statistics and

           4    two of the most common are the central

           5    posterior interval and the highest posterior

           6    density interval.  So without describing what

           7    they are, I'll just tell you that there are

           8    different types.

           9              I want to talk a little bit about

          10    post-market surveillance, which is section 7.

          11    The pre- market clinical study on which

          12    approval was based, could serve as a prior

          13    information towards a -- perhaps there is a

          14    condition of approval study or some kind of

          15    post-market study.  So that would bridge the

          16    two distinct phases of regulation.  The

          17    pre-market and the post-market in the total

          18    product lifecycle, so we are considering that

          19    as Susan Gardner was mentioning -- our office

          20    director.

          21              If the pre-market study was

          22    Bayesian, say, the posterior distribution or




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            104


           1    some modification of it could serve as the

           2    prior distribution going in the post-market.

           3    Also, one particular activity where -- we are

           4    doing, in terms of post-Bayesian

           5    post-surveillance, is using Bayesian

           6    statistics methodology as a way to data-mine

           7    the large databases we have of adverse events

           8    in the post- market.

           9              And they can have particular

          10    advantages in that area, and to give an

          11    example, there -- you can think of a class of

          12    devices, say with 100 possible adverse events

          13    to the patient for that kind of class of

          14    devices, and maybe 50 possible device events,

          15    say, some kind of malfunction.  There might

          16    be 100 manufacturers that have -- I don't

          17    think it's realistic, but that might be on

          18    the market right now, and they may have -- a

          19    number of different devices that are on the

          20    market in this class, and there might be even

          21    models within those devices.

          22              So, when you look at all the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            105


           1    possible combinations, there can be huge and

           2    -- so you are going to see many spurious

           3    signals that might concern you but they're

           4    expected by chance alone.  So there has to be

           5    some way of controlling the false positive

           6    rate in the post market.  And as I mentioned

           7    earlier, Bayesian hierarchical modeling could

           8    be used as a way of adjusting for this

           9    multiple looking, or multiple testing for

          10    potential signals that might concern us.

          11              And it could be used -- can be

          12    employed on the adverse event/device

          13    combinations to reduce the number of spurious

          14    signals thereby illuminating the true signals

          15    that we are all -- want to get at.  And what

          16    happens briefly with the hierarchical

          17    modeling is that the spurious signals can get

          18    pulled back to normal levels because the

          19    strength is borrowed from data and other

          20    combinations of adverse events and devices.

          21              CDRH is currently developing

          22    Bayesian hierarchical modeling tools




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            106


           1    specifically designed to data- mine

           2    post-market medical device reports, and as

           3    Susan Gardner had mentioned, we are

           4    contracting with a company by the name of

           5    Lincoln Technologies, which is -- specializes

           6    in developing Bayesian software for data-

           7    mining of large databases.

           8              So that concludes my presentation

           9    and Laura will be -- Laura Thompson.  I will

          10    introduce her.  She will be the last speaker

          11    in the first half of the meeting, and she is

          12    going to talk about additional information

          13    requested by FDA in Bayesian analysis, and

          14    also issues having to do with computation.

          15    Thanks.

          16                   (Applause)

          17              MS. THOMPSON:  Okay, thanks Gene.

          18    So I'm going to talk about basically

          19    everything else that's in the guidance

          20    including some additional practical and

          21    computational issues as well as some

          22    potential information requested by FDA or




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            107


           1    rather CDRH when a sponsor submits an IDE.

           2    So I'll be talking about Section 6.4 and some

           3    parts of Section 6.4 and 9.2 and the

           4    guidance, which encompass model checking and

           5    model selection as well as Section 9.3 and

           6    9.1, which cover additional information

           7    requested by CDRH and then finish with

           8    computational issues.

           9              Okay.  While model checking is

          10    important in any analysis -- but it tends to

          11    be especially important in a Bayesian

          12    analysis because of the model complexity

          13    that's often permitted with these types of

          14    analyses and a tool frequently used in model

          15    checking in a Bayesian analysis is the

          16    posterior predictive distribution.  Now,

          17    Telba Irony talked about this distribution in

          18    her session where it was referred to as the

          19    predictive distribution without the

          20    posterior, but it's the same thing here.

          21              My reason for putting posterior

          22    predictive distribution is to emphasize that




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            108


           1    it's a distribution that is evaluated or

           2    examined after running the trial.  So they

           3    call it posterior -- so to reiterate, the

           4    posterior predictive distribution is the

           5    distribution of a new future hypothetical

           6    observation from the trial under the

           7    assumptions of the current model and of

           8    course after the trial data have been

           9    collected.

          10              And so the model checking idea is

          11    to -- with this distribution is to generate

          12    new hypothetical data from the posterior

          13    predictive distribution, and see how well it

          14    matches observed data from the trial and I'll

          15    be a little bit more clear about what,

          16    "matches" means here in a few slides.  So in

          17    this way, we compare data generated from the

          18    model to data observed from the trial, and in

          19    the next couple of slides I give some brief

          20    examples of the use; where this distribution

          21    might be used.

          22              In the first example, suppose we




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            109


           1    have a model that assumes a constant event

           2    rate across time, there could be some reasons

           3    for why the event rate might be non- constant

           4    including mid-course modifications made to

           5    the trial that effect the event rate.

           6    Physicians may learn better overtime how to

           7    perform a procedure, there may be new

           8    therapies that become available in the course

           9    of the trial et cetera, and so you could use

          10    the posterior predictive distribution to

          11    check this constant event rate assumption by

          12    predicting the observed event rate in the

          13    second half of the trial using the posterior

          14    predictive distribution computed using the

          15    data from the first half of the trial.

          16              The second example has to do with

          17    sensitivity analysis for missing data.  Often

          18    we assume that missing data follows the same

          19    distribution as observed data in the trial.

          20    And a posterior predictive distribution can

          21    be used to check this assumption and to

          22    perform a sensitivity analysis to the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            110


           1    assumption.  One product that has been

           2    approved -- the BAK/Cervical Interbody Fusion

           3    System did in fact use an analysis like this.

           4              The summary of safety and

           5    effectiveness is referenced in the guidance,

           6    including the URL where you can find the

           7    actual document.  Okay, now, the assessment

           8    of how well the posterior predictive data

           9    match the observed data from the trial can be

          10    done using a concept or quantity called, "The

          11    Bayesian p-value."  And this is the

          12    probability of observing a result at least as

          13    extreme as what you observed in the trial

          14    where the reference distribution here is the

          15    posterior predictive distribution.

          16              And without going into more

          17    details, I list some references where you can

          18    find more information.  The first reference,

          19    the Gelman et al. is in our guidance.  The

          20    second reference was inadvertently omitted

          21    from the guidance, and one of the coauthors

          22    happens to be one of our distinguished panel




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            111


           1    members for this afternoon Brad Carlin, so

           2    that reference will end up in the final

           3    version of the guidance.

           4              MR. CARLIN:  Bless you.

           5              MS. THOMPSON:  And in addition,

           6    Bayesian deviance measures can also be used

           7    for model checking, and then they also use

           8    the predictive distribution, again further

           9    references -- for those types of measures can

          10    be found in the guidance.

          11              Model selection is very briefly

          12    covered in Section 9.2 in the guidance.  I

          13    can say a few words about it here, perhaps

          14    more as needed in the guidance.  FDA or

          15    rather CDRH here recommends that the general

          16    form of any statistical model be chosen a

          17    priori, and that may be done in collaboration

          18    with FDA.

          19              However, there might be times when

          20    although the general form of a model is

          21    pre-specified, maybe some specific form of

          22    the model would have to wait until the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            112


           1    analysis is done.  Some examples of that

           2    might be what particular covariates out of a

           3    pool of covariates that you're going to

           4    consider might be included in a regression

           5    model what transformations on those

           6    covariates are going to use in the model,

           7    those types of decisions.

           8              In cases like that, what we would

           9    probably recommend is that you pre-specify

          10    what type of model selection method or

          11    criterion you're going to use to choose what

          12    model would be the final one for the -- for

          13    inference purposes.

          14              And the guidance talks about one

          15    criterion, posterior odds that can be used to

          16    choose among several plausible models briefly

          17    posterior odds refer to the ratio of the

          18    posterior probability of the model to its

          19    compliment and just like what posterior

          20    probability, the higher the posterior odds,

          21    the more favorable is the model, so you would

          22    imagine that you would choose the model that




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            113


           1    has the highest posterior odds.

           2              Okay, I want to move to section

           3    9.1.  This section lists some items that are

           4    potentially requested by CDRH when a sponsor

           5    submits an IDE and I've listed all of them I

           6    think on the next seven or so slides, I'll go

           7    over each of them briefly.  First, and

           8    arguably one of the most important is to

           9    include detailed information about the prior

          10    that you're planning to incorporate into the

          11    study and most importantly this would include

          12    details about any previous studies that

          13    you're going to summarize for your prior

          14    information.

          15              Second, we would like to have you

          16    mention what your criterion for success for

          17    your study is.  So this is how you will

          18    decide whether or not your trial is

          19    successful and frequently the criterion will

          20    involve posterior probability.  So for

          21    example, if it's desired that a treatment

          22    effect is greater than zero, then the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            114


           1    criterion for success might be that the

           2    posterior probability, that the treatment

           3    effect is greater than zero is greater than

           4    some high value for example, 95 percent, so

           5    that's an example of what a criterion for a

           6    successful trial might be.

           7              Okay, also we would like

           8    information about your method for choosing

           9    sample size.  Now, as Telba Irony discussed

          10    already, this typically involve simulation,

          11    perhaps to obtain the expected sample size

          12    when the design involves interim looks, and

          13    to obtain maybe a fixed sample size when the

          14    design doesn't involve interim looks, which

          15    case you might assume a range of different

          16    treatment effects and sample sizes and

          17    calculate the posterior probability of the

          18    study claim for each of those sample size --

          19    combination of sample size and expected

          20    treatment effect.

          21              And then, I guess, for all the

          22    sample sizes that need a particular high




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            115


           1    posterior probability of the study claim, and

           2    one example is 0.95, then you choose the

           3    sample size for the trial that is consistent

           4    with a reasonable assumption about the

           5    treatment effect.

           6              Okay, in addition, one possible

           7    approach for choosing sample size is to

           8    ignore prior information and calculate sample

           9    size using the Frequentist method.  And then

          10    analyze the data using the prior information

          11    in a Bayesian way, and in some cases, this

          12    can lead to a conservative estimate of sample

          13    size.

          14              Okay, we may also request

          15    tabulation of Frequentist power for a range

          16    of various treatment effect sizes and sample

          17    sizes, or expected sample sizes.  And what we

          18    are asking for here is that you simulate your

          19    trial, perhaps many thousands of times, and

          20    then tabulate the proportion of times that

          21    the trial is successful using your Bayesian

          22    decision rule.  So that is what this table




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            116


           1    would give.

           2              And note that for a treatment

           3    effect size that is consistent with a null

           4    treatment effect, then that estimate of power

           5    will provide the type I error rate.  And as

           6    mentioned previously, this type of table will

           7    often have to be approximated via simulation.

           8    And I give some shells of examples of what a

           9    Frequentist power table might look like.

          10    This is a very simplified example.

          11              I mean, it is another version what

          12    Telba Irony showed in one of her slides.  One

          13    of the main differences in this table is that

          14    the second column, sample size that is shown

          15    on the screen now, is a fixed sample size.

          16    Not an expected sample size.  So just to,

          17    kind of, describe what is in this table,

          18    let's say, for example -- I don't think I

          19    have a mouse here.  Well, we will just have

          20    to point out what's in the table.

          21              Let's see -- your far left-hand

          22    side, suppose we have a, what we could call




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            117


           1    the null hypothesis, H0, that a success rate

           2    P is about 50 percent, so chance, or less

           3    than 50 percent.  And an alternative

           4    hypothesis, which we call H1, sort of, like

           5    the research hypothesis, that the success

           6    rate exceeds 50 percent, and suppose the

           7    decision rule is to declare a trial

           8    successful if the posterior probability of

           9    the alternative hypothesis is at least 95

          10    percent.

          11              Now, for a true rate consistent

          12    with either of these two hypotheses and for

          13    each of the three fixed sample sizes that are

          14    given in the table, what we would ask for, is

          15    that you estimate the probability of meeting

          16    the endpoint in each of those six conditions

          17    given there.  So this would be the proportion

          18    of the simulations out of the several

          19    thousand that you simulate where the trial is

          20    considered successful.

          21              So the proportion of trials where

          22    the posterior probability of the alternative




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            118


           1    hypothesis does indeed exceed 0.95 or the

           2    decision rule is met.  And then, in the

           3    table, note that the first three rows for

           4    probability meeting the endpoint, correspond

           5    to power, because the true rate that is

           6    assumed there, falls under the research

           7    hypothesis, and the last three correspond to

           8    type I error rates, because the true rate

           9    falls under the -- is included in the null

          10    space.

          11              And then the next table is

          12    basically the same thing, but just emphasizes

          13    that if you have interim looks in your trial,

          14    we ask that you simulate them when computing

          15    operating characteristics.  Telba Irony

          16    talked a lot about this.  So in the second

          17    column here, though I have got just three

          18    schedules of interim looks that might be

          19    proposed, and so we would ask that if you are

          20    going to build the table like this that you

          21    incorporate the interim looks and the

          22    schedules of the interim looks when




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            119


           1    tabulating power and type I error rate.

           2              Okay.  We also may request that you

           3    compute the prior predictive probability of

           4    the study claim.  This is the probability of

           5    the study claim prior to running the proposed

           6    trial, and Gene Pennello talked a little bit

           7    about this.  Now with the posterior

           8    predictive distribution, I talked about at

           9    the beginning of this session; we obtained

          10    the distribution of new data from the trial,

          11    after running the trial.

          12              With the prior predictive

          13    distribution, we obtained the distribution of

          14    new data from the trial, before running the

          15    trial.  And in this slide -- the next few

          16    slides -- it is the prior predictive

          17    distribution that we use to obtain the prior

          18    predictive probability.  It can be used as

          19    the measure of the informativeness of the

          20    prior information, as Gene Pennello

          21    mentioned.  We request that this be computed

          22    at the design stage.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            120


           1              Usually it has to be computed using

           2    simulation.  And the general recommendation

           3    is that it not be too high.  One way to think

           4    about why this is so is, the prior predictive

           5    probability of the study claim is the

           6    probability that the study will be successful

           7    at the end of the trial before collecting any

           8    data.  Now if that is exceptionally high,

           9    like 0.99, there is a, kind of, a -- somewhat

          10    of a logical question about whether the

          11    trials should even be run?

          12              And in fact, if you think about

          13    that point in the trial times zero as being

          14    an interim look, then you have, you know,

          15    close to one probability that your trial will

          16    be successful before running the trial, then

          17    as an interim look, you would stop at that

          18    point.  So the general recommendation is that

          19    it not be too high.  Any more recommendation

          20    about how high it should be is, kind of, a

          21    case-by-case basis.  And it depends on the

          22    quality of the prior information and how




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            121


           1    confident we are in that prior information.

           2    So that's about all the specific

           3    recommendation I think I can give regarding

           4    that.

           5              So basically, it is evaluated on a

           6    case-by-case basis.  Gene Pennello mentioned

           7    this in his slide -- if the prior predictive

           8    probability of the study claim is too high

           9    there are some remedies to reduce the

          10    probability.  If you are using a hierarchical

          11    model, then inflating the between-study

          12    variance is one method, so you are borrowing

          13    less from the prior studies with the current

          14    study, and that will reduce the probability.

          15              And the second point here talks a

          16    little bit about how we may evaluate the

          17    magnitude of the prior probability of the

          18    claim.  One thing that we pretty much know

          19    for sure though is that in some sense it has

          20    a lower bound because if you compute the

          21    prior predictive probability of the study

          22    claim, where you have a non- informative




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            122


           1    prior or no prior information, well, you

           2    know, you have prior information, you want it

           3    to be higher than that or else there is no

           4    reason to use the prior information.

           5              So in some sense it has a lower

           6    bound, and of course that is considered when

           7    the FDA makes its evaluation of an IDE.

           8    Okay, to finish up -- the next few slides, I

           9    want to talk about information that is in the

          10    appendix.  The appendix gives a brief

          11    overview of some technical issues regarding

          12    estimation in a Bayesian analysis, and I have

          13    tried to simplify it even more in the next

          14    three slides, and I hope that I haven't

          15    simplified it to the point where it is wrong,

          16    but I will try to go through this and see

          17    what happens.  One of the reasons why

          18    computations are important in a Bayesian

          19    analysis is because many quantities of

          20    interest in this type of analysis involve

          21    calculating a mathematical integral.

          22              Some examples are the posterior




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            123


           1    mean, the posterior probability of

           2    hypothesis, or here, we are integrating over,

           3    with respect to the prior -- posterior

           4    distribution.  And so, you know, computing an

           5    integral can be easy if it can be done on

           6    paper; can be harder if it can't.  And so,

           7    you know, you need to do it numerically.  And

           8    that's one of the reasons why computation

           9    always comes up when we talk about Bayesian

          10    and it usually comes up when we talk about

          11    modern Bayesian analysis.

          12              Markov chain Monte Carlo techniques

          13    or MCMC techniques provide Monte Carlo

          14    approximations to these integrals and this

          15    allows the use of more realistic, complex

          16    statistical models and resulting posterior

          17    distributions.

          18              Typical MCMC techniques simulate

          19    draws from what's called a proposal

          20    distribution, and this is typically a simpler

          21    approximation to the posterior distribution.

          22    And of course you would want to draw from the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            124


           1    posterior distribution itself.  It may not be

           2    easy to do so, so you start with something

           3    that's -- that you know how to draw random

           4    variance from and you call that a proposal

           5    distribution.

           6              Now, each draw from the proposal

           7    distribution is either accepted or rejected

           8    as a valid draw from the posterior

           9    distribution itself based on a comparison of

          10    the posterior distribution to the proposal

          11    distribution at the drawn value.

          12              If the drawn value is rejected as a

          13    valid draw from the posterior distribution,

          14    then the simulated draw is taken to be the

          15    previously accepted drawn value.  So you are

          16    drawing values in a sequence and then you are

          17    collecting them altogether.  And these draws

          18    form what's called Markov chain, which

          19    eventually converges, so that subsequently

          20    sample draws actually come from the posterior

          21    distribution.  And then those sample values

          22    can be used for posterior inference.  And one




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            125


           1    example is computing posterior probabilities.

           2              However, the point at which

           3    convergence of the chain is achieved is not

           4    known a priori, and must be assessed using

           5    what we have, which is just sampled values

           6    from the chain itself.  FDA recommends that

           7    you check convergence of the chain.  There

           8    are some diagnostic methods in the literature

           9    that can be used to do that and we list some

          10    references in the guidance.  The guidance

          11    also mentions a few remedies if convergence

          12    appears to be slow.  Those include employing

          13    an alternative model parameterization,

          14    sometimes re-parameterizing the model so that

          15    it's linear and the parameters can help --

          16    usually does help convergence.

          17              Using a proposal distribution that

          18    more closely resembles the posterior

          19    distribution, of course, that will usually

          20    help.  Introducing auxiliary variables to

          21    simplify MCMC sampling, something called data

          22    augmentation can help convergence and also




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            126


           1    avoiding a type of non-informative prior

           2    called an improper prior which can get in the

           3    way of convergence.

           4              And references for these slides are

           5    included in the guidance.  Of course, we are

           6    not trying to give you a complete lesson on

           7    MCMC sampling but we feel that it's important

           8    to include at least an overview because of

           9    its importance in our evaluation of

          10    submissions.  And with that said, I list some

          11    typical software that we see being used to

          12    conduct this type of estimation.  WinBUGS was

          13    mentioned before.  Certainly, you can write

          14    your own code in compiled language and SAS

          15    and the R environment et cetera.

          16              As with any submission, FDA will

          17    then request the program code that you use as

          18    well as the data itself and we may review the

          19    calculations that you've done using an

          20    alternative software.  And I think that's it.

          21                   (Applause)

          22              MR. CAMPBELL:  Thank you, Laura.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            127


           1    We have a little time this morning before we

           2    break for lunch.  And I thought it would be

           3    good if we gave folks an opportunity to ask

           4    questions and bring up brief comments knowing

           5    that this afternoon we'll have lots of time

           6    in our two panel discussions for the audience

           7    to become actively involved.  You've sat here

           8    patiently all morning and you probably have

           9    some questions that you might like to ask

          10    those of us who gave the presentations this

          11    morning.  So I would like to encourage you to

          12    do that.

          13              It's important to note, I hope,

          14    that we've managed to talk about Bayesian

          15    statistics without inflicting any formulas on

          16    you.  So if you are not a statistician

          17    hopefully you haven't felt disadvantaged in

          18    any way at all.

          19              So are there any questions for any

          20    of the speakers?  Are there any issues that

          21    might have come up?  If there is a question,

          22    if you could step up to the microphone and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            128


           1    identify yourself, say who you are and your

           2    affiliation that would help our transcription

           3    expert.  Dr.  Lachenbruch?

           4              MR. LACHENBRUCH:  Oh, you found me

           5    out.

           6                   (Laughter)

           7              MR. LACHENBRUCH:  Yeah.  I'm

           8    semi-retired but still officially employed by

           9    FDA.  And one question that I had that has

          10    bugging me a little bit, not WinBUGging but

          11    it has to do with when one is looking at some

          12    events that may have a very low prior

          13    probability or zero -- for example, if you

          14    are looking at an adverse event that we know

          15    if it -- if the adverse event were as high as

          16    5 or 10 percent we would not under any

          17    circumstances license a product.  But we

          18    would like to have a non-informative prior.

          19    So if you used a prior that went from say, a

          20    uniform prior between zero and one, you would

          21    be giving some weight to values that would be

          22    totally unacceptable.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            129


           1              So if you instead said I want to

           2    have a prior that went from -- was uniform

           3    between 0 and 0.1, that I think would imply

           4    that the support of your prior was

           5    constrained.  And so if I'm not mistaken, the

           6    support of the posterior would also be

           7    constrained.  So what would you recommend

           8    doing in those circumstances?

           9              MR. CAMPBELL:  Well, I'm going ask

          10    some of my folks here sitting in the first

          11    row on the right to help me respond to that.

          12    I think Telba mentioned a little bit about

          13    uniform priors in her presentation.  And

          14    maybe Telba, you would like to come up to the

          15    microphone, knowing that the emphasis that

          16    we've had this morning is based on

          17    quantitative information and so usually our

          18    quantitative information would not be in the

          19    form of a uniform prior.

          20              MS. IRONY:  Well, you're right.  If

          21    you will give zero prior probability to any

          22    event, no matter how strong the evidence, the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            130


           1    evidence is that that event has a higher

           2    probability.  You never correct that.  So the

           3    first rule for the Bayesian then, that's

           4    called Cromwell's rule, is that you never

           5    give probability zero to anything, unless you

           6    are absolutely, absolutely certain that

           7    that's impossible to happen.  Usually we are

           8    not absolutely certain that something is not

           9    going to happen.

          10              Of course, you know, talking about,

          11    let's say blood pressure, you won't be giving

          12    a probability of that value being millions or

          13    in the millions range.  So you have to have a

          14    reasonable range, but the big rule is not to

          15    give zero probability to any event.  So if

          16    you are talking about, let's say, a

          17    proportion and you know that it goes from

          18    zero to one, you know that's impossible

          19    that's going to be larger than one, you can

          20    give a zero probability to a proportion

          21    that's larger than one.  But you give some

          22    probability, no matter how small it is, to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            131


           1    any values between zero and one.

           2              MR. CAMPBELL:  Thank you.  Yes?

           3              MR. KESSLER:  I'm Larry Kessler

           4    from the Center for Devices.  And the

           5    question I have relates to some of the

           6    comments that Laura Thompson made when she

           7    mentioned that you may request frequentist

           8    tables, which concern the structure and the

           9    analysis of the study.  So my question is

          10    two-fold.  Number one, is that a way of

          11    backing away from a true Bayesian approach

          12    from the center, and number two, does it at

          13    all put a company under some degree of

          14    jeopardy when they are trying to go down the

          15    Bayesian road and they also have to submit

          16    frequentist-looking tabulations about the way

          17    their study is designed?

          18              MR. CAMPBELL:  Well, I'm going to

          19    let Laura help answer this question but I

          20    should at least hasten to point out that my

          21    friends who are dyed-in-the-wool Bayesians

          22    think this is terrible that companies be




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            132


           1    asked to essentially explain the frequentist

           2    properties of their Bayesian designs.  But

           3    with that as an introduction I'm sure Laura

           4    will say much more about it than that.

           5                   (Laughter)

           6              MR. CAMPBELL:  And talk about the

           7    burden that really falls on companies who

           8    have to produce these kinds of tables.

           9              MS. THOMPSON:  Well, it wasn't my

          10    idea --

          11                   (Laughter)

          12              MS. THOMPSON:  -- but it's -- I

          13    guess in the transition, if you can call it

          14    that, to a full Bayesian approach, if we ever

          15    get there, as a regulatory agency we are

          16    concerned with proving devices that are not

          17    actually successful.  So the concept of a

          18    type I error rate is important to use.  The

          19    power is also important to us because we

          20    don't want to run trials that are not useful

          21    to run.  And so it doesn't -- to us it's

          22    useful information to perform these




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            133


           1    frequentist tables, to run the simulation

           2    many times and tabulate the frequency with

           3    which these rates occur because we think it's

           4    useful in -- to help us evaluate risk and

           5    benefit.

           6              And so as long as we believe it's

           7    useful I imagine we'll continue to do it.

           8              MR. CAMPBELL:  Thank you.  The

           9    notion of the type I error is something that

          10    the FDA takes very seriously and whether

          11    it's, you know, exactly the probably of a

          12    type I error rejecting the -- a point-null

          13    hypothesis or whether it's some other

          14    calculation there needs to be some assurance

          15    for the U.S. public that when the FDA

          16    approves or clears a device that it has

          17    reasonable assurance of safety and

          18    effectiveness, to quote the law.  And one way

          19    to do that is to worry about type I error or

          20    its analog.

          21              Now, you might say, well, why worry

          22    about power, but power or one minus the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            134


           1    probability of type II error, statistical

           2    power, is an important concept too, because

           3    ethically we would not like companies to run

           4    studies with low power.  And so asking

           5    companies to do simulations for type I error

           6    and simulations for power we find to be very

           7    important or -- certainly the analog.  Next

           8    question.

           9              MS. AJONGWEN:  Patience Ajongwen

          10    from Ortho- Clinical Diagnostics.

          11              MR. CAMPBELL:  Could you --

          12              MS. AJONGWEN:  Patience Ajongwen

          13    from Ortho- Clinical Diagnostics.  One of my

          14    questions has been answered already.  The

          15    other question I have is just for

          16    clarification.  If you study a trial with a

          17    uniform prior, and you have stated interim

          18    looks.  Can you use one of the looks to

          19    change your -- use the information, calculate

          20    the procedure and use that information as a

          21    prior for the continuation of the trial?

          22              MR. CAMPBELL:  Right, so the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            135


           1    question as I understand it is suppose you

           2    started with a uniform prior, you're going to

           3    have interim looks.  Along the way you would

           4    be updating the prior with your new

           5    information.  And so as Telba described along

           6    the way your posterior -- your interim

           7    posterior changes, and that's done

           8    automatically in a Bayesian approach.  I

           9    think that's an answer to the question that

          10    you posed.

          11              MS. AJONGWEN:  Well, it --

          12              MR. CAMPBELL:  So it's

          13    automatically done.  You don't have to worry

          14    about it.  What I thought you were going to

          15    ask was well, what would happen as you went

          16    along, you said, "Gee, the uniform prior was

          17    wrong and I really wish I had picked a

          18    different prior" and that would create some

          19    problems.

          20              MS. AJONGWEN:  Actually, that was

          21    really --

          22              MR. CAMPBELL:  Oh, that was your




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            136


           1    question?  Okay.

           2                   (Laughter)

           3              MS. AJONGWEN:  The second by just

           4    clarification, Telba said that we don't need

           5    sample sizes, because for Bayesian approach

           6    you don't really need the sample size.  You

           7    use a parameter space but I think the last

           8    speaker said that we still, you know,

           9    determine sample size.  So I just wanted

          10    clarification of that because I was a little

          11    bit confused.

          12              MR. CAMPBELL:  Okay.

          13              MS. AJONGWEN:  -- if my sample size

          14    is not important in the Bayesian theorem, why

          15    do I have to determine a priori -- that is

          16    what I want to ask.

          17              MR. CAMPBELL:  Okay.  I think I'm

          18    going to ask Telba to try to sort out the

          19    difference between the sample space and the

          20    parameter space and this whole difficulty

          21    about sample size calculation.

          22              MS. IRONY:  Now, I guess if you




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            137


           1    were doing just a purely Bayesian experiment

           2    in order to learn, you wouldn't need to

           3    determine a sample size.  However, because we

           4    are a regulatory agency, and also we're

           5    dealing with real sponsors and companies that

           6    have financial constraints and other types of

           7    constraints, we would like to get, as I said,

           8    a minimum sample size and a maximum sample

           9    size.

          10              In addition, in order to be able to

          11    simulate those tables, you need to have a

          12    minimal and maximal, and you need to tell

          13    when you are going to perform your interim

          14    looks.  So that's more a regulatory issue

          15    than a pure Bayesian issue.  One thing I just

          16    wanted to clarify a little bit with respect

          17    to Larry Kessler's question, was as before,

          18    about the Frequentist tables.

          19              The Bayesian's don't require it,

          20    but there is nothing in a Bayesian paradigm

          21    that is against it.  I mean, you can compute

          22    them, and there is nothing wrong with doing




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            138


           1    that.  I had the same misconception when I

           2    started working at the FDA, because I've

           3    never seen that in a Bayesian approach.  But

           4    I think it's an excellent idea, and, you

           5    know, no Bayesian will be against it, you

           6    know, it's just --

           7                   (Laughter)

           8              MR. CAMPBELL:  You'll get your

           9    chance this afternoon, Larry.

          10                   (Laughter)

          11              MR. CAMPBELL:  Yes.

          12              MR. LU:  My name is Michael Lu.

          13    I'm from Edwards Life Sciences.  On the

          14    overview section of the guideline, you

          15    mentioned that the interim analysis -- the

          16    interim look of the data won't impact the

          17    likelihood and the consequent that they won't

          18    impact the posterior distribution.  But on

          19    the later section, which is mentioned, the

          20    FDA also want the company to submit when

          21    you're going to do the interim look.

          22              I was a little confused of why we




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            139


           1    need that.  If we define clearly stopping

           2    criteria, which should decide that when we

           3    get enough information to stop and to make

           4    the decision, is that appropriate that we

           5    continue the look that if that criteria would

           6    be made, then to decide okay, now we got

           7    enough data, so we can stop, you know?

           8              MR. CAMPBELL:  That's am excellent

           9    question, and it really goes to the

          10    likelihood principle.  And quite frankly,

          11    this bothered me for quite a long time.  I

          12    guess what I would do is distinguish the

          13    evidence from decisions you make about the

          14    evidence.  And the likelihood principle would

          15    say the evidence is what the evidence is.

          16              But if it -- if you are in a

          17    situation where you need to make decisions,

          18    and in our case we are worrying about

          19    simulating the Frequentist properties

          20    associated with those decisions in a clinical

          21    trial, then the interim looks do have

          22    implications.  So there is this sense I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            140


           1    suppose that, oh, in Bayesian statistics you

           2    can look at will.  And in terms of likelihood

           3    principle that's absolutely true.

           4              But in terms of making decisions in

           5    a regulatory environment, those looks have

           6    implications.  They have implications in

           7    terms of your simulation of the type I error,

           8    they have implications in terms of power, and

           9    we worry about that.  So that's my defense of

          10    what appears to be -- I think your question,

          11    Michael, is, sort of, the difference between

          12    the likelihood principle and the way we are

          13    implementing it here, implementing Bayesian

          14    statistics with -- in this document.  Okay.

          15    Yes.

          16              MS. SETHURAMAN:  My name is Barathi

          17    Sethuraman and I'm from St. Jude Medical.  I

          18    had a question for Dr.  Irony, and I guess

          19    this is also, kind of, a follow-up to this

          20    likelihood principle.  I'd like to understand

          21    -- you mentioned that the Frequentist methods

          22    do not follow the likelihood principle.  I'd




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            141


           1    like you to clarify just philosophically,

           2    what that means.

           3              MR. CAMPBELL:  I think -- she's

           4    going to answer it, but I'm going to help

           5    her, and say that what she really meant to

           6    say was they don't always follow the

           7    likelihood principle.  In many cases, the

           8    Frequentist property -- the Frequentist

           9    statistics do follow likelihood principle.

          10    But I'm going to let Telba contend with this

          11    since she -- it is her slide and her toys.

          12              MS. IRONY:  Okay.  That's a touchy

          13    issue.  But what I can do is, like, to give

          14    an example.  The example that I have on top

          15    of my head is the following.  Let's say, you

          16    have two sponsors, two companies, okay?  They

          17    have the same data and they rely on p-values.

          18    One performed an interim look, and didn't

          19    stop at the interim look, and went to the end

          20    of the trial, okay, relying on the p-value.

          21              The other one had exactly the same

          22    result, did not perform the interim look.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            142


           1    Okay, the one that did not perform the

           2    interim look obtained the p-value that's,

           3    let's say, at the end is 0.43, and got the

           4    device approved.  The one that did perform

           5    the interim value had to -- interim analysis

           6    had to have a penalty because of that, and

           7    0.43 might not have been enough for approval.

           8              So that's a violation of the

           9    likelihood principle, okay?  But we can --

          10    one can argue that, look, the person that

          11    performed the interim look had the chance of

          12    having -- stopping at that point so the type

          13    I error increased, and didn't stop, but could

          14    have.  And that could have is something

          15    that's not reflected in the likelihood.

          16              You know, the likelihood is only --

          17    reflects only the data you obtained.  It

          18    doesn't reflect the fact that you could have

          19    stopped, but didn't.  So that's one example

          20    of violation of the likelihood principle.  I

          21    guess that's what they -- I don't know if

          22    that answers the question or not.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            143


           1              MS. SETHURAMAN:  Sort of.

           2              MR. CAMPBELL:  Thank you, Telba.

           3    The thing to remember about that particular

           4    example is we would ask a company to

           5    pre-specify exactly what they would intend to

           6    do, and hopefully that would all have been

           7    specified in advance.  Yes.

           8              MR. BERRY:  Don Berry, MD, Anderson

           9    and Berry Consultants.  I got up here to ask

          10    a question of Laura and I will, but while I

          11    was standing here, I generated a couple of

          12    comments on what's been said.  The likelihood

          13    principle -- so Telba's example and your

          14    description of it, and getting back to

          15    Larry's point, doesn't that say that this

          16    Bayesian approach where you're checking the

          17    operating characteristics becomes Frequentist

          18    and -- Frequentist in a way that in fact is

          19    not consistent with the likelihood principle?

          20                   (Laughter)

          21              MR. BERRY:  That was -- let's put

          22    that as a rhetorical question.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            144


           1              MR. CAMPBELL:  Okay.  Right.

           2              MR. BERRY:  These are important

           3    issues.  Maybe they'll come up this

           4    afternoon.  These are important issues, and I

           5    take Laura's point that this is a

           6    transitional period.  I mean, eventually

           7    we'll all be doing wonderfully Bayesian

           8    things.  In the meantime, there may be a

           9    transitional period where we have to touch

          10    these -- I think, really quite important

          11    bases.

          12              The -- these are not -- these kinds

          13    of things, adjusting for multiplicities,

          14    adjusting for interim analyses, it's really

          15    quite important.  The operating

          16    characteristics -- one way to calculate a

          17    very important probability, the Bayesian

          18    predictive probability of success of the

          19    trial, is to write down the probability of

          20    success of the trial for each value of the

          21    parameters.

          22              That's the operating




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            145


           1    characteristics.  That's the power curve.

           2    And then if you take the next step, to

           3    average with respect to the values of the

           4    parameters and the probabilities associated

           5    with the parameters that gives you the

           6    Bayesian probability.  So in a way it's an

           7    interim step along the way, which leads me to

           8    my question for Laura.

           9              She said you don't want to have a

          10    predictive probability of success at the

          11    beginning of the trial that's 99 percent.  It

          12    is quite reasonable and appropriate for

          13    someone to design a trial where they have a

          14    very optimistic prior on the basis of an

          15    eventual analysis, which will use the

          16    regulatory prior, possibly a non- informative

          17    prior.

          18              So there could be two priors in the

          19    same trial, one that's the thing you are

          20    trying to demonstrate and the other is your

          21    own prior, which may be quite optimistic, and

          22    you might reasonably have a 99 percent




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            146


           1    probability of success of a trial, even

           2    though the FDA's prior would not allow for

           3    that.  So is that consistent, Laura, with

           4    what you were saying?

           5              MR. CAMPBELL:  Come up here and --

           6              MS. THOMPSON:  You may have to

           7    rephrase it in a question.  You're saying you

           8    could have -- that it's reasonable for a

           9    company to have a high prior predictive

          10    probability of study success based on their

          11    prior, but there could be other -- certainly

          12    yeah, non-informative prior would not likely

          13    have that high probability of success -- and

          14    then I don't understand the question.

          15              MR. BERRY:  All right.  So you

          16    could design a trial that has the

          17    characteristics that you wanted to have,

          18    recognizing in the trial design, in the SAP

          19    that you are going to have to persuade the

          20    FDA panels, the community, of the

          21    appropriateness and efficacy of your devise.

          22    And so in the trial design, you have a prior




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            147


           1    distribution, which is the -- let me call it

           2    the regulatory prior, but the design is

           3    driven by, and is based on information that

           4    you have that you are not incorporating in

           5    the regulatory product.

           6              MS. THOMPSON:  Okay.  Right.

           7              MR. BERRY:  But in your prior --

           8    and it's indeed while you -- why you are

           9    running the trial in the first place --

          10              MS. THOMPSON:  Right, right.

          11              MR. BERRY:  -- because you really

          12    believe in your product.

          13              MS. THOMPSON:  Okay.  One slide

          14    that I had mentioned, I think that deals a

          15    little bit with that concept is, when I said,

          16    well, you could -- this had to do with sample

          17    size -- you could determine what might be

          18    considered a conservative sample size using a

          19    non- informative prior, which certainly would

          20    be acceptable to FDA, something that will be

          21    equivalent to using no prior information.

          22              But then analyze your trial using




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            148


           1    the full prior information you have.  So in

           2    some sense, yeah, you're saying, well, this

           3    is -- these are going to be -- this is going

           4    to be the -- how I'm going to operate the

           5    trial according to what, you know, FDA has,

           6    but when I'm going to make a decision, it's

           7    going to be based on the prior information

           8    that I have.

           9              That may not be just purely a

          10    statistical answer.  It may also have to do

          11    with, you know, whether -- you'd have to also

          12    convince clinicians as well, so.  Did I

          13    understand your question or not?

          14              MR. BERRY:  I don't know.  Did you?

          15              MS. THOMPSON:  Well, I -- did I

          16    answer -- are you --

          17              MR. BERRY:  So one of the decisions

          18    that you mentioned, the thing that you said

          19    may not be statistical is running the trial

          20    in the first place.  That decision is up to

          21    the company.  Whether the company runs the

          22    trial --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            149


           1              MS. THOMPSON:  Right.

           2              MR. BERRY:  -- depends on what they

           3    understand and know about the biology, know

           4    about their device, and are willing to or not

           5    predict success.

           6              That information is a perfectly

           7    reasonable thing from the Bayesian

           8    perspective for them to use in setting up the

           9    design of the trial, the sample size

          10    included.  And then persuade the FDA on the

          11    basis of operating characteristics that this

          12    is a reasonable thing to do, recognizing that

          13    the eventual analysis will not use the

          14    information that they have in their heads and

          15    in their hearts.

          16              MR. CAMPBELL:  Let me jump in,

          17    because I don't think there's a disagreement

          18    here.  In fact, many large companies have

          19    decision-analytic arms that make marketing

          20    decisions.  And in order to make those

          21    decisions, they might use prior information

          22    that might be subjective, might be




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            150


           1    non-quantifiable, and yet it's certainly up

           2    to the company to proceed with that.

           3              And they might use that to

           4    encourage investigators to participate in the

           5    trial.  And so I don't think there is a

           6    difficulty here as long as one is willing to

           7    separate, sort of, the regulatory prior, if

           8    you want, from priors that companies might

           9    have for decision-making purposes.  I think

          10    that's your point.

          11              MR. BERRY:  Yes.

          12              MR. CAMPBELL:  Did you want to --

          13              MS. IRONY:  Yeah.  I have one other

          14    point though, if I understood your question.

          15    Let's say the company, as you said, has a not

          16    optimistic prior and decides to pursue the

          17    clinical trial.  When you submit it to the

          18    FDA, the FDA prior might be so pessimistic

          19    that we might see -- not one to put this

          20    patients necessary at risk even on the trial,

          21    and we might say, "Okay, we don't want to run

          22    this trial, because we think that's very,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            151


           1    very likely that the device will not work."

           2              So it can work also on the reversal

           3    side, in other words, we might have a -- such

           4    a pessimistic prior that despite the

           5    sponsor's optimistic prior we might even deny

           6    the trial in terms of that particular device.

           7              MR. BERRY:  Okay.  And so if you do

           8    in fact have -- if you do allow the trial,

           9    and you have a pessimistic prior, that is

          10    precisely what the company has to understand,

          11    and build it into the design of the trial.

          12    And they might come to you and say, "You have

          13    a pessimistic prior, we have an optimistic

          14    prior, let's start out the trial cautiously,

          15    recognizing your prior.

          16              And if it turns out that maybe

          17    things are going in the positive direction,

          18    we'll continue the trial.  Hence, a wonderful

          19    Bayesian adaptive design."

          20              MS. IRONY:  Well, but I have --

          21              MR. CAMPBELL:  I'm going to cut

          22    this off.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            152


           1                   (Laughter)

           2              MR. CAMPBELL:  I think this will

           3    make for an interesting discussion this

           4    afternoon.  We have time for one more quick

           5    question, and then we'll break for lunch.

           6              MR. MUGGLIN:  Hi.  I'm Andy Mugglin

           7    from the University of Minnesota.  I have a

           8    point first of all -- I think food for

           9    thought, and then I'll ask a question.  The

          10    point is that I think that my intuitive

          11    understanding of probability, is not

          12    necessarily as good as I'd like to believe it

          13    is.  And I think that's true of many of us,

          14    and in particular, on the subject of

          15    predictive probabilities, and perhaps when I

          16    start a trial, what is the prior probability

          17    of my success criterion versus -- you know,

          18    it was mentioned today that that shouldn't be

          19    too high, or the prior predictive probability

          20    shouldn't be too high.  But what exactly that

          21    is too high is left open.  And I acknowledge

          22    that.  But what does it mean if that prior




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            153


           1    probability is 0.8?

           2              And how much evidence does it take

           3    to go from 0.8 to 0.9 versus, you know, 0.95

           4    to 0.99 versus 0.5 to 0.9 and so forth.  And

           5    the level of information is very nonlinearly

           6    related to the probability.  So in fact, if

           7    you start with a prior of 0.99 and you needed

           8    to move to a posterior of 100, it would take

           9    infinite information.  So this is something I

          10    think that should make us all think about.

          11    What is it you can move to have a high

          12    probability of success when I start, relative

          13    to how high that probability is when we

          14    finish?

          15              I guess the question I'll ask is

          16    this one, to FDA.  With all of the things

          17    that have to go into this, including the --

          18    appears to be extra model checking, extra

          19    simulations, and so forth., in the FDA's

          20    view, do you hold a Bayesian submission to a

          21    more rigorous examination perhaps, than you

          22    might with an off-the-shelf, more tried and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            154


           1    true Frequentist one?

           2              MR. CAMPBELL:  It's an excellent

           3    question, and I guess I'd like to address

           4    that head on.  I guess what my response would

           5    be is that, "No, we do not."  And that the

           6    only thing that makes it a little more

           7    burdensome for us concerns the software

           8    issue.  Because there is no validated

           9    software out there, as would be for any new

          10    methodology, there is an additional challenge

          11    and additional burden that the FDA was --

          12    would have in allowing companies to come with

          13    that new methodology no matter what it was.

          14              And that burden is, in this case

          15    because there is no validated software, we

          16    might need to run our own simulations, our

          17    own MCMC to make sure it converges with the

          18    software -- this particular piece of software

          19    that might be different from the one that the

          20    company did.  But that's no different than

          21    what would happen with any new methodology

          22    for which the use of the methodology occurs




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            155


           1    before the software is fully developed and

           2    validated.

           3              So my response is no -- that it is

           4    not given a review that is any more out of

           5    the ordinary, any more detailed than any

           6    other submissions.  And you should know that

           7    we spend a lot of time on other submissions

           8    that aren't Bayesian.  One of the perceptions

           9    that we have within the agency is that CDRH

          10    in terms of statistics does Bayesian

          11    statistics.

          12              Well, we do that, but it probably

          13    doesn't take more than 5 or 10 percent of our

          14    time and effort.  It is not the case that we

          15    spend half our time doing Bayesian things and

          16    division of biostatistics at CDRH.  So I

          17    hope --

          18              SPEAKER:  Would like to.

          19              MR. CAMPBELL:  Well, and that may

          20    change over time, but that's up to the

          21    company.  It's not up to us.  We are not

          22    going to tell companies you must do a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            156


           1    Bayesian submission.  It's up to the

           2    companies to figure out whether for them it's

           3    attractive or not.

           4              So with that let me suggest that we

           5    take a break for lunch.  I think this has

           6    been a very interesting discussion.  I would

           7    encourage you to think about the document and

           8    (off mike ) you might make.  We'll be back at

           9    1:15.  Please come back promptly.  Thank you.

          10                   (Whereupon, at 11:54 a.m., a

          11                   luncheon recess was taken.)

          12

          13

          14

          15

          16

          17

          18

          19

          20

          21

          22




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            157


           1              A F T E R N O O N  S E S S I O N

           2                                             (1:20 p.m.)

           3              MR. KESSLER:  Good afternoon.  The

           4    crowd seems to have thinned a little tiny

           5    bit.  Mostly, people are coming back from

           6    lunch.  A few people got all the information

           7    they needed from this morning lecture,

           8    they're off planning their Bayesian designs

           9    over in the Office of Device Evaluation right

          10    now as we speak, I'm sure.

          11              A few logistic things to start.

          12    First of all, the guards, the security guards

          13    have asked that at the break, we don't

          14    congregate right at the entrance.  And so if

          15    you can disperse a little bit, they'd

          16    appreciate it.  I've also been asked to tell

          17    you about soda machines on the second floor.

          18    If you want coffee, go to the next building,

          19    meaning, out those doors, make a left, go

          20    across to the next building, you're on the

          21    second floor.  So if you need sort of a

          22    wakeup at the break time.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            158


           1              A few other things.  There are

           2    supposed to be people on the phone, just so

           3    you know, who've been able to dial in.  They

           4    are listen-in only.  So just in case you're

           5    there's outside people who wanted to call in

           6    for the day.  If you would like to see the

           7    presentations that were made this morning,

           8    they are all on the Web.

           9              Again, I cannot emphasize to you

          10    too strongly that Greg and his team would

          11    very much like comments submitted to the

          12    Docket.  The comments you make today we'll

          13    make it into a transcript and we'll certainly

          14    take under consideration, but they would be

          15    much more valuable to us as well if they're

          16    submitted in writing to the Docket by August

          17    21st.

          18              Again, our guidance documents, in

          19    essence, are always open.  So there is no

          20    prohibition from sending in comments after

          21    August 21st.  However, they would tend to go

          22    to Greg.  The process for dealing with them




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            159


           1    is a little harder because of revisions.  So

           2    if you can get something in by the 21st of

           3    August, we'd really appreciate it.

           4              Again, when we open the discussion

           5    to the floor today, as we will after the

           6    panelists each make their brief

           7    presentations, we'd appreciate it if you

           8    would clearly tell us your name and what your

           9    company you're from so we can include that on

          10    the transcript.  And this afternoon, after

          11    the panel makes their presentations, we'd

          12    like a healthy interchange between the

          13    audience and the panel.

          14              The purpose of the panel and, in

          15    fact, the purpose of the day is not to defend

          16    the guidance.  Questions, clarifications,

          17    comments are certainly welcome, but we're

          18    hoping for an open interchange in order to

          19    move the documents forward.  So I'd like to

          20    turn to the panel today and before I do, I'd

          21    like to make just a couple of brief comments.

          22              First of all, we have an




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            160


           1    outstanding panel this afternoon.  You're

           2    going to hear from Susan Alpert, who is the

           3    senior vice president, chief quality and

           4    regulatory officer at Medtronic and a former

           5    member of the Center for Devices.  She headed

           6    the Office of Device Evaluation.  We were

           7    co-office directors and she actually has an

           8    enormous wealth of knowledge about the agency

           9    and regulatory issues.

          10              You will hear from Don Berry,

          11    professor and chairman, Department of

          12    Biostats at the MD Anderson Cancer Research

          13    Center.  And frankly, he deserves a lot of

          14    the credit for a lot of the Bayesian work

          15    that we've done in the Center for Devices.

          16    Greg and I collaborated with Don very early

          17    in this process and a lot of his work has

          18    gone into formulating some of our ideas and

          19    direction.

          20              You'll hear from Barbara Buch,

          21    who's deputy director, Division of General,

          22    Restorative, and Neurological Devices in the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            161


           1    Office of Device Evaluation, give you a very

           2    practical orientation to what ODE looks for.

           3    And I hope you'll have questions of her.

           4              Bailey Lipscomb, who is the vice

           5    president and -- of Clinical Affairs for

           6    Medtronic Sofamor Danek division and has some

           7    experience with the Bayesian application to

           8    us -- will talk to us as well.

           9              And finally, Janet Trunzo, who is

          10    the executive vice president of Technology

          11    and Regulatory Affairs for AdvaMed, the

          12    Advance Medical Technology organization

          13    downtown, the biggest device trade

          14    organization in the United States, maybe even

          15    in the world.  And Janet, again, helped Greg

          16    sponsor the Bayesian meeting that we had

          17    about eight years ago.  So she's been with us

          18    all along and it's great to have her on the

          19    panel.

          20              It's not often one has the

          21    opportunity to witness cultural change.  And

          22    it is even less often that we have the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            162


           1    opportunity to actually participate in the

           2    formulation of this kind of change.  We've

           3    had the good fortune to both develop and

           4    affect a paradigmatic shift in the device,

           5    design, conduct, and analysis of clinical

           6    trials.  And we're very excited about that.

           7              We sincerely believe that this

           8    meeting will take us another step along that

           9    road.  And, in fact, it's in the service, as

          10    Dan Schultz and others talked about this

          11    morning, to try to get beneficial medical

          12    devices, safe and effective devices to the

          13    market, to patients as efficiently and

          14    effectively as we possibly can.  That's our

          15    job in the Center for Devices and that's what

          16    the -- many of the companies will tell you,

          17    that's their job as well.  So we hope that

          18    the work that we've done here will further us

          19    in that regard.

          20              This afternoon, after the panel

          21    each speaks for about five minutes, there are

          22    questions we might consider.  So here's a few




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            163


           1    things for you to think about.  Does this

           2    document, meaning, the guidance document, go

           3    too far or not far enough in the use of

           4    Bayesian statistics in medical device

           5    studies?  Does it provide a reasonable

           6    framework for the use of Bayesian statistics

           7    in those studies?  If you have experience

           8    with Bayesian statistics in this industry, do

           9    you think the draft describes current

          10    practice?  Are there specific items you would

          11    like to see added or removed and if so, can

          12    you tell us why?  And finally, are there

          13    particular sections in the document that

          14    should have major revisions?  If you have any

          15    of those questions or comments, we'd

          16    appreciate hearing from them.

          17              I'm going to turn now first to

          18    Susan Alpert, Dr.  Alpert from Medtronic.

          19    And each panel member in turn will talk for a

          20    very brief five minutes.  If you see me or

          21    hear me clear my throat, or get up, or

          22    fidget, it's because you're going close to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            164


           1    five minutes.

           2              MS. ALPERT:  Thanks, Larry and

           3    thanks for all the nice comments.  I want to

           4    make a couple of introductory comments, sort

           5    of general comments.  And then I do have some

           6    specific comments about the guidance and then

           7    some specific comments about a couple of

           8    other things that were mentioned this

           9    morning.

          10              First of all, I would like to thank

          11    FDA for the guidance.  I think having a

          12    guidance in a statistical area that doesn't

          13    talk only in statistical jargon but it's

          14    actually written in English that other people

          15    can understand is actually a good thing.  For

          16    example, I think this is the kind of guidance

          17    that we in industry could take and share with

          18    a lot of other people, not just the people

          19    who design clinical trials who may be

          20    familiar, but with people in R&D and in the

          21    business side to understand what the change

          22    is and where we're going with the change.  So




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            165


           1    I really complement the group.  Thank you

           2    really for doing that.

           3              But I also want to say that using

           4    this, actually using this is going to take a

           5    lot of education of those people, the people

           6    outside of the group that comes and talks

           7    about Bayesian statistics, and designs

           8    clinical trials, and worries about the way in

           9    which we use statistics to manage our

          10    products, and worries about how small can we

          11    make a trial or how short can we make a trial

          12    and how do we get the right data because this

          13    takes a lot of education in terminology, as

          14    we heard this morning, and a lot of education

          15    in use.

          16              And part of that education is the

          17    fact that this takes a very different amount

          18    of work upfront.  And I think although it's

          19    said in the guidance, that's probably

          20    something that needs more emphasis in the

          21    guidance and in the early part of the

          22    introductory stuff to the guidance.  This is




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            166


           1    not standard.  You get somebody to come in

           2    and work with you or you hand it to one of

           3    your statisticians to kind of do a little

           4    statistical analysis.  This is a lot of work.

           5    It's worth it, but it's a lot of work upfront

           6    in the design and in figuring out whether

           7    it's the right thing to do.

           8              I think a little more in the

           9    guidance as well about when it's not the

          10    right thing to do.  The -- some examples,

          11    perhaps, of the kinds of situations where

          12    Bayesian -- right now because of our lack of

          13    experience and understanding and the lack of

          14    all the tools that Bayesian may not be the

          15    right thing to start with in some situations.

          16    And I think we need to grow into this a

          17    little bit.  But as Donna-Bea and Dan said

          18    this morning, a lot of the way in which we

          19    develop products clearly lends itself to

          20    this.  So I'll say that.

          21              A couple of comments to industry,

          22    in general, that I would just want to make,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            167


           1    one is that there is a one-line, or maybe

           2    it's two, comment in the guidance about

           3    agreement meetings.  One, it should be

           4    expanded and two, if you are not aware of

           5    what agreement meetings are and what they can

           6    do for you, read FDAMA, read the guidance on

           7    agreement meetings, and take more agreement

           8    meetings because what it will do for you is

           9    get a commitment from the ODE reviewer and

          10    the statistician as to what you are doing in

          11    the design of the trial.

          12              Don't expect to get, you know, your

          13    entire plan laid out and all of it agreed to

          14    in all of the detail, but the basic approach

          15    because that's the only way you can be sure

          16    that it's not going to change because if you

          17    read this guidance, you will see that there

          18    are many places where there is going to be a

          19    second look at how you're doing the

          20    statistics, a second look at whether the

          21    model is right, a second and third look on

          22    the statistics.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            168


           1              You want a commitment that, in

           2    fact, your general approach and the use is

           3    going to be accepted and you don't lose that

           4    with other looks or with changes of reviewers

           5    or changes of other people.  So as a general

           6    approach, you want to get some of this locked

           7    down.

           8              A second thing is that for

           9    companies, there's -- it needs a lot more

          10    commitment on your part to decisions for when

          11    you're going to make decisions and whether

          12    you're going to have utility analysis and

          13    what your criteria are really for a win and

          14    to be committed to it, which means if you

          15    don't make it, you've failed and it's over

          16    and you have to do another trial.  And that's

          17    a big risk and we all have that issue with

          18    our management.

          19              A third thing is that the issues

          20    around sample size.  Again, for companies,

          21    you know, the methodology isn't supposed to

          22    be locked on sample size, but regulation is.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            169


           1    And there are lots of things that drive

           2    sample size and they may not have to do with

           3    how you design your clinical trial.  If

           4    you've got something brand new, it may be

           5    that there is a need to have a certain amount

           6    of experience with a product.  If you've got

           7    a combination product, it's a certain amount

           8    of exposure to some other product.

           9              Irrespective of what you can do

          10    with a tool like Bayesian to make your trial

          11    small, you may still need an experience with

          12    the product that's broader than the absolute

          13    necessity to show safety and effectiveness in

          14    your clinical trial because it's not only

          15    about one clinical trial, it's really about

          16    the product.  And if you've got something

          17    really new, you need to pay attention to

          18    that.

          19              Bayesian won't always work.  It

          20    won't always be a smaller or shorter trial.

          21    It may be a larger or a longer trial.  So you

          22    can't commit to Bayesian and then ignore the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            170


           1    fact that you may want to look at Frequentist

           2    as well in order to see where you're going.

           3    So I -- you know, the issue -- and then the

           4    issue is this is going to cost money upfront.

           5    Again, this is all stuff for the industry.

           6    This is going to cost more money to do the

           7    upfront work, but it saves you a lot at the

           8    end, if it works.  And we have to get into

           9    the habit of developing the skill set.  And

          10    at the beginning, it's going to be hard

          11    because we're going to have make investments

          12    to get that skill set developed in our

          13    companies in order to then get all the

          14    benefits later on.

          15              In terms of the guidance, the early

          16    stuff in the language -- English language

          17    stuff is good and then there's references to

          18    various statistical stuff.  I think a little

          19    bit in the middle would be helpful, little

          20    stuff that ties with examples and things, on

          21    some of the things that are in the guidance,

          22    some expansion would be good.  I also looked




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            171


           1    at a question and the question came up during

           2    this morning's question and answer period

           3    about the fact that you look through -- that

           4    Bayesian is a constant look at the data.  And

           5    the questions and the concerns of -- that FDA

           6    has about our looking at data on an ongoing

           7    basis and what bias that introduces,

           8    potentially, into trials is an issue that we

           9    need to pay attention to.

          10              Sensitivity analysis needs a little

          11    more explanation and description in the

          12    guidance document.  The caveats on the use of

          13    literature as your prior, I think, could use

          14    expansion again because I think there is a --

          15    some risk there.  And then a real

          16    understanding of what the risk is in choosing

          17    a model and having that model challenged, I

          18    think, is necessary.

          19              The last thing I want to say is.

          20    Susan Gardner talked this morning and then

          21    there was some presentation about how

          22    Bayesian might be used in post-market.  And I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            172


           1    want to put a caveat on the table and a

           2    suggestion to FDA.  It might be really good

           3    to have -- to use Bayesian to do some

           4    data-mining.

           5              First of all, let's recognize that

           6    the adverse events database is not a complete

           7    database.  The adverse events database is

           8    uneven across lots of different products.

           9    Even within a product family, each of us,

          10    each company makes products that are very

          11    different and the adverse events profiles are

          12    very different, they are not the same.

          13              And I think it's going to take a

          14    lot more knowledge about product and not just

          15    knowledge about how to use the statistics in

          16    the database in order to make that viable.

          17    And I suggest that OSB work with the industry

          18    on that because I think there is a lot of

          19    risk there for the agency as well as for

          20    companies.  And we need to work on that more

          21    carefully together.  It's a big deal and it's

          22    very different, brand new and we need to work




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            173


           1    on it.

           2              MR. KESSLER:  As usual, Dr. Alpert

           3    puts a half- hour talk in six minutes, so

           4    thank you.  Dr. Berry.

           5              MR. BERRY:  You notice how he got

           6    in the fact that you went over five minutes?

           7              MS. ALPERT:  Of course, exactly.

           8              MR. KESSLER:  It's a warning for

           9    you.

          10                   (Laughter)

          11              MR. BERRY:  So Greg asked more

          12    questions than Larry read off, something like

          13    a total 20 questions and he said, "And you

          14    have five minutes to address these."  And he

          15    said, "You can't say the word drug."  But I'm

          16    going to say the word drug once.  What is

          17    happening in CDRH is having an influence

          18    throughout the rest of the medical world, so

          19    much so if you read The Wall Street Journal a

          20    few weeks ago, you saw that lots of drug

          21    companies are getting into the act and doing

          22    Bayesian adaptive designs.  So it's a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            174


           1    veritable tsunami.

           2              Does the guidance go far enough but

           3    not too far?  I think that's correct.  Does

           4    the draft describe actual practice?  I've

           5    dealt with the FDA a good bit over the last

           6    10 or so years for companies.  And I think it

           7    does exactly describe what has happened, the

           8    statisticians at the FDA have taught me, they

           9    are, as regulators go, incredibly willing to

          10    discuss things, and give into points when

          11    they decide that maybe their -- the original

          12    position was not correct.  So it's been very

          13    positive with the statisticians and also

          14    positive with the non- statisticians.

          15              Some clinicians have said -- have

          16    recommended to companies taking a Bayesian

          17    approach and some have said to me, "Well, if

          18    it's Bayesian, then it's okay."  When should

          19    you consider Bayes, this is one of the

          20    greatest questions and we talked this morning

          21    about using -- when you can use prior

          22    information for predicting future results




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            175


           1    incorporating other information in the trial.

           2    Information from outside the trial,

           3    information that's not the primary endpoint,

           4    but that's in the trial.

           5              The guidance uses the word,

           6    "surrogates," and my advice to the -- to Greg

           7    is to get rid of the word, "surrogates"

           8    throughout.  It's a hot button term that

           9    means different things to different people

          10    and the way we use it and have used it

          11    successfully in device development is not

          12    what most people think of as being

          13    surrogates, the apprentice criteria for

          14    surrogates for example.  It is as auxiliary

          15    variables, it is a helpful variables.  It is

          16    things -- and the guidance says something

          17    about validation.  We validate within the

          18    context of the trial, we incorporate the

          19    information and if it turns out not to be

          20    predictive or correlative, we learn that, and

          21    we don't use it.

          22              Heavily -- I mean, one of the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            176


           1    major, major distinctions between the

           2    Bayesian approach and the Frequentist

           3    approach, and now speaking empirically, not

           4    philosophically is modeling.  Bayesians tend

           5    to build models that may be complicated for

           6    relationships among things and then learn

           7    about the relationships, something that is of

           8    course, not prohibited in the Frequentist

           9    approach, but is not generally done.

          10              Advantages and disadvantages --

          11    Telba talked about saving time and resources

          12    on average -- everybody stresses the

          13    possibility that it could be bigger including

          14    Susan just now, but in contrast what Susan

          15    said when it is bigger in the Bayesian

          16    approach, it's because it has to be, it's

          17    because you don't know the answer, you're

          18    looking at the data.  You know, you're doing

          19    something really revolutionary in medical

          20    research, you look at the data and you decide

          21    that you don't know the answer and you're

          22    going to have to continue the trial.  So you




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            177


           1    get an answer with greater precision and

           2    therefore greater power.

           3              An advantage, better treatment of

           4    patients in the trial, better treatment of

           5    patients outside the trial.  Dr. Tillman

           6    mentioned, and Susan mentioned working in

           7    advance and getting the team.  This is a

           8    positive thing quite apart from the adaptive

           9    nature.  Simply getting together and saying,

          10    "What would I do if such and such, what are

          11    the implications of these data?"  In view of

          12    my running out of my five minutes, I'll skip

          13    the thing about -- and I'm going to say -- on

          14    page 7, it makes a statement, "FDA will carry

          15    out a detailed statistical review of a

          16    Bayesian submission," that sounds ominous.

          17    Greg talked about that this morning, it

          18    shouldn't be ominous.  One last point -- can

          19    I make one last point Larry?  Resource

          20    considerations --

          21              MR. CAMPBELL:  You have a whole

          22    minute.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            178


           1              MR. BERRY:  I have a whole minute?

           2    Okay, one minute, all right.  I can make this

           3    point.  Dr. Gray said this morning, you need

           4    statistical advice and you may say, "Where do

           5    I get it?"  As --

           6              MR. CAMPBELL:  Wait, wait, wait, no

           7    advertisement --

           8                   (Laughter)

           9              MR. BERRY:  This is not an

          10    advertisement, this is not -- it is not an

          11    advertisement.  And you have to subtract 10

          12    seconds for that.

          13                   (Laughter)

          14              MR. BERRY:  Where do you get it,

          15    you make -- and I want to consider just two

          16    possibilities.  Many of your companies don't

          17    have statisticians, only the bigger ones do,

          18    and you may want to hire a statistician.  And

          19    this is -- speaking about this specific

          20    instance, but also the general point.  Let's

          21    say, you have two possibilities.  You can

          22    hire a brand new statistician who is a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            179


           1    whiz-bang at MCMC and knows Bayesian stuff a

           2    la a graduate school, that's one choice.  And

           3    the other choice is you can hire a

           4    Frequentist statistician who knows clinical

           5    research and doesn't -- you know, remembers a

           6    little bit of a Bayes theorem -- is not

           7    antagonistic to the Bayesian approach, but

           8    know something about it, which should you

           9    choose?  The answer is the latter.

          10              You will be disappointed if you

          11    hire a new Ph.D.  Statistician expecting that

          12    they are going to be able to carry your ball

          13    to the agency and understand the nuances of

          14    clinical research, you really should have

          15    somebody who can train that person, and if

          16    you had your choice, choose the grey haired

          17    Frequentist instead of the bright-eyed

          18    Bayesian that's all I'll say, thanks.

          19              MR. CAMPBELL:  You know, I'm dying

          20    to make a comment about grey-haired Bayesians

          21    but I'll pass this to Barbara.

          22              MS. BUCH:  Well, I want to thank




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            180


           1    Susan and Don for saying a lot things that I

           2    was actually going to point out.  I also want

           3    to stay -- say that I'm not a statistician,

           4    what I know about Bayesian Statistics is

           5    about a drop in the ocean and I've learned it

           6    here at FDA.  So my approach today is going

           7    to be a little bit different than what may've

           8    been assigned.

           9              I think the guidance document is

          10    not only guidance for industry, but also for

          11    FDA staff.  And I would like to submit that

          12    we should be thinking about training our own

          13    staff with this document as well as industry.

          14    So some of the things that I'm going to talk

          15    about today are coming from the perspective

          16    of a reviewer and from CDRH, so take it as

          17    such.

          18              I'd like to start out just saying

          19    that I applaud the frequent assertion in the

          20    guidance document that this Bayesian

          21    Statistics is not a substitute for poor

          22    clinical design, and this is something that's




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            181


           1    held very high at FDA.  And you know, you

           2    shouldn't deviate from sound clinical trial

           3    methods to do something that's hot and sexy

           4    and new and exciting.  That being said, the

           5    things that come up from our point of view

           6    are five major areas.

           7              One is when do we use this, when is

           8    it appropriate, and if you ask a Bayesian,

           9    they will say, "always."  And we may not

          10    agree with that.  So it has been mentioned it

          11    takes a lot of upfront time and financial

          12    interest, and it's a good thing to reiterate

          13    that coming to FDA early and talking about

          14    these things, seeing whether this is an

          15    approach that you want to pursue is the first

          16    step, and is the most important.

          17              I think the guidance needs to go

          18    into a little bit more about when this

          19    approach should be used and what the pros and

          20    cons are.  For us, the planning phase is the

          21    most important and we have the most influence

          22    in that area.  So I think one of the topics




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            182


           1    that plagues clinicians in particular is when

           2    is a prior appropriate, and how do we decide

           3    that.  So I'd like to see a link maybe in the

           4    guidance, or some more time spent in that

           5    area to develop some criteria, and to help

           6    guide both industry and us to decide, you

           7    know, how do we decide a prior is good or

           8    bad, and some of the things that were

           9    mentioned earlier in the day go to that as

          10    well.

          11              So -- the third thing is I'd like

          12    to see some more information on the decision

          13    analysis and the errors that can be made

          14    during that process, deciding when and what

          15    kinds of decisions that you're going to use

          16    to guide your analysis, and then when it

          17    comes down to the analysis, I agree

          18    completely, which is I thought I'd never say

          19    this, what -- Dr. Berry's comment about

          20    surrogates.  Those are things that -- usually

          21    in the center of devices, we do not like to

          22    use.  We like to use endpoints that are




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            183


           1    directly measurable and equate with the

           2    outcomes that are desired.

           3              But some of the questions that we

           4    come up with in analysis plague us, and I put

           5    it to the floor to help put those kinds of

           6    things into this guidance and help FDA as

           7    well as industry answer some of these

           8    questions, and -- you know, things like how

           9    stringent do we need to be with the

          10    exchangeability concept.  How do we

          11    distinguish between actual data or imputed

          12    data when it's predicted, how do we -- one of

          13    the big problems that clinicians have is how

          14    do we reconcile for ourselves -- and this is

          15    something Dr. Alpert brought up -- is that

          16    multiple looks at the data, and changing the

          17    clinical trial design during the trial don't

          18    influence or bias the outcome, and how do we

          19    know when they are based on the simulations

          20    that these things, when we look at the

          21    simulations, aren't going to be confounded by

          22    the real patients who usually don't read the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            184


           1    textbook and don't do what they're supposed

           2    to do in the trial when it comes to the

           3    outcomes.

           4              And then, finally, what do you do

           5    when it doesn't work?  And how do we

           6    understand and know what to do when we've

           7    chosen the wrong prior even when we've agreed

           8    to?  And how do we reconcile that?  And then

           9    finally, the thing that comes up from a

          10    regulatory point of view and influences the

          11    clinician greatly is labeling.  And how do we

          12    put that terminology into understandable

          13    terms and understand ourselves what it means,

          14    and be able to translate that information to

          15    a user or a patient.

          16              And so those are the kinds of

          17    things that come up from our point of view.

          18    And the final thing I'd like to talk about

          19    and submit as something for the group that's

          20    writing this document is the comment that has

          21    been brought up before about predictive

          22    probability, where you have a situation where




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            185


           1    the prior information, "Overwhelms the

           2    current data potentially creating a situation

           3    where unfavorable results from the proposed

           4    study get masked by favorable prior results."

           5    This is deep to what FDA does.  And it's been

           6    mentioned several times.  And I think that

           7    comment has to be further expanded in the

           8    document to understand how we avoid those

           9    situations, and how we can identify those

          10    things when they happen.

          11              MR. CAMPBELL:  Thanks Barbara.

          12    Bailey.

          13              MR. LIPSCOMB:  The division that I

          14    work for at Medtronic markets spinal

          15    implants, and mostly they are Class I and II

          16    that require 510(k)s.  However, in recent

          17    years we've developed some Class III products

          18    that require PMA or PMA supplements.  And the

          19    earliest one of these PMA approvals that we

          20    received was back in 1999, and we use

          21    Bayesian statistics in the approval of that

          22    product and every PMA we've had since then




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            186


           1    has used Bayesian.

           2              In fact, probably on Greg's slide

           3    this morning, I think our -- we probably

           4    account for about 30 percent of those

           5    approvals that he was mentioning.  So we got

           6    into it at a good time with FDA.  They had a

           7    growing interest in using Bayesian

           8    statistics.  So I think we've kind of grown

           9    up together in this process.  With regard to

          10    the guidance document, and really from my

          11    vantage point not being a statistician is

          12    kind of like, what do we see, you know, kind

          13    of from the clinical side and the design of

          14    the study side is that I think the guidance

          15    document is a good document.  I think the

          16    drafters did a good job putting it in lay

          17    terms where it's easily understandable.  In

          18    fact, I actually began to understand some of

          19    this stuff.

          20              The points are well taken in

          21    Bayesian methods.  I think they more easily

          22    accommodate interim analysis.  And I think




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            187


           1    the useful available data can increase the

           2    likelihood of coming to an earlier conclusion

           3    in a study.  And perhaps you may be able to

           4    reduce the sample size and study as well.

           5    And I agree whole heartedly, I think it's

           6    real wise to speak early and speak often with

           7    FDA if you have a desire to go down the

           8    Bayesian path.

           9              Now, in terms of what the guidance

          10    document says, and kind of what we've seen in

          11    real life, there is a lot of correlations

          12    there, but a couple of things I think are

          13    worth mentioning.  One is what was brought up

          14    this morning about appeasing of the

          15    frequentist mindset is real.  We do the

          16    simulations to look at Type-I and II error.

          17    We look -- we have to set the number and

          18    timing of the interim analysis and preset the

          19    sample sizes, and so on.  And so that's a

          20    real thing.  So if you're going to go the

          21    Bayesian way, you better be prepared to spend

          22    the time on the front end to do that stuff.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            188


           1    The other thing is prior, the use of prior

           2    information.  We -- most of the times in our

           3    studies, we've used non-informative priors

           4    and it's works well for our types of studies.

           5              However, if you desire to use

           6    informative priors this is real controversial

           7    at FDA.  In fact, there are few times that

           8    we've actually tried to do it.  We've found

           9    that small differences in study or small

          10    differences in what we have data wise versus

          11    what we want to do new data wise can really

          12    create some stumbling blocks in terms of the

          13    use of the prior information.  And, frankly,

          14    I know what the guidance document says that,

          15    you know, you might be able to, you know, get

          16    some prior information from patient

          17    registries and things like that.

          18              For our types of studies at least,

          19    I think it would be very difficult, if not

          20    impossible, to have a good warm feeling about

          21    doing that and getting approval in a timely

          22    manner.  In terms of conclude remarks, our




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            189


           1    division in Medtronic where I work in

           2    Memphis, we are Bayesian.  I see no reason

           3    not to go Bayesian.  I think the pros

           4    outweigh the cons.  I think it works well

           5    with us, and we plan to use it as long as FDA

           6    is receptive to the Bayesian studies in the

           7    device area.  I think it all work -- also

           8    works well for us at Medtronic in Memphis,

           9    and the fact that we have run a large number

          10    of clinical studies that have large number of

          11    patients.

          12              So in future, hopefully, we will be

          13    able to glean some of the information and use

          14    that information as prior information.

          15    Frankly, I don't see major issues with the

          16    guidance document.  I know there is going to

          17    be comments made, and it's going to be

          18    comments made to the docket.  And those, I'm

          19    sure, will be incorporated.  I think it has a

          20    right depth.  I think it has a reasonable

          21    framework.  And I think it really does

          22    describe what the current mind set and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            190


           1    practice is at FDA.

           2              And Don, I got a question for you.

           3    If you are a pure Bayesian, why in the world

           4    do you need a guidance document?

           5                   (Laughter)

           6              MR. LIPSCOMB:  So you can --

           7    perhaps you can answer that.  The other thing

           8    is a final comment that I have in echoing

           9    what Barbara said, and we didn't get together

          10    on this one.  But I really do think it's

          11    incumbent on the ODE reviewers at FDA to

          12    understand a little bit about Bayesian, you

          13    know the principles and pros and cons of

          14    Bayesian, and how it compares and contrasts

          15    with a frequentist approach.  So I think that

          16    they should, you know, really do some

          17    training and embrace this guidance document

          18    when it's published.

          19              The other thing too is that I

          20    really urge the statisticians. Dr. Gray had a

          21    slide this morning where he was talking about

          22    this element of review on the IDEs and PMAs




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            191


           1    on -- it has the engineers and the clinicians

           2    and the statisticians.  I really urge the

           3    statisticians to have an equal -- be an equal

           4    player in those discussions to be assertive

           5    in these comments, and not default to the

           6    engineers and clinicians.

           7              I really -- the statistical part is

           8    very, very important.  And so I think that

           9    they need to step up to the plate on this.

          10    And finally, I like to thank the FDA for this

          11    invitation and the guidance document.

          12              MR. CAMPBELL:  Thanks, and our last

          13    speaker will be Janet Trunzo from AdvaMed.

          14              MR. TRUNZO:  Thank you.  I like to

          15    start with some general comments.  Greg

          16    pointed out this morning that FDA began to

          17    think about Bayesian statistics in 1997.  And

          18    then Greg came to HIMA at the time, and asked

          19    me to work on developing this workshop that

          20    was held in 1998.  But I think there is a

          21    significance to the date of 1997.  In 1997,

          22    FDAMA was passed and a keystone provision in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            192


           1    FDAMA and very important provision to the

           2    industry was the concept of least burdensome.

           3              And at the time when Greg came to

           4    me to put on this workshop, I saw the link to

           5    the least burdensome concept, and I though

           6    that Bayesian statistics, because of its

           7    promise to reduce sample size and the

           8    duration of the study, was indeed a least

           9    burdensome method of getting product

          10    approval.  And, therefore, I was very pleased

          11    to see that finally in the least burdensome

          12    guidance document, if you look at it, there

          13    is a statement and a link to Bayesian

          14    statistics.  So that all happened in 1998.

          15              And I really commend Greg for all

          16    the persistence and the effort he's put in,

          17    in promoting Bayesian statistics within the

          18    agency, the hiring of the Bayesian

          19    statisticians, the internal training at FDA

          20    and of course, with the industry.

          21              And now we have the guidance

          22    document.  And we also, at the same time, not




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            193


           1    only do we have these burdensome concepts.

           2    We have this new initiative that FDA has

           3    recently put out within the last two years

           4    called the Critical Path Initiative.  And

           5    when you take a first look at it, it -- there

           6    do not seem to be a lot of projects related

           7    to medical devices.

           8              But I see the Bayesian statistical

           9    method of promotion that's here today is

          10    indeed an example of a device related, or a

          11    Critical Path related initiative for devices.

          12    So I think that Critical Path is really an

          13    example of least burdensome.  And I like to

          14    --I wanted to connect all those dots.  But

          15    with any new initiative the devil is always

          16    in the detail.  And when you look at the

          17    guidance documents, defining when it is

          18    appropriate and when it is not appropriate to

          19    use the Bayesian methodology is an important

          20    feature.

          21              And if you think about the

          22    resources that are going to be needed to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            194


           1    promote this initiative we talked about the

           2    early agreement meetings and doing those

           3    meetings.  That means that not only does the

           4    industry have to prepare for it, but FDA is

           5    going to have to leverage all of its Bayesian

           6    expertise in those meetings too.  And I like

           7    to make one comment about the agreements that

           8    are reached in those early agreements.

           9    Remember 1997 had least burdensome, it also

          10    had these early agreement meetings in which

          11    those decisions were to be -- could not be

          12    changed.

          13              I think it's important for FDA to

          14    share ownership with the industry on the

          15    agreements that are reached because when it

          16    comes to the panel meeting there may be some

          17    challenge to those agreements.

          18              And I think it's important for FDA

          19    to share that in order to get the Bayesian

          20    methods more understood and more welcomed.

          21    And then also when there is any change in an

          22    initiative, I think I have, one minute, when




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            195


           1    there is any change in any kind of an

           2    initiative like this where you're moving from

           3    an old to the new, there is always this

           4    tendency to hold on to the old.

           5              And Bailey just mentioned this

           6    about the frequentist approach and the fact

           7    that he had to do all of the simulations on

           8    the Type-I and Type-II area.  I believe that

           9    at some point in time there should be a point

          10    in which we let go as we get more accustomed

          11    to a new initiative like Bayesian that we let

          12    go of the past and move forward in the

          13    future.  And I really do look forward to

          14    FDA's successful implementation of this

          15    initiative.  And AdvaMed in the industry are

          16    willing to work with FDA to promote it and to

          17    educate.  Thank you.

          18              MR. CAMPBELL:  Thanks Janet.  I'd

          19    like the audience to give a little round of

          20    applause for the panelists in view of the

          21    great job.

          22                   (Applause)




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            196


           1              MR. CAMPBELL:  I know the panelists

           2    probably have responses to each other, but

           3    before they jump on top of that, I'll give

           4    them a couple of minutes.  I'd like those of

           5    you in the audience who have questions, or

           6    want to talk about things that we've begun to

           7    talk about this morning, start to get

           8    yourselves ready, go the microphones, line

           9    up.  We have 45, 50 minutes to talk.

          10              One more point.  The primary part

          11    of this panel is about planning

          12    considerations, when to choose a Bayesian

          13    versus a frequentist approach, what to

          14    consider, how to approach the FDA?  When --

          15    so more of the technical statistical detail

          16    some of which came up this morning.  There

          17    will be a following panel for which some of

          18    those comments maybe more appropriate.  I'm

          19    not trying to dissuade anyone from making a

          20    comment, just trying to tell you the focus of

          21    the two panels.  So while people are not

          22    coming to the microphones quite yet, I'll let




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            197


           1    Susan climb in and people collect.

           2              MS. ALPERT:  He knew that.  I just

           3    wanted to comment on -- Bailey talked about

           4    Medtronic Sofamor Danek's use of Bayesian in

           5    their great success.  We are a large company,

           6    35,000 almost 36,000 people worldwide.  We

           7    have 10, is that the right number, Lisa,

           8    clinical groups both in the -- we have groups

           9    in the U.S. and outside.  Our experience

          10    hasn't all been successful.

          11              We've had experience with

          12    cardiovascular devices where the consultant

          13    and the statisticians in house and the team,

          14    and everyone agrees on the trial.  And FDA,

          15    in the end, couldn't agree to have a Bayesian

          16    approach to the trial.  So it's not always --

          17    it's not going to always work.

          18              Now, this early days and there is

          19    not a lot of experience, but just so you know

          20    that we've had a lot of positive experience.

          21    We also have had experience of doing the work

          22    and not winding up with a Bayesian approach




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            198


           1    to a trial.  So the selection issue is really

           2    good.

           3              Two more -- couple of more comments

           4    that came up, one is the issue around

           5    labeling.  We need some examples of what

           6    labeling would be because it's hard even for

           7    me to figure out how the labeling would look

           8    really different based on a Bayesian approach

           9    because it's about safety, and effectiveness,

          10    and device performance.  It's -- so it's kind

          11    of what labeling would change, how do we talk

          12    to our customers and the users and patients

          13    about it.

          14              We need some guidance on adaptive

          15    trial design.  I know it's early, but I think

          16    that's a separate issue.  But since it's come

          17    up in the guidance a couple of times, I think

          18    we need to do that.  One last comment.  Telba

          19    implied this morning in her presentation that

          20    the driver in planning for Bayesian is

          21    safety.  And if the safety -- if the risks

          22    are high, the trial -- the device doesn't




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            199


           1    work.

           2              I'm going to put a question on the

           3    table, not always true.  We deal at Medtronic

           4    in high-risk devices and high-risk patients.

           5    We have many devices where the benefits are

           6    high and so are the risks.  And a high level

           7    of risk doesn't mean that device is not

           8    successful.  So I think -- I just didn't want

           9    to have everybody to walk away thinking that

          10    if you have a lot of risk in Bayesian

          11    approach, your device can't go to market.

          12    It's just that there's more to it than that.

          13    Last comment, train the panels, please.  It's

          14    part of the planning because we ought to be

          15    going to panel.  We can go to closed panel

          16    meetings and get them to agree to and

          17    understand the Bayesian approach if it's

          18    really innovative.  Let's use it because

          19    that's the way to avoid those last minute

          20    things both for the Agency and the companies.

          21              MR. CAMPBELL:  Great.  Don't forget

          22    to identify yourself please.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            200


           1              MR. TRESS:  Yeah, Mark Von Tress

           2    from Alcon Laboratories.  I run a lot of

           3    large clinical studies and they'll have 20 or

           4    30 end points in them that we have to

           5    examine.  Many of them are very expensive,

           6    and the results converge early, and we're

           7    just waiting on final safety data.  Is there

           8    any thought among the panelists about doing

           9    Bayesian sub-study, so that we can stop

          10    collecting the expensive data?

          11              MR. BERRY:  That's a great idea.

          12    And so it's an interim.  You -- when you get

          13    the answer to the one question, you continue

          14    to accrue perhaps in a less controlled

          15    fashion for the safety aspect.

          16              MR. TRESS:  Yeah, that's what we'd

          17    like to do.  And we have some expensive tests

          18    that we like to start.

          19              MR. BERRY:  So propose it.

          20              MR. TRESS:  Okay --

          21              MR. LIPSCOMB:  How would you set

          22    your sample size on the front end if do that?




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            201


           1    Do you just have a range of sample size?

           2              MR. BERRY:  Well, you have a sample

           3    size for the one end point and continue until

           4    you satisfy that.

           5              MR. LIPSCOMB:  Yeah, each one,

           6    yeah.

           7              MR. BERRY:  And the other could be

           8    say adaptive -- I think you maybe looking for

           9    a very rare -- excluding a very rare event

          10    and need, say, 2000 patients or something,

          11    but draw a conclusion about efficacy really

          12    quite early on.  You might continue to follow

          13    some to get the efficacy things, but make it

          14    a lot less expensive and propose that to the

          15    FDA and I'm sure they'll consider it.

          16              MR. TRESS:  Yeah, typically the

          17    sample size that we've got to reach for

          18    safety is 300.  And our visual acuity results

          19    converge, you know, like in 50 patients.  So

          20    -- and thank you.

          21              MR. CAMPBELL:  It's a great

          22    comment, thank you.  Steve.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            202


           1              MR. GOODMAN:  I -- Steve Goodman

           2    from Hopkins.  Hopefully, what I say now

           3    won't steal from my five minutes on the next

           4    panel to extend my time.

           5              MS. ALPERT:  Take advantage.

           6              MR. GOODMAN:  I wanted to just make

           7    a comment about this issue of, you know,

           8    choosing between the various approaches.

           9    There is almost no, you know, good sensible

          10    frequentist design that couldn't be mimicked

          11    with the Bayesian design.  And when it can't

          12    be, usually it's pretty silly design, that is

          13    the Bayesian analysis shows why it doesn't

          14    make that much sense.  So what's different

          15    about the Bayesian design in those cases -- I

          16    mean, there are many cases where you really

          17    can't mimic the Bayesian design, but in the

          18    simple cases.

          19              What's different, and this has been

          20    said many times, is the Bayesian design

          21    forces explicitness about things that were

          22    often uncomfortable putting in the table.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            203


           1    And I have absolutely been in situations

           2    where I propose a really what was a standard

           3    frequentist design.  It was the -- all the

           4    parameters, the sample size was set with

           5    just, you know, the five-minute calculations,

           6    and then in a sense put it forward as a

           7    Bayesian design with expressing what the

           8    implicit priors and -- were, and other

           9    parameters.  And that was not accepted.  In

          10    other words, they weren't accepting the

          11    implicit assumptions of the frequentist

          12    design.

          13              So I think it's very, very

          14    important that we not talk about the two

          15    procedures as one which uses a lot of, you

          16    know, the sort of cartoon version, one which

          17    uses prior information in a particular way

          18    that's particularly suspect whatever,

          19    whatever.  Both methodologies have

          20    assumptions built in.  They have assumptions

          21    about data that you haven't seen about the

          22    efficacy of -- the presumed efficacy of the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            204


           1    device that you're looking at.  And one put

           2    them on the table and the other one doesn't.

           3              And we don't know collectively how

           4    to talk about many of the things that are put

           5    on the table.  And I think that's the

           6    training that is talked about.  We have to

           7    even things as -- I mean, in a frequentist

           8    domain, and we don't even know how to talk

           9    about the "difference of interest".

          10    Everybody knows the trial has to have at

          11    least 80 percent power.  But all trials have

          12    80 percent power, literally every single one.

          13    It's just the question for what.

          14              And we don't know how to talk about

          15    that thing.  We have agreed.  There is a

          16    mutual conspiracy of silence in the

          17    frequentist design that we won't really talk

          18    about that.  And the Bayesian designs don't

          19    really let us get away with that so much, and

          20    then we're forced to talk about things we

          21    really don't want to talk about.

          22              So it's important to frame things




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            205


           1    correctly and not sort of tar the Bayesian

           2    approach with properties that the other

           3    approach has alleged not to have and

           4    understand what we have to learn about.  And

           5    one last thing, I think that the FDA, it was

           6    mentioned that some of the -- that the panels

           7    have to be taught.  I think maybe you should

           8    try to figure out because of the worked

           9    examples here are going to be so critical,

          10    the FDA should try to figure out some way to

          11    keep a library of the designs that it accepts

          12    even if it has to anonymize the device or the

          13    company, I don't know what the legal issues

          14    are going to be.  But you can provide

          15    different frameworks for how things are done,

          16    and right now, it's sort of passed on.  It's

          17    an oral tradition and then Dr. Lipscomb

          18    presents the design that he did, and there

          19    is, you know, rumors about what Don Berry

          20    did.  But -- and some things are actually

          21    published.

          22              SPEAKER:  You didn't mention you.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            206


           1              MR. GOODMAN:  And some things are

           2    actually published, but if the FDA from now,

           3    from both archivally and now going forward,

           4    could figure out a way to keep in a sense a

           5    library, a teaching library of examples.

           6    That would be the most powerful -- more

           7    powerful than this guidance document, because

           8    that will fill in all the blanks here.  And

           9    that's really where we're going to learn as

          10    we go forward, because the answer to many

          11    things with the -- from the speakers was

          12    while we were going to work that on a

          13    case-by-case basis.  So we are going to have

          14    to see the case-by-case basis for those

          15    decisions.

          16              MR. CAMPBELL:  Thank you.  I

          17    hesitate to ask for comments, but I will

          18    anyway.

          19              MS. BUCH:  Could I just address

          20    that real quick?

          21              MR. CAMPBELL:  Please.

          22              MS. BUCH:  What you suggest is a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            207


           1    great idea.  We are bound by certain

           2    confidentiality issues.  Some of that does go

           3    on internally.  As far as training panel

           4    members that may bend the bounds of our

           5    ability legally to do that.  But it's good

           6    idea, and there may be a way to develop

           7    something that is similar, but you need a --

           8    we need permission from the companies who

           9    develop these things, and it's -- a lot of it

          10    is part of their proprietary information or

          11    their trials.  If it's published, it's one

          12    thing.  But anything that's not available to

          13    everybody is not something that we can offer

          14    to everybody.  So there's a little limit on

          15    it.

          16              MR. CAMPBELL:  Good.

          17              MR. LIPSCOMB:  I think there is one

          18    comment about the panel training, and I don't

          19    know the legal ramifications in what you can

          20    and can't do, but I think it's really

          21    incumbent on the FDA, when they are

          22    assembling a panel and a sponsor of a PMA




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            208


           1    comes in, and they are using Bayesian

           2    statistics that hopefully that the

           3    statistician on the panel is a Bayesian

           4    person --

           5              MS. BUCH:  That we do.

           6              MR. LIPSCOMB:  Not a frequentist or

           7    vice-versa, because you don't want to get

           8    your PMA shut down, just because the guy

           9    don't agree with how you did your statistics.

          10              MS. BUCH:  I agree, Bailey.

          11              MR. CAMPBELL:  Susan, at you.

          12              MS. ALPERT:  Yeah, two comments.

          13    One is, I think, the issues -- some of the

          14    issues that we -- that we are coping with in

          15    this planning process have to do with the

          16    fact that those of us who are in the planning

          17    part are in companies, and one of the things

          18    that companies want to know, that management

          19    wants to know is how big is this trial, how

          20    long is it going to last, and what is it

          21    going to cost?

          22              And it's wonderful to talk about




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            209


           1    designs where you -- where you don't

           2    necessarily know that upfront and it grows,

           3    but it's also very, very difficult for people

           4    in the clinical group who want to do that to

           5    convince their management that it is an

           6    appropriate business decision to invest

           7    without really knowing exactly what's going

           8    to happen.

           9              And I think we have to, in this

          10    transition period, deal with that, and some

          11    of it will be outcomes driven, some of it

          12    will be using the SSEDs.  But we have to

          13    understand that there is a driver out there

          14    that is going to impact, because clinical

          15    trials are the most expensive thing we do in

          16    the development of most medical devices.  It

          17    overwhelms all of the rest of the development

          18    when you have something really novel that

          19    needs -- that really needs a good clinical

          20    trial with real data to demonstrate safety

          21    and effectiveness.  So I think we have to --

          22    we have to keep that in mind.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            210


           1              One comment to the issues on

           2    panels; panel members are, in fact, special

           3    government employees.  They're bound by

           4    confidentiality.  The panels can be trained

           5    in private issues.  They are allowed to have

           6    access to proprietary information.  They

           7    don't even have to know which company it is,

           8    as long as they know not to talk about the

           9    details, but -- so if there is an opportunity

          10    to do that -- to do the private training of

          11    the panel members, it might also be worth

          12    having some discussion at all the panel

          13    meetings about statistical approaches in a

          14    public way, so that -- or when they come up,

          15    so that people understand that these aren't

          16    going to look different.

          17              MS. HOGAN:  Janice Hogan from Hogan

          18    & Hartson.  I have two related questions.

          19    When clients come to us and they ask us,

          20    "Would a Bayesian design be good for us?"

          21    There are two things that have been surfacing

          22    frequently.  One is the special case of a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            211


           1    combination product, and there is a

           2    perception, I think, in industry that may be

           3    CDRH, to its credit, is ahead of some of the

           4    other centers.

           5              I'd just like to hear sort of

           6    practical experiences from the panel on

           7    whether there are special issues to be taken

           8    into consideration, if you are thinking about

           9    using Bayesian on a combination product since

          10    we don't always know from the beginning to

          11    what extent multiple centers will ultimately

          12    be collaborating on the substantive review.

          13              And the second is another -- just

          14    pragmatic reactions.  Companies often say to

          15    us, "Bayesian statistics sounds great, we'd

          16    love to do it.  We are a small company.  We

          17    are not Medtronic or Avid or Guidant or any

          18    of these places.  We can't hire a dedicated

          19    Bayesian statistician.  Can we pull this off,

          20    and to what extent does this put additional

          21    pressure on the execution of the study?"

          22              MR. CAMPBELL:  Let me give you a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            212


           1    couple of responses.  First of all, we would

           2    like to think of ourselves in the Center for

           3    Devices, as ahead of all the other centers,

           4    always.

           5                   (Laughter)

           6              SPEAKER:  Yeah.

           7              MR. CAMPBELL:  And those of you in

           8    Ceeder (?), I didn't mean to say that.

           9                   (Laughter)

          10              MR. CAMPBELL:  I think we have more

          11    -- I think we have lot of experience in this.

          12    I think the question of combination products

          13    is an excellent question.  And I don't think

          14    that we've wrestled with too many of those

          15    situations quite yet and we're developing it.

          16    What's really outstanding is that we do have

          17    an Office of Combination Products, if there

          18    is a lot of coordination.  So again one of

          19    the things that we've emphasized and our

          20    sister agencies emphasize this, sister

          21    centers, the same we do, come to us early and

          22    often.  So the earlier you are thinking --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            213


           1    this may be a possibility with a combination

           2    product.  Let's make sure we identify the

           3    review team, and see if there is agreement

           4    about how to proceed.

           5              And, frankly, that's going to be

           6    the case with a combination product, not only

           7    with the statistical design, but with

           8    manufacturing, because the way we approach

           9    things like GMPs are different.  We even have

          10    different approaches to laboratory issues

          11    among the centers.  So I don't think there is

          12    either in CBER or CDER a prohibition against

          13    this kind of approach, but I do think in the

          14    case of a possible Bayesian design, you would

          15    like to get on the early end.

          16              Now, combination products, as Greg

          17    mentioned earlier, will tend to be looked at

          18    differently, because by and large they are

          19    very novel.  So there is extra scrutiny.  I

          20    don't think that it makes it worse for a

          21    Bayesian than a frequentist analysis.  It may

          22    make it harder for a company that may be




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            214


           1    uncomfortable, but I think that we'd be

           2    welcoming to try and look at some of that.  I

           3    think Greg would be eager to try and explore

           4    that with you.  In terms of small companies,

           5    I think, I will go back to what Don says, or

           6    maybe he'd like to make some more comments

           7    about that, because I think it is a challenge

           8    for a small company, but I think a lot of

           9    stuff at the Agency asks can be a challenge.

          10    Don?

          11              MR. BERRY:  No, it is a challenge.

          12    And even if they did want to hire a Bayesian

          13    or to get access to a Bayesian, there are not

          14    many who are adept to doing this sort of

          15    thing.  And with respect to your other

          16    question, the combo question, I just want to

          17    point out that it is something that's

          18    enormously important in drugs.  And we don't

          19    know how to do it.

          20              I do think that the Bayesian

          21    approach through its modeling has an inside

          22    track, but we are not there yet.  And we've




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            215


           1    got to get them.  I think, oncology, you

           2    know, where I live, it's -- everything is

           3    combination products, and we don't really

           4    understand.  When you take this three-drug

           5    regimen, we don't understand the contribution

           6    of the various drugs.  They are just kind of

           7    tradition has developed into this setting.

           8    We know how to add drugs to something, we

           9    don't know how to subtract.

          10              MS. ALPERT:  A comment.  It was

          11    very important, I think, that Dr. Patricia

          12    Love from the Office of Combination Products

          13    was here this morning.  I don't think she is

          14    here anymore.  But they clearly have an

          15    interest, and I think, to your point and to

          16    the point that we made with one of the

          17    previous speakers is the issues around when

          18    do you need more patients for safety, because

          19    frequently in combinations it's because there

          20    is a question about the safety of a biologic

          21    or pharmaceutic being used in a way that's

          22    novel on a device that's already been




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            216


           1    established.  And it would seem to me that

           2    this is exactly the kind of situation in

           3    which Bayesian approaches are to be used.

           4    And so, you know, I think -- I think there is

           5    a lot of opportunity there.

           6              MS. BUCH:  Can I make a comment?

           7    The answer to your question is yes.  CDRH

           8    does entertain those.  In fact, there is a

           9    PMA that was approved.  There are multiple

          10    ways to get around.  Both of the comments

          11    that have just been made are true.  They do

          12    provide a challenge, because an unknown

          13    biologic, for example, requires us to look at

          14    things a little bit more extensively.  One

          15    other ways that CDRH is using that a little

          16    bit more innovatively is to look at

          17    post-approval studies to answer some of the

          18    questions that may not have been answered in

          19    the initial trial.

          20              And those kinds of things come up

          21    with rare events that we follow, and if you

          22    can get to a point where you feel that the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            217


           1    efficacy and the safety, in general, have

           2    been addressed, some of the other questions

           3    that come up along the way, we have gone to

           4    post approval.  So there is actually a

           5    precedent for a combination product being

           6    approved using a Bayesian approach.

           7              MR. CAMPBELL:  Are we done?

           8              MS. HOGAN:  Yes, thank you.

           9              MR. LU:  I'm Michael Lu from

          10    Edwards Lifescience.  I really recommend that

          11    these guidelines, because it's not only

          12    trying to describe the regulation, also to

          13    provide education tool for the people, for

          14    statistician and the non-statistician who

          15    doesn't familiar with Bayesian statistics.

          16    In the reality, we know the frequentist, the

          17    statistimist where established, which, you

          18    know, are not common terminology I'm familiar

          19    with clinician and with medical communities.

          20    So I hope we can expand it a more, a kind of

          21    a comparison, the different

          22    statistimistology.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            218


           1              In Section 4.8 of page 14, we did

           2    have a section to describe how the Bayesian

           3    and the frequentist, the approach would

           4    differ.  It mentions, give a brief

           5    description -- I hope the guidance go a

           6    little further in terms of try to address

           7    those basic frequentist, the concept that

           8    like a p-value, statistical power and some

           9    other common concept, how that's a Bayesian

          10    statistics that cover those concept.  It may

          11    not be one-to-one match, but at least in some

          12    way, it would be addressed -- it will be

          13    covered so that -- to the customer or to the

          14    clinician.  They will feel, you know, safe

          15    about what our traditional clinical trial

          16    does will be also achieved by the Bayesian

          17    trial.

          18              MR. BERRY:  Larry, can I address

          19    the trial?

          20              MR. CAMPBELL:  Yeah.

          21              MR. BERRY:  So if this -- if this

          22    were a document that said everything that was




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            219


           1    to say about the Bayesian approach, you

           2    wouldn't be able to carry it.  It can't do

           3    that.  It has to address the things that are

           4    specific to devices and the issue of what is

           5    it -- what is a p-value?  What is a p-value

           6    from the perspective of a Bayesian?  Are

           7    there analogies?  What are the different

           8    measures of inferential evidence?  Those they

           9    can touch on those, but they can't explain

          10    them -- I mean, they can't cite all the

          11    literature, and give all of the examples, et

          12    cetera, that's what the references that they

          13    give are supposed to do.

          14              And what you should check is, go to

          15    those references, see if it answers your

          16    questions, and if it doesn't make a comment

          17    to them saying, "I can't find anywhere where

          18    it says what a p-value is," or "I can't find

          19    what a Bayesian hypothesis test is in any of

          20    the references that you refer to."  So that

          21    -- I think you can charge them with doing

          22    that, but you can't expect that they are




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            220


           1    going to talk about the philosophy and the

           2    differences between the two approaches.

           3              MR. LU:  I have another suggestion.

           4              MR. CAMPBELL:  Go ahead.

           5              MR. LU:  On the same aside, a

           6    section, which is the section, a 5.7, the --

           7    which has some verbiage that suggests

           8    specific particular criterion to stop the

           9    trial and a proper stopping criteria maybe

          10    based on specific amount of information about

          11    the permit.  I hope the guideline can give a

          12    little more specific or let them expand a

          13    little more on that.  Because the way people

          14    that are used to the frequentist, we know

          15    that there is a combination, Type-I errors

          16    control, the 0.05 level, and that the power

          17    where they reached it by either 80 percent or

          18    90 percent.  And that become a -- there is a

          19    quantitative commission.

          20              Here we say the -- a sufficient

          21    high probability for pre-specified high

          22    possessives, which, you know, how high it




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            221


           1    would be sufficient and high, of those I

           2    don't know if it's appropriate to addressing

           3    the -- if it kind of addressed in the

           4    guideline, which it would be very helpful,

           5    you know, for a company to start, to design

           6    that.  Otherwise, just to try to figure out

           7    what -- how high is a sufficient high, would

           8    be a lot challenge discussing.

           9              MR. BERRY:  So this morning several

          10    people talked about operating

          11    characteristics.  They said that some of

          12    these Bayesian designs are very complicated.

          13    If you imagine, for example, that you are

          14    running a trial, where you are looking, let's

          15    say, every month or periodically, and you're

          16    saying depending on what the data are, "I'm

          17    going to predict what the future is, and I'm

          18    going to base my continuation of the trial

          19    and possible modifications of the trial on

          20    that calculation, and then I'm going to go

          21    get more data and do the same thing, and the

          22    same thing, and the same thing."  That's




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            222


           1    going to be very tedious.

           2              Now, factor in that we have to

           3    figure out, this is a prospectively-defined

           4    study.  We have to figure out what is the

           5    probability that you stop the trial in a

           6    positive claim, when in fact nothing is going

           7    on that there is no benefit or that we are

           8    going to stop the trial on a positive claim

           9    and there is a benefit, you know, calculating

          10    power and Type-I error.  We have to be able

          11    to say what that is.

          12              To do that, it means we have to

          13    usually simulate.  And the simulations, each

          14    simulation has to be this complicated thing

          15    of stopping for futility, stopping for

          16    positivity.  You can't do it analytically,

          17    and you can't -- there may be some sort of

          18    standard design that we have written down

          19    that you could do it for.  You could do it

          20    once numerically and then forever have it.

          21    But more generally people design specific

          22    trials.  And so you have to do this.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            223


           1              And we can't say in advance what it

           2    is.  You have to do it, sometimes by trial

           3    and error.  So here is what we do.  We say,

           4    "Let's suppose, we stop at a basis of greater

           5    than a 97 percent probability of this.  Let's

           6    see how that works," and Scott runs 10,000,

           7    you know, virtual trials and he sees that the

           8    false positive rate is not what the FDA

           9    wants.  So he goes back and says, "Well,

          10    97.3."  We can't -- you can't have a formula

          11    that's going to do that.  It has to be done

          12    by simulation.  So I can't answer your

          13    question as, you know, this is the standard,

          14    but this is a method for doing it.

          15              MR. LU:  Right.  I understand.  You

          16    are talking about the approach to really

          17    evaluate the power.  But try to get an

          18    agreement with the Agency on, you know, how

          19    -- what we define as a success.  There is

          20    some criteria -- there is -- there have to

          21    have some base, so we can't have to stop.

          22              MR. BERRY:  I was taking that for




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            224


           1    granted that you've touched that base.  It's

           2    a critical base.  You have to make a

           3    decision.  You have to get an agreement with

           4    the Agency as to what is success, and is

           5    success going to be claimed on the basis of

           6    such and such, and then you can built that in

           7    to your model.  When you are all done, you go

           8    to the Agency and you say, "This is -- this

           9    is our design, and it has a 97.3 cut off."

          10    They take your program or write their own

          11    program or do WinBUGS if it's possible,

          12    sometimes it isn't, to see if they get the

          13    same result and they buy in or not to what

          14    the overall conclusion is.  But the endpoints

          15    and the success criteria are set in advance.

          16              MR. CAMPBELL:  And that's no

          17    different in terms of setting endpoints and

          18    criteria, an agreement from what we would

          19    need to do generally in the frequency zone,

          20    another thing that Susan mentioned earlier,

          21    "Come to the agreement table," because it

          22    really structures a lot of what you were




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            225


           1    doing in the design and analysis.

           2              MR. BERRY:  Right.

           3              MR. CAMPBELL:  Thank you.

           4              MS. BUCH:  And it's not one answer

           5    for everybody.

           6              MR. JACQUES:  Louis Jacques,

           7    Division of Items and Devices in the Centers

           8    for Medicare and Medicaid Services.  If I

           9    could borrow some of your language for a bit

          10    with tongue slightly in cheek, I might

          11    suggest that the ultimate outcome may well be

          12    coverage and payment by various payers and

          13    that regulatory approval is a sometimes

          14    imperfect surrogate for some of that.

          15              As most payers, we tend to look not

          16    just at one trial, but we look across a wide

          17    body of evidence that if this comes to

          18    fruition, is going to include both

          19    frequentists as well as Bayesian types of

          20    trials.  And one other things that I would

          21    suggest, some attention be paid to is when

          22    one is looking across trial paradigms, what




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            226


           1    is the most appropriate way of sort of

           2    smushing all of that into one answer.

           3              Now, if history, or maybe future,

           4    which will retrospectively consider history,

           5    suggests that these things really do answer

           6    the same questions in somewhat the same way,

           7    but more efficiently, there is a certain

           8    comfort there.  If one finds that, "Gee, all

           9    these existing frequentist data is saying

          10    this", and "Hey, there is this kind of new

          11    way of doing things, and it is sort of saying

          12    something else."  I think there is going to

          13    be maybe some convincing to do of the payers

          14    that we are indeed able to compare apples to

          15    apples and not sort of apples to watermelons

          16    or something else.  So I only sort of throw

          17    that out as you think about this, think about

          18    how one would compare these things, because

          19    we are generally looking at more than one

          20    trial.

          21              MR. BERRY:  And the same is true in

          22    devices in setting up a prior distribution




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            227


           1    perhaps hierarchically.  One looks across

           2    trials, and that's really the whole point of

           3    the Bayesian approach is to say, on the basis

           4    of everything I know, such and such.  And now

           5    I have to convince you that what I know is in

           6    fact, what you should know.  And so the

           7    Bayesian approach is one of a synthesis and

           8    it's probably the most important thing that

           9    sets us apart -- sets it apart from the

          10    frequentist approach, that's specific to a

          11    particular setting, a particular trial.

          12              And one answer, you say you want to

          13    get one answer, an approach to that the

          14    Bayesians have to offer, as Gene Pennello

          15    talked about this morning and Telba discussed

          16    it to some extent, this hierarchical modeling

          17    with the levels of experimental units, the

          18    one being the trial, and something of concern

          19    to you and something of concern to the Agency

          20    would be the heterogeneity in that population

          21    of trials.

          22              And you might -- the setting the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            228


           1    prior distribution becomes a prior

           2    distribution on that pop -- super population,

           3    which is the critical thing.  It's not so

           4    much the individual parameters, but what you

           5    said about heterogeneity and that super

           6    population.  But it is something that is

           7    really the Bayesian approach is ideally

           8    suited for.  It doesn't mean that the

           9    frequentist can do this, they can.  It's just

          10    that it's natural, and it is something that

          11    the Bayesian tools and computational tools

          12    are really ideally suited for.

          13              MS. ALPERT:  It reminds me that one

          14    of the things that need to be expanded in the

          15    guidance is the issues around how you manage

          16    the differences between populations when you

          17    are using a variety of trials as your prior

          18    or as Don is pointing out in combining trials

          19    for something like CMS.  Because I think that

          20    wasn't as clear for the non-statistician as

          21    it could be in terms of both how you use

          22    trials, what you know and don't know about




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            229


           1    them, and then how do you manage differences,

           2    if you've got trials taking place in

           3    different populations, because it also speaks

           4    to the issue around pre-market trials and

           5    post-market trials, which is something that

           6    CMS is getting into.

           7              And the patients are not the same,

           8    because the construct for pre-market trials

           9    is frequently different than the construct

          10    for post market trials for very valid

          11    reasons.  And anticipating that you're going

          12    to have the same performance in, for example,

          13    a Medicare population broadly as you get in a

          14    broader population with younger people with

          15    the same device, it's going to be different,

          16    because the populations have different

          17    variables in them and different co-factors

          18    and co-morbidities.  We've got -- we've got

          19    to figure out how to do that better, but I

          20    think that some explanation of the ability of

          21    Bayesian statistics to measure and capture

          22    that is important.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            230


           1              MR. CAMPBELL:  I want to expand

           2    that on a second.  So in a sense the

           3    advantage of a Bayesian approach is to allow

           4    you to try and incorporate systematically

           5    different studies with different population,

           6    different characteristics.  And the

           7    disadvantage of the Bayesian approach is it

           8    allows you to incorporate different

           9    population with different studies, different

          10    characteristics, and there is going to be a

          11    degree of clinical, epidemiologic,

          12    statistical judgment that's needed to be

          13    brought to a table in order that we can come

          14    to a consensus about what this looks like and

          15    why, and how we should go forward.

          16              And I think in the reimbursement

          17    context, you deal with that all the time as

          18    well.  You've got two studies from Israel,

          19    one from China, you got a registry from

          20    United States, they all look promising,

          21    they've got some differences.  How do you put

          22    it all together?




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            231


           1              And the advantage is that the

           2    Bayesian approach gives you some structure

           3    for talking about some of these explicitly.

           4    I think, Steve Goodwin mentioned that here's

           5    a vehicle for talking about some things that

           6    the conspiracy of silence among the

           7    frequentists, let's sit on the side,

           8    sometimes, and here we are forced to deal

           9    with it.  I think it's a shared problem, and

          10    I like the fact of thinking of FDA as a

          11    surrogate for CMS.  That's -- could you fund

          12    our budget now?

          13              MR. JACQUES:  Interagency rivalry.

          14              MS. BUCH:  Exactly.  Can I say out

          15    something?

          16              MR. CAMPBELL:  Please.

          17              MS. BUCH:  I think one of the

          18    things that this meeting really brings out is

          19    that we probably have to change our framework

          20    of thinking a little bit, and we have to take

          21    a step back from the statistics and

          22    understand what they're actually saying and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            232


           1    meaning.  And so what you said is absolutely

           2    correct.  You can get different outcomes from

           3    the same trial using different manipulations

           4    of the data, and what you have to do at the

           5    end of the day is understand what each

           6    dataset means.

           7              And so that's what I'm, you know,

           8    putting out to everybody that we have to

           9    really think about that, take the statistics

          10    as a part of the equation to look at the

          11    whole picture.

          12              MR. MUGGLIN:  I also want to echo a

          13    point made by Susan, by Dr. Alpert --

          14              MR. CAMPBELL:  Name.

          15              MR. MUGGLIN:  I'm sorry, Andy

          16    Mugglin, University of Minnesota.

          17              MR. CAMPBELL:  Thank you.

          18              MR. MUGGLIN:  Having been in the

          19    industry a couple of years, five years

          20    actually, and worked on a few Bayesian trials

          21    with FDA, I learned the hard way that the

          22    practical implications of implementing a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            233


           1    Bayesian trial are much, much harder, and of

           2    greater scope than the technical ones, the

           3    statistical ones.  And I just want to kind of

           4    reiterate that point.  It rang in me like a

           5    bell, and I even tried to quantify some of

           6    those hours on a couple of occasions, and

           7    discovered it -- at least from my experience,

           8    it was more than a factor of 10 or something

           9    like that of the man-hours that we put into

          10    these particular trials that we were looking

          11    at.

          12              A lot of it is education, education

          13    of executives, education of regulatory

          14    personnel within the company, scientists, and

          15    so forth, with very little notification.

          16    Somebody says "What is the Bayesian method,

          17    why should we do it?" or, "I hear we should

          18    be Bayesian because it shrinks sample size."

          19    It is enormous amount of work.

          20              It happens on the front-end

          21    especially, but it also happens on the back

          22    end as you run these analyses, and you cannot




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            234


           1    just hit a button.  I mean, we all -- you

           2    know, we all, as statisticians, resent that

           3    term anyway, but I think there's more

           4    back-end analysis of time for the MCMC and

           5    model fit, and so forth, and this is one of

           6    the cause.  And I think the document, it

           7    would be well to play it up although it is in

           8    there.

           9              Related to that is the idea of when

          10    should one be Bayesian, and perhaps this

          11    doesn't go in the document, but -- I mean,

          12    the shorter answer is when the costs are less

          13    than the benefits, but coming to terms what

          14    some of those are, I think, is a useful

          15    thing.

          16              I have a question and perhaps it

          17    goes outside the scopes of

          18    "feel-free-to-sidestep-this," but it has to

          19    do more with the publication later, and

          20    whether -- to what extent do you stick with

          21    the Bayesian paradigm.  And again this is a

          22    practical question here.  When you're




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            235


           1    publishing results of a trial, it seems to me

           2    that if you're dealing with the primary

           3    objectives and so forth you should sick with

           4    the paradigm of the study.

           5              But sometimes there are extra

           6    analyses that people want to throw in, or

           7    even secondary publications and there's

           8    nothing about a dataset that is inherently

           9    Bayesian or frequentist.  And I don't know

          10    that there's a consensus in the community

          11    about what paradigm to publish in, and so I'm

          12    curious as to reactions from the panel.

          13              MR. CAMPBELL:  And you're thinking

          14    about publishing --

          15              MR. MUGGLINS:  Medical journals for

          16    instance, yes.

          17              MR. CAMPBELL:  -- in a medical

          18    journal what would you present?

          19              MR. MUGGLIN:  Yeah.

          20              MR. CAMPBELL:  Which is separate

          21    from -- just to make sure, to clarify for the

          22    audience that when we, in the Center for




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            236


           1    Devices, and you and the company start down

           2    the Bayesian road with a trial, we expect

           3    that to be start-to-finish, and not at the

           4    middle for you to say, "Oh, gee, I just

           5    really want to change my direction."  That's

           6    what I talked about.  Talking about the

           7    studies finished -- right.

           8              MR. MUGGLIN:  Correct, no, right,

           9    correct.  It is a Bayesian trial from start

          10    to finish, but there will be perhaps 10

          11    papers out of it at the end.  Should they all

          12    be Bayesian?

          13              MR. BERRY:  Well, as you know,

          14    Andy, it's very difficult to do a frequentist

          15    analysis of something that wasn't designed

          16    consistent with a frequentist approach.  So

          17    it's easy to do Bayesian things and medical

          18    journals, maybe 20 years ago, would have

          19    said, "What?  Bayesian?"  But now New England

          20    Journal, JAMA publishes Bayesian approaches,

          21    and so the thing that I would expect is that

          22    you would continue and do a Bayesian




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            237


           1    publication.  If you tried to do p-values and

           2    the like, you get -- and you get a certain

           3    type of statistical review or those say how

           4    can you do this, it's not consistent with the

           5    design.

           6              MR. CAMPBELL:  Other comment?

           7              MR. MUGGLIN:  Thanks.

           8              MS. ZHANG:  I'm Xiaozheng Zhang

           9    from Edwards Lifesciences.  I actually pretty

          10    appreciate the guideline itself.  It's easy

          11    to understand, even not statistician.  And

          12    morning's it's been, you know, further --

          13    presentation really helped further understand

          14    what does Bayesian mean, and what we're going

          15    to involve, if we want to practice that.

          16              And in the guideline, very front

          17    page, you talk about a guideline's guideline,

          18    not necessarily requirement, however,

          19    recommendations, suggestions, and which

          20    clarifies some of the point, I think, really

          21    going to help industry to practice and take

          22    the time to get more experience.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            238


           1              However, I want to further point

           2    out some of the details in the guidelines.

           3    We may want to look into it.  If we get into

           4    the detail, make sure we have explanation,

           5    otherwise, rather, you know, not have the

           6    detail.  The reason, when industry started

           7    practice, they see guideline as guideline

           8    even if it says it's not requirement, and I

           9    give example.  Here in the page 17 talk about

          10    if we have a pre-information, we may, you

          11    know, possibly look at the following aspect.

          12    For example similar on the protocol, that's

          13    understandable, and the point, that's

          14    probably okay, and target population, that's

          15    fine.  But I kind of puzzling what size has

          16    to be similar, and physician have to be

          17    similar, and time frame have similar.

          18              For example, at the beginning of

          19    the documentation you talk about, okay, you

          20    take our C-6 (?) experience.  That's

          21    contradict with -- your size has to be

          22    similar.  So I don't know what is behind or




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            239


           1    background of this discussion.

           2              MR. CAMPBELL:  Let me try and

           3    address at least briefly about the issues

           4    where we talked about similarity between

           5    protocols and designs, things like that.

           6    There's always a tension in the FDA about

           7    writing something in guidance that's general

           8    versus writing a great deal of detail, and

           9    being prescriptive or postscriptive.  And

          10    some of it has to do with allowing a wide

          11    variety of different kinds of applications or

          12    different kinds of examples as well as the

          13    legal requirements and fighting we get in

          14    just terms of getting guidance through our

          15    Office of Chief Counsel.

          16              So let me just talk about them very

          17    briefly on both sides.  In the first case,

          18    you have to remember, and a lot of people

          19    forget this, Susan doesn't forget this, the

          20    Center for Devices regulates a variety of

          21    products that by noticeable factor exceeds

          22    the regulatory purview in terms of variety of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            240


           1    other centers.  We regulate Band- Aids, we

           2    regulate MRI machines, we regulate pacemakers

           3    and implantable cardioverter defibrillators,

           4    we regulate catheters, we regulate in vitro

           5    diagnostics, and they're all called

           6    "devices."

           7              So if we tell you that the protocol

           8    has to look like this and it's similar, well

           9    that may work really well for DGR&D General

          10    and Restorative, and the people in in vitro

          11    diagnostics will wonder, what are you talking

          12    about?  This is apples and oranges.

          13              So in some sense for the world of

          14    device, we tend to leave things a little more

          15    generally.  Separately as we start to write

          16    examples down, if we say in the guidance

          17    document these are what two protocols need to

          18    look like to be similar, then folks in the

          19    Chief Counsel will say, are you being

          20    prescriptive, are you telling people what to

          21    do as opposed to the general "how"?

          22              I actually think, we've made it




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            241


           1    several times, more examples would be

           2    helpful, and I think Greg has heard that

           3    message and we'll about that later.  But in

           4    terms of trying to give you a "cookbook" for

           5    protocol similarity, or time frame

           6    similarity, something like that, I think

           7    that, you know, it's hard to go too much

           8    further without starting to limit your

           9    flexibility as a statistician or as a

          10    company.  Bailey first.

          11              MR. LIPSCOMB:  Just a comment along

          12    those lines and on that page, I agree

          13    wholeheartedly with the last couple of

          14    bullets about physicians and going back to

          15    the same investigation centers.  Sometimes

          16    that's just completely impractical to do

          17    that, and I do know though that we've seen

          18    that kind of question before though, and so I

          19    would really think of -- sure, the protocol,

          20    the end points, obviously that has importance

          21    to it, but necessarily going back to the same

          22    surgeons that had the same level of training




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            242


           1    and so on and so forth sounds good, but I

           2    don't think that's really very doable.

           3              MR. BERRY:  This is an argument for

           4    making this a guidance and not a requirement

           5    because I would hate to see something like

           6    this be a requirement.  It's a -- indeed a

           7    fundamental aspect of science is trying to

           8    understand in the context that a device is

           9    going to be used.  And if you say I'm going

          10    back to the same sites and the same

          11    physicians, you'll learn very well about

          12    those sites and physicians, but you learn not

          13    enough, and much less about how it's going to

          14    be used in clinical practice.

          15              There's an argument for saying get

          16    different sites and different physicians.

          17    The notion of target population in some

          18    diseases and conditions, the most important

          19    thing that's changing from one study to

          20    another is the disease.  So in breast cancer,

          21    for example, if you say I'm going to do --

          22    I'm going to look at the same breast cancer




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            243


           1    population today as I did five years ago,

           2    you're kidding yourself.  You can't identify

           3    that population.  It's forever gone, and the

           4    disease has changed -- is changing so rapidly

           5    both in terms of its treatment and detection

           6    that it's a completely different ball game.

           7    So you're kidding yourself.

           8              What you can hope for is that the

           9    device effectiveness is the same now as it

          10    was then in different target populations and

          11    that you model this.  And by the way, an

          12    important aspect of the borrowing that Gene

          13    talked about, you borrow not on the basis of

          14    what the population is looking like and what

          15    the events are looking like in the

          16    population, but on the device effectiveness

          17    in comparison with the control.

          18              So it's essential to have a control

          19    throughout the period.  It doesn't mean it

          20    has to be equal or randomized, but you've got

          21    to have a control throughout the period.

          22              MR. CAMPBELL:  We're going to run




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            244


           1    out of time.  So I just want to make the last

           2    two questions from --

           3              MS. SETHURAMAN:  I think he's

           4    before me --

           5              MR. CAMPBELL:  Oh, you have

           6    follow-up?

           7              MS. ZHANG:  I've a quick comment.

           8              MR. CAMPBELL:  Quick comment.

           9              MS. ZHANG:  Very quick.  Actually,

          10    I totally agree with Gene from AdvaMed and

          11    when she made the comments, you know, in some

          12    point maybe that you need to re-look at all

          13    the requirement and refresh it, and I kind of

          14    jotted down what the requirement when we

          15    started to utilize this method.  And I look

          16    at the Type-I, it's important to power, it's

          17    important in submitting of electronic, you

          18    know, the calculations and panel review, and

          19    sample size, and minimum, maximum, and

          20    simulations, agreement prior to meeting, all

          21    those things, and it probably a kind of very

          22    initiative, and then because we're both new,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            245


           1    we tried to get confident.

           2              I can see the conservative side

           3    from regulatory review.  However, you know, a

           4    step back from the -- industry-wise, you're

           5    probably scared up, you know, kind of

           6    thinking about it, should I practice or get

           7    involved with those details.  So I totally

           8    support Gene from AdvaMed, the suggestion at

           9    some point, we review it, and give some

          10    confidence both from the FDA and the

          11    industry.

          12              MR. CAMPBELL:  Fair enough.

          13              MR. LEWIS:  Roger Lewis,

          14    Harbor-UCLA Medical Center.  I was struck in

          15    reading the document, that there was a nice

          16    emphasis early on, on the process by which

          17    both, the sponsor and the Agency, might agree

          18    on pre-study prior.  And there was a

          19    discussion in the early lectures about the

          20    focus that the Agency has on avoiding,

          21    approving devices that are ineffective, or

          22    failing to approve devices that will be




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            246


           1    effective, so the study-wise error rates if

           2    you will.

           3              So when I got to the simulation

           4    section, what I expected to see was a

           5    requirement that the error rates that the

           6    Agency would be interested in for the study

           7    designs would be conditioned on that prior

           8    that the Agency themselves agreed to early

           9    on.  And so I have two questions.

          10              One is, why didn't I see that, and

          11    the second question I have is that instead I

          12    saw a table where they wanted error rates

          13    conditioned on point estimates for the thetas

          14    and now one in some.  And those things it

          15    never actually said that you'd still be

          16    looking for 0.05 for Type-I error rate, but

          17    by not explicitly saying we are no longer

          18    held to this, it sort of seems like another

          19    one of those conspiracies of silence, sort of

          20    suggest we really are still looking for that

          21    rate.  I was wondering if anybody could

          22    address those two things.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            247


           1              MR. CAMPBELL:  Well, the first

           2    thing I can say is I know it's not Laura's

           3    idea.

           4                   (Laughter)

           5              MR. CAMPBELL:  Comments from the

           6    table?  I might have to turn to Greg on this

           7    one.

           8              MR. LIPSCOMB:  I think you'd turn

           9    to Greg.

          10              MS. ALPERT:  Get it from Greg.

          11              MR. CAMPBELL:  Greg?

          12                   (Laughter)

          13              MR. CAMPBELL:  I actually have a

          14    couple of really good answers, but I'm a

          15    little scared --

          16              MR. LEWIS:  Not ready to say them.

          17              MR. CAMPBELL:  No, I'm not sure I'm

          18    right.

          19              MR. LEWIS:  That's a great stand.

          20              MR. CAMPBELL:  Well, Roger we

          21    talked about this actually before lunch.  I

          22    think those are good points, and there's




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            248


           1    nothing that would prevent a company from

           2    doing that.  It is a bit more work.  It's

           3    perhaps easier to do what we suggest than to

           4    do what you suggest in terms of complexity,

           5    but it's not that much more difficult.

           6              It certainly makes more sense when

           7    we have an agreement with the company about

           8    the prior information.  In some cases, we do

           9    not and so when the submission comes in, we

          10    haven't really thought about whether to agree

          11    or not, and at that point, it's difficult to

          12    figure out how to approach things like the

          13    frequentist Type-I error, and power.

          14              So that's a partial answer, but not

          15    a complete answer, and it's certainly

          16    reasonable for a company to come and suggest

          17    something like that to us.  Remind me what

          18    your second point was?

          19              MR. LEWIS:  Second question was, in

          20    the simulations that are still conditioned on

          21    point assumptions about efficacy --

          22              MR. CAMPBELL:  Yes, yeah.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            249


           1              MR. LEWIS:  And that would seem on

           2    the same rates.

           3              MR. CAMPBELL:  Yeah, it's not an

           4    accident that we didn't say, oh, it has to be

           5    -- you know, the Type-I error has to be 0.05

           6    or less and the power has to be 80 or 90

           7    percent or whatever, and that's because there

           8    are situations where we could imagine that

           9    one might want to have a different target.

          10    So you're quite right, it's not in the

          11    document.  It doesn't say that explicitly,

          12    and I guess that's all I probably want to say

          13    about it.

          14              MR. CAMPBELL:  I want to add one

          15    thing to that.  I think the Center for

          16    Devices prides itself on not depending on the

          17    Holy Grail of p-equals 0.05, and I think for

          18    a variety of reasons, this is not to be

          19    derogatory to anybody else who approaches

          20    things that way for whatever reason they do,

          21    but the Center for Devices approves a variety

          22    of things with a variety of mechanisms, and a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            250


           1    variety of evidence because what we

           2    understand about the way the body works, the

           3    way the products work -- Susan and I have

           4    been through this a long time -- we're not

           5    slavishly devoted to p-equals to 0.05 and

           6    power of 0.80.  It's just not something that

           7    we've been hidebound.

           8              And, in fact, if you look at the

           9    legislative mandate that underlies CDRH, even

          10    Congress, way back when in the mid-'70s

          11    recognized that for devices the wall is a

          12    little bit different, and we tend to approach

          13    it.  So you're not going to see something so

          14    prescriptive and shouldn't.  Fair?

          15              MR. LEWIS:  And what I was hoping

          16    to see was, instead of it being un-addressed,

          17    an explicit statement of that flexibility.

          18              MR. CAMPBELL:  Yeah.  Okay.  Good

          19    point.

          20              MS. ALPERT:  I find myself needing

          21    to make a comment having been a drugs

          22    reviewer in a previous life.  And that is




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            251


           1    that there is a real good reason why those

           2    things work in a drugs environment and don't

           3    work necessarily in a device environment in

           4    the sense of how we design trials, what we

           5    know, I mean actually devices have always

           6    been Bayesian if you really want to think

           7    about it, because they're designed to do a

           8    specific thing.  It's not really trying to

           9    figure out what they do.  We kind of know

          10    what they do, and we're building on the prior

          11    in our knowledge and our expectation for how

          12    well they do it, which is a lot more that's

          13    known than when most pharmaceutical trials

          14    begin.

          15              And the size and expectation, both

          16    for testing and for use are just very

          17    different.  We live in very different worlds,

          18    and so there's a lot of good reasons to

          19    manage differently even in the statistical

          20    world, and that's not to say that Bayesian

          21    isn't a proper approach for a lot of what's

          22    changing in the world particularly of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            252


           1    biologics, of biologic drugs which is a whole

           2    different world than standard

           3    pharmaceuticals, for a lot of the same

           4    reasons about knowing, and having

           5    expectations and understanding a little bit

           6    more about what things do.

           7              So, you know, I think we'll see

           8    that there too, but you know, the worlds have

           9    been different, they're acknowledged to be

          10    different, things will evolve differently in

          11    a drugs world, a biologics world, in a

          12    medical-device world, and despite the fact

          13    that some people in the world, not just in

          14    the FDA, but in the larger world, think that

          15    everything ought to be regulated the same,

          16    it's a battle we fight around the globe in

          17    terms of not trying to put it all in one box,

          18    and treat it all the same way.

          19              MR. CAMPBELL:  Last comment,

          20    question?

          21              MS. SETHURAMAN:  It's Barathi

          22    Sethuraman again from St. Jude Medical.  One




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            253


           1    of the questions I had was the guidance

           2    automatically assumes that you're going to

           3    have -- especially with this hierarchical

           4    model, you would have complete data,

           5    including covariates on your prior, on prior

           6    data.  Is there any situation where FDA would

           7    allow us to consider summary statistics from

           8    other clinical trials to be used for prior

           9    information?

          10              MR. BERRY:  You're asking the FDA?

          11              MR. CAMPBELL:  I would think the

          12    answer was -- yes, Barbara, I think --

          13              MS. BUCH:  I think we're open to a

          14    lot of discussion if you have -- you know,

          15    there's something actually in the guidance

          16    that says you have to provide a justification

          17    for your approach.  That's sort of a mantra

          18    that we have at FDA.  And so if you can

          19    provide a justification that is something

          20    that we can understand and accept, we're open

          21    to anything that you would bring to the

          22    table, you know.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            254


           1              MR. CAMPBELL:  I think Dr.

           2    Zuckerman from the Office of Device

           3    Evaluation has a comment to make about this,

           4    I suspect.

           5              MR. ZUCKERMAN:  Just from the

           6    cardiovascular viewpoint, I think Barb has

           7    underlined that we're always open to

           8    discussing things with you.  But I think one

           9    of the limitations of utilizing this approach

          10    in the cardiovascular arena is that given

          11    that cardiovascular disease is a

          12    heterogeneous disorder very dependent on

          13    covariates, et cetera, a rate limiting factor

          14    is in many applications right now, not having

          15    access to line data for better modeling

          16    purposes.  So it's a question you should ask

          17    upfront to try to get back to Susan's point

          18    about whether this is going to be least

          19    burdensome or most burdensome.

          20              MR. CAMPBELL:  Thank you.  I want

          21    to thank the audience and the panel I want to

          22    make a couple --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            255


           1              MR. BERRY:  Can I answer some of

           2    the questions?

           3              MR. CAMPBELL:  Some of the

           4    questions?  You've got a minute.

           5              MR. BERRY:  It will be very short.

           6              MR. CAMPBELL:  Go ahead.

           7              MR. BERRY:  Okay.  So Barbara said

           8    we need more decision analysis, and I once

           9    defined a fully Bayesian approach as

          10    incorporating a decision analysis explicitly

          11    in making inferences.  So I completely agree.

          12              She said there's -- it's a

          13    possibility that data contradict the prior.

          14    This is something that should be part of the

          15    planning.  So you should say, "Suppose I get

          16    these data what am I going to conclude," and

          17    "If I get a posterior distribution that I

          18    can't calculate because it's in the

          19    neighborhood where the prior probability is

          20    zero, the likelihood is in the neighborhood

          21    where the prior probability is zero," then

          22    you've got to go back and say this is the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            256


           1    wrong prior.

           2              Bailey said the use of prior

           3    information is controversial.  This is really

           4    important and was discussed somewhat this

           5    morning.  If you go to the Agency and you

           6    say, "I got this trial that I want to use as

           7    a prior distribution for the coming trial,"

           8    the Agency will worry.  They'll worry that

           9    you had another trial out there that you're

          10    not putting forward, that was negative, and

          11    would have been -- you know, make it more

          12    difficult to run the next trial.

          13              They may worry that, in fact, when

          14    you don't come forward with a trial that

          15    you're proposing, that there's negative

          16    trials out there.  It's a very, very

          17    complicated setting.  What it means is -- and

          18    this is my answer to Bailey's other question

          19    as to why do we need a guidance document in

          20    the first place if we're Bayesians -- we have

          21    to be perspective.  It's because we are human

          22    and, therefore, of very limited intellect and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            257


           1    there is ample evidence of that in the world

           2    today that we don't know.  If we're running a

           3    trial, and we say, "Well, now let's do this,

           4    and now let's do that," we pick up a

           5    population that seems to be benefiting,

           6    that's a dangerous thing to do.  Every

           7    statistician knows it's a dangerous thing to

           8    do, and you can't quantify the likelihood.

           9              Telba said that the Bayesian

          10    inference is based on the likelihood.

          11    Sometimes, because we don't know the kinds of

          12    data, and the part of the design is the data,

          13    we don't know how to calculate the

          14    likelihood.  So it's only in the perspective

          15    settings.

          16              So back to the use of the prior

          17    being controversial and suggesting this study

          18    be a prior distribution for the next study,

          19    if that's done perspectively -- I've

          20    suggested this to the Agency, and Greg and

          21    others have indicated that they'll think

          22    about it -- if you say to the Agency, I'm




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            258


           1    setting up this program, I'm going to do

           2    spinal implants, and I'm going to proceed as

           3    follows.  When I get a study, it's going to

           4    provide information for the next study.  So

           5    the whole thing is meta-analysis, but it's

           6    done perspectively so that everything that

           7    happens I'm going to tell you about, I'm

           8    going to include in my likelihood function.

           9    That kind of a thing may be more credible to

          10    the Agency.

          11                   (Laughter)

          12              MR. BERRY:  I'm done.

          13                   (Applause)

          14              MR. CAMPBELL:  Please hold the

          15    break to 15 minutes, go on, come back, enjoy,

          16    don't congregate on the front entrance.

          17                   (Recess)

          18              MR. CAMPBELL:  Okay, I think we're

          19    about ready to get started.  Before we get

          20    stared though, I thought I would take the

          21    liberty of the moderator, and since there are

          22    a fair number of statisticians here in the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            259


           1    audience, sort of do a commercial for a

           2    statistical group that we're trying to build

           3    at the American Statistical Association.

           4              This is an interest group on

           5    statisticians interested in medical devices.

           6    And at the Joint Statistical Meetings in

           7    Seattle, in about two-and-a-half weeks, there

           8    will be 12 sessions on medical devices, all

           9    kinds of medical devices, including one or

          10    two on Bayesian Statistics.

          11              And, in addition, there will be an

          12    organizational meeting of the statisticians

          13    interested in medical devices at this meeting

          14    on Monday, the first Monday in August at 5:00

          15    o'clock.  So if you're interested in finding

          16    out more about this, please slip me your

          17    business card, and I'll be glad to let you

          18    know about that.

          19              The other advertisement is the

          20    statisticians at the FDA, partner with the

          21    Biopharmaceutical Section of the American

          22    Statistical Association, and will put on




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            260


           1    what's called the FDA industry meeting in

           2    late September this year at the Marriott

           3    Wardman Hotel in Washington.  And I think the

           4    dates are September 26th through 28th, and

           5    again -- 27th through 29th; thank you

           6    Estelle.  And so if you want more information

           7    about that, you're welcome to let me know as

           8    well, and I'll send that to you.

           9              Okay, it's a pleasure to begin now

          10    this second panel meeting.  The second panel

          11    is going to think about practical issues, and

          12    I think we have a very good panel here.  We

          13    have Brad Carlin, who is a professor at the

          14    University of Minnesota.  We have Ralph

          15    D'Agostino, who is a professor at Boston

          16    University, and serves on the Harvard

          17    Clinical Research Institute.

          18              We have Professor Steven Goodman,

          19    of Johns Hopkins, who is a professor of

          20    Oncology, Pediatrics, Epidemiology, and

          21    Biostatistics, and the editor of the journal,

          22    Clinical Trials.  We have Roseanne White, who




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            261


           1    is now at Abbott Vascular, from -- used to be

           2    at Guidant, working at the same place, so

           3    with a different change of name, who is a

           4    regulatory fellow at that institution.  And

           5    we also have Dr. Bram Zuckerman, who is the

           6    Director of the Division of Cardiovascular

           7    Devices in the Office of Device Evaluation.

           8              And so I think we're in for quite a

           9    treat.  As before, we're asking each panelist

          10    to try to limit themselves to about five

          11    minutes, and give you folks an opportunity to

          12    contribute as well.  So, without further ado,

          13    Brad, take it away.

          14              MR. CARLIN:  Thanks Greg, thanks

          15    for inviting me here today.  And thanks to

          16    all of you for sticking around this late in

          17    the day, even though National Airport is four

          18    hours from here, apparently.

          19                   (Laughter)

          20              MR. CARLIN:  It's a great document,

          21    and I would certainly want to add my voice to

          22    all those people who've already talked about




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            262


           1    how good the document is, and how overdue it

           2    is, and how essentially needed it is.  And I

           3    think, you know, on the whole, my feelings

           4    about it are very positive.  I especially

           5    like Section 6.5 because it mentions my work,

           6    and thank you for including the Carlin- Lewis

           7    Book.  So I wanted to add that to the docket,

           8    now that's fixed.  I could spend five minutes

           9    just introducing myself, but if Don can do

          10    it, I can do it too.  So, I'm going to try to

          11    be brief, and talk about just a few key

          12    issues.

          13              After all, the overriding concern

          14    here as most folks spoke on is practical

          15    things.  And practically, the bottom line for

          16    us, I guess, is are people going to buy this,

          17    are they going to buy into this, and really

          18    do it.  And is -- does this document give

          19    them enough of the tools and so forth that

          20    they need to do that?

          21              And so to the first issue, I think

          22    people are buying it, and certainly the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            263


           1    growth in the applications are -- is

           2    undeniable, and it's spreading into the drug

           3    side as well as the device side, where it's

           4    been healthy for maybe a decade.

           5              The document stresses the need for

           6    pre-IDE meetings with FDA and that's our

           7    current theme in the document, I keep saying,

           8    and if you're wondering about anything just

           9    call us and talk to us, we'll be happy to

          10    talk to you.  It seems like and as Andy

          11    Mugglin, my colleague, has said a couple of

          12    times; this is a crucial part, you have to be

          13    willing to work with FDA early and often.

          14    And it may seem that there is more upfront

          15    burden associated with this, but provided

          16    that that everyone is onboard with that, I

          17    don't think there's any reason to doubt that

          18    the success will be any different, or the

          19    level of scrutiny overall will be different.

          20              But early, it will be more intense,

          21    and there will be some sophistication that

          22    you'll need to have.  And you may need to, I




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            264


           1    think as Don said, you may need to hire some

           2    people, or consult with some people that have

           3    some understanding of these issues.

           4              And it really is -- there is quite

           5    a lot here, isn't there?  There is a lot of

           6    things about what's a p- value, and what's

           7    power, and how do Bayesians think about that,

           8    and what's the difference between a posterior

           9    probability and a p-value and, you know,

          10    these things are things that Ph.D.

          11    statisticians take for granted, and put on

          12    the Ph.D. exam all the time.

          13              But that doesn't mean everybody

          14    associated with devices really, necessarily,

          15    understands it all.  So there is a lot of

          16    upfront stuff here to think about.  One of

          17    the main things that I was concerned about,

          18    and this has already come up a couple of

          19    times, is the issue of flexibility versus

          20    these simulations that one needs to do.

          21              The document itself touts

          22    flexibility of Bayesian Statistics, and reads




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            265


           1    like a Don Berry press release a couple of

           2    times where it talks about, you know, we can

           3    be flexible, we can be adaptive, if things

           4    can go wrong, no problem, we'll work it out.

           5    But then at the end of the document, we get

           6    into all of the stuff about these frequent

           7    simulations.  And you have to say how many

           8    times you're going to peek at the data, and

           9    Laura had an example, well, she -- well it's

          10    30, or it's 60, or it's 90.

          11              Well, of course, you're not really

          12    going to do that, are you?  You're going to

          13    be able to peek at the data exactly when 60

          14    or 90 pops up.  So the simulations that

          15    you're doing are kind of illustrative in some

          16    sense.  They're the kinds of things that

          17    might happen, but again, the reason we got

          18    into this Bayesian -- they're not "the

          19    reason" but "a reason" we got into this Bayes

          20    game in the first place, was to gain some of

          21    that Bayesian advantage that we'll -- we

          22    don't have to worry so much about when we




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            266


           1    peek.  And the likelihood principle saves us

           2    from a lot of grief that p-value has got us

           3    into.

           4              Now, if we have to give all that

           5    back in terms of pre-specifying looks, the

           6    data, and every single aspect of the design,

           7    you know, I start to question if it's worth

           8    it.  I'm one of those people that isn't that

           9    sold on the need for simulating Type-I error,

          10    and simulating power, at least not in a

          11    traditional way.

          12              I would, at least, like to see the

          13    thing that Roger brought up earlier, which is

          14    you could simulate your fake theta values

          15    from the prior that you've agreed on with

          16    FDA, or maybe from some other source that

          17    need not even be the same prior that's used

          18    at the analysis phase.

          19              And that's another thing that's

          20    missing from the document is that the prior

          21    that you sample your fake thetas from when

          22    you're generating your design simulation




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            267


           1    values need not be the same prior that goes

           2    into the posterior calculation.  That prior

           3    might be more conservative, it might be a

           4    little more reflective of FDA's skepticism

           5    about the whole operation.

           6              You want to be -- you want to

           7    generate fake data (sic) values that are

           8    really likely to happen, but you want to play

           9    it safe when you actually do the analysis.

          10    So, that's something that's now I think

          11    fairly standard in these -- in this area, and

          12    that's not in the document.

          13              There is a couple of other

          14    ambiguities in the document; I won't make fun

          15    of the document at great length, I'll do that

          16    in the docket, don't worry.  The one point on

          17    page 22, it lists five different books that

          18    you can go to, to look for design advice.

          19    And I was thinking to myself, well, any of

          20    these five are okay?

          21              I mean there -- the document is --

          22    the devil is in the details, as Janet said




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            268


           1    earlier, and the details are largely missing

           2    from this document.  I realize the document

           3    wasn't intended to provide a lot of detail,

           4    but many of you are still in this room

           5    because you are statisticians and you do

           6    crave some -- you know, guidance on how to

           7    really do the nit picky details of what's to

           8    be done here.

           9              And in that regard, I was

          10    disappointed in some of the very end of the

          11    document.  I thought that the appendix --

          12    some of the things about software, sort of

          13    out-of-date code, and as -- plus -- as

          14    mentioned about WinBUGS, OpenBUGS and BRugs.

          15    For those of you who don't know BRugs is a

          16    product that incorporates OpenBUGS, the open

          17    source version of WinBUGS, into R itself.

          18              So you can download R for free from

          19    the website, and then load BRugs, which is

          20    all the OpenBUGS commands from within R.  So

          21    all of this can be done fairly automatically

          22    including the simulations that you need to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            269


           1    do.  These looping calculations are important

           2    where you're running Gibb samplers at every

           3    stage of a sampler, of an outer iteration or

           4    an outer simulation study.  That can all be

           5    done fairly easily.

           6              So I would like to see the document

           7    get updated in that respect.  But again, on

           8    the whole, I thought it was a really good

           9    document.  One, more thing I'm -- my five

          10    minutes are up, but one more thing is the

          11    issue of software validation.  Greg mentioned

          12    earlier in his remarks that it's very common

          13    for new areas like Bayesian Statistics to not

          14    have a validated piece of software but I'm

          15    sure that'll get taken care of in the long

          16    run.

          17              Well, and again, academic

          18    Bayesian's view, WinBUGS have validated.  I

          19    mean this is what we've -- this is our SAS, I

          20    mean this is the hunk of software that you

          21    try first.  And if it doesn't work then you

          22    write your own code, you get -- or you get




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            270


           1    Scott Berry to write your own code.

           2              SPEAKER:  Uh-huh.

           3              MR. CARLIN:  But you know, that's

           4    the package, so I'm not sure there's a plan

           5    for validating WinBUGS or producing validated

           6    -- I'm not even sure exactly what that means.

           7    And the current method of validation seems to

           8    be, well, you run it in WinBUGS, and then you

           9    try it in Scott's software, and if you get

          10    the same answer, you feel pretty good about

          11    it.  You know, I think we need to talk a

          12    little bit more about how we're going to

          13    validate software in the future, what's our

          14    plan for doing that.

          15              MR. CAMPBELL:  Dr. Agostino.

          16              MR. D'AGOSTINO:  Yeah, I'd like to

          17    make -- well, first of all, thank you for

          18    inviting me to the session.  And then I'd

          19    also want to make a couple of comments before

          20    I go into detail.  I've been affiliated with

          21    the FDA for over 30 years on a number of

          22    advisory committees, and reviewed a number of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            271


           1    submissions, and with the drug, excuse me,

           2    with devices in particular.  Five-six years,

           3    I've been quite active in advisory committee,

           4    or again, I have an affiliation with Harvard

           5    Clinical Research, where we look at stents

           6    all the time.

           7              And my comment in this is that most

           8    of the submissions to the FDA, and most of

           9    the submissions to the devices is not using

          10    -- or do not use Bayesian techniques; that

          11    should be made quite clear.  And so I think

          12    when you -- when we look at this document,

          13    and here's my second opening comment is I'm

          14    looking at it as a person who deals with a

          15    lot of clinical trials and also on a number

          16    of advisory committees.

          17              And what is the reason one would

          18    look at Bayesian approach?  Why does one want

          19    to actually look at a Bayesian approach?

          20    Occasionally, I will get a call saying, "At

          21    Harvard Clinical Research, do you have

          22    somebody who can do Bayesian analysis, a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            272


           1    statistician?"  We say yes.  They come in,

           2    and then we tell them after 15 minutes that

           3    they really don't have to do a Bayesian

           4    analysis if they want an approval.

           5              But then the question becomes why

           6    would you do it, should you do it, would you

           7    gain something from it?  And I think the

           8    emphasis, from my point of view, is their

           9    Bayesian techniques offer something, which

          10    the previous frequency techniques don't

          11    offer.  They offer us an awful lot more,

          12    there is a nice way of looking at the data,

          13    there is a systematic way of analyzing the

          14    data.

          15              And we should think about them, but

          16    in order to do it, we have to convince people

          17    that they're useful.  And in particular, with

          18    the advisory committee, we have to inform the

          19    advisory committees, how in the world do they

          20    interpret it, and how they put it in the

          21    right setting.

          22              So the comments I'm going to make




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            273


           1    sort of flow from that.  If you're talking

           2    about dealing with trials and you're talking

           3    about advisory committee meeting -- advisory

           4    committee members in this document, what's

           5    useful.

           6              The first thing is the vocabulary.

           7    The people reading this will want a familiar

           8    vocabulary or vocabulary they can become

           9    comfortable with.  When you start using

          10    exchangeability, you define it, but people

          11    who do trials all the time understand

          12    generalized ability, do understand

          13    compatibility and comparability.  So this is

          14    not a -- the vocabulary is not really

          15    off-the-wall different, and having the

          16    guidance document sort of show you where

          17    these pieces of vocabulary, sort of meld into

          18    clinical trial ideas.

          19              I think it is something that could

          20    be handled a little bit better, and maybe the

          21    document should be looked at carefully for

          22    that.  The other thing is that -- comes up,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            274


           1    the very first thing that comes up after you

           2    talk about -- after you finish the talk about

           3    the vocabulary, is what you mean by "priors."

           4    And I think that, maybe examples or maybe

           5    some understanding of how one deals with

           6    these priors, this uninformative priors

           7    scares the hell out of people because they

           8    don't really know what that means.

           9              You say, well, it -- you just lay

          10    out a prior over everything that's equal; and

          11    it is not really understood.  Then you start

          12    talking about making a prior out of previous

          13    work, I can imagine what would happen in the

          14    advisory committee when you start laying out

          15    the priors to individuals not very

          16    comfortable with Bayesian techniques, but

          17    very bias in the abuse of what the field is

          18    about.  Moving on, another problem I think

          19    with the Bayesian techniques in the sense of

          20    laying them out is the models.  The models

          21    get very elaborate very quickly.

          22              A lot of clinical trial is, myself




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            275


           1    included, like to think of using robust

           2    techniques.  And you develop a test, you do a

           3    lot of careful randomization, and then at the

           4    end of the day when you have all your data,

           5    you can do simple, relatively simple

           6    analysis.

           7              The impression you get from the

           8    Bayesian is that it gets very complicated,

           9    and that the techniques are very dependent on

          10    the models.  And this notion of just how much

          11    is dependant, how much can you appeal to

          12    robust techniques, I think really are how

          13    robust are other techniques.  And how do the

          14    simulations fill that has to, I think come --

          15    be made clear, and come through a little bit

          16    more forcefully.

          17              Some of the other issues is we

          18    talked about interim analysis.  One of the

          19    things that I hear all the time is how

          20    Bayesian techniques will save interim

          21    analysis -- it'll save data, because you'll

          22    be able to do interim analysis on the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            276


           1    clinical trials.

           2              And if you respond to those interim

           3    analysis, the Bayesian techniques will always

           4    save you, and you'll be able to stop to study

           5    drop arms, change sample size, make decisions

           6    based on these techniques without any sort of

           7    loss of validity at the end.  I'm not sure

           8    that's really true, and I think that the

           9    dangers of interim analysis, that we have

          10    with frequency methods also hold somehow with

          11    the Bayesian and that has to be made clearer.

          12              When you come to end of the day and

          13    you do your analysis, one of the big things

          14    that drives, in my experience, advisory

          15    committees, is they come together, they are

          16    called together because something different

          17    didn't happen, or something different

          18    happened than what you expected or what they

          19    wanted.

          20              If the study has a primary outcome,

          21    and that primary outcome turns out to show

          22    significant differences between the groups,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            277


           1    then nobody is worried about it.  It's when

           2    the primary outcome shifts a bit because the

           3    one that was declared in the protocol, the

           4    group that was declared in the protocol as

           5    the primary group in the primary outcome

           6    mortality at six months or something like

           7    that isn't significant.  But stenosis at

           8    three months is statistically significant,

           9    and there's a shifting.

          10              Well, the sense is, from some

          11    people, innocently, is that Bayesian

          12    techniques will allow you to shift from this

          13    original primary outcome to the -- moving to

          14    another primary outcome, and I don't think

          15    that's true.  And I don't think the document

          16    is -- maybe shouldn't, but I don't think the

          17    document is addressing those type of uses.

          18              Other things that happen is

          19    multiplicity in shifts to subsets of

          20    individuals.  There is a lot of sense that

          21    somehow rather, because the Bayesian

          22    techniques are dealing with priors and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            278


           1    dealing with groups, and dealing with

           2    likelihoods, that they can shift around to

           3    subsets, they can shift around to different

           4    outcomes without any loss of validity.

           5    Again, I don't think that's true, and I don't

           6    think the document necessarily makes that

           7    clear.

           8              Another thing that I find of

           9    interest to me is when is the Bayesian

          10    technique useful.  At Harvard Clinical

          11    Research, we have tons of data, literally,

          12    tons of data, they're all stents, we have --

          13    lots of data are on stents that have been

          14    accumulated over the years.  It makes a

          15    beautiful historical control, with

          16    appropriate techniques, we can talk about

          17    one-arm studies, we can talk about using

          18    these historical controls to build a valid

          19    analysis, not just a look-see, but a real

          20    valid analysis in the Bayesian context.

          21              And I think maybe if the document

          22    gave some more examples on when it's -- when




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            279


           1    is it useful, when can you actually use these

           2    techniques, and what has been successful, I

           3    think that would be a help to the user and to

           4    the reader.  Another issue is how do you look

           5    at your results of the study of the success

           6    criteria.  Now, I was expecting when I read

           7    the document that it would really run.  And I

           8    think it's what Brad was saying, and others

           9    have said that would really run the Bayesian

          10    -- it would really run the Bayesian view of

          11    success criterion.

          12              And then suddenly there's all this

          13    Type-I, Type- II, now myself as a confirmed

          14    Frequentist as opposed to a Bayesian, that's

          15    nice to see that.  But I think it sort of

          16    takes the wind out of the Bayesian argument

          17    or the Bayesian structure and maybe it should

          18    be that way.  But I think this is going to --

          19    it will create some difficulty in terms of

          20    understanding how these fit in differently,

          21    or are they different.

          22              And lastly, I want to make a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            280


           1    comment about the software.  I am worried

           2    about the validation; Brad seems to have less

           3    concern, and is the FDA considering, for

           4    example, having code saved and made

           5    available.  Are there ways that the FDA can

           6    be helpful in terms of showing what has been

           7    successful, how codes have been put together,

           8    and how (off mike) studies that have actually

           9    worked out and been successful.  Thank you

          10    for your attention.

          11              MR. CAMPBELL:  Thank you, Ralph.

          12    Professor Goodman.

          13              MR. GOODMAN:  Thank you.  Well, I

          14    used a few of my minutes before; I can't dare

          15    to inflict too many more on you now.  So I'll

          16    try to be very short.  I too thought this is

          17    a big -- great step forward, and the comments

          18    I make, I don't want to be a knock on it.

          19    Because I think it -- even in its current,

          20    maybe incomplete and imperfect form, it's

          21    light-years better than what we had before

          22    which was basically nothing.  So I think it's




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            281


           1    great, and a wonderful first step.

           2              Just one or two things.  First, I

           3    was both surprised and a little bit concerned

           4    that it -- there was no use, and I mentioned

           5    this to Greg at one of the breaks, there was

           6    no mention of actual measures of evidence

           7    like the Bayes factor or likelihood ratios or

           8    ways to compress the likelihood function,

           9    although that's of course all part of

          10    Bayesian technology.

          11              I have found that looking at,

          12    particularly, looking at the function of the

          13    prior as summarizing the likelihood function

          14    and talking in terms of evidence to provide,

          15    first of all, common ground with those who

          16    are not comfortable talking in terms of

          17    priors and posteriors and belief.  And also

          18    it shows that we are all worried about the

          19    same thing, which is what does the evidence

          20    say?

          21              And I think that even though Bayes

          22    factors have their own issues and don't --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            282


           1    you can't -- they are not as flexible and you

           2    can't do as many things you can with

           3    posteriors, they are a very, very important

           4    tool both for planning, for communication,

           5    and I think that there really should be some

           6    mention of that and direction of people

           7    reading it.  Many of whom this will be the

           8    first Bayesian document they have ever read

           9    or maybe could ever understand.

          10              So I think it is very, very

          11    important to have some of that literature

          12    referred to and have it at peoples'

          13    fingertips, because in fact in the diagnostic

          14    testing arena where they are familiar with

          15    likely ratios and things like that, people

          16    already know how to talk that way.  So, I

          17    spoke about this at the earlier workshop that

          18    we had about a few years ago, and there are

          19    many people who've written about that.  So

          20    I'd like to see more reflection of that.

          21              With respect to the -- this issue

          22    of the stopping rules and the operating




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            283


           1    characteristics, it's been mentioned several

           2    times, so I just want to give my own spin on

           3    it.  I do think -- I understand the appeal,

           4    but I think it's a surrogate for something

           5    else, and that you're concerned about and I

           6    want to put it on the table because the way

           7    it's framed right now, people could be in a

           8    -- get themselves in a real sort of

           9    conundrum.

          10              Because, you can have a situation

          11    where you have two different designs and they

          12    could involve different sample sizes and

          13    different stopping or monitoring plans that

          14    have the same operational characteristics,

          15    and yet they will have -- I'm sorry, they

          16    will have different operational

          17    characteristics, but they'll have the same

          18    inference based on the posterior.

          19              And, you can't have it both ways,

          20    you can't have the FDA saying it's going to

          21    react differently to different operating

          22    characteristics, and yet it is going to base




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            284


           1    inferences on the posterior.  And this was

           2    brought up this morning.

           3              Now, I think what this is a

           4    surrogate for is a fundamental discomfort

           5    with using the posterior, which is in itself

           6    a surrogate for discomfort with the priors.

           7              And, the reason is, I think it's

           8    important for people to understand, the

           9    reason that we stress monitoring boundaries

          10    in a Frequentist framework is that, in

          11    theory, the probability of a Type-I error can

          12    go to one.  If you just keep looking, you're

          13    bound to see eventually or with very, very

          14    high probability a significant P-value.

          15              That's not true with the Bayesian

          16    analysis.  There is a bound on how frequently

          17    you can come up with a -- with the incorrect

          18    answer.  And it's directly related to the

          19    strength of the evidence that you're seeking.

          20    So, when you monitor that -- and that bound

          21    is achieved when you look constantly.  That

          22    is after every subject.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            285


           1              So, the issue -- there is a direct

           2    link between the strength of evidence and the

           3    frequency of error.  Now, that link becomes

           4    weaker when the prior is wrong.  So, when the

           5    prior is wrong, then it doesn't go to one but

           6    the bound is -- the connection is not as

           7    close.  So, the difference is that you see

           8    between that bound and the different

           9    monitoring rules are -- shouldn't affect your

          10    inference.  But I think what you're really

          11    concerned about is that the -- you're not so

          12    sure about the prior.

          13              And so I think the real answer to

          14    this is not to say let's look at simulated

          15    operating characteristics.  But let's try to

          16    figure out how to better represent the

          17    uncertainty due to uncertainties in the prior

          18    that is robust Bayes or averaging over those

          19    results.  Because in the end, even though it

          20    was said that, you know, well, we feel more

          21    comfortable knowing what the Type-I error

          22    rate and Type-II error rate is.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            286


           1              I think that the posterior, the

           2    probability that you're right or wrong at the

           3    end of the day, I don't know what else --

           4    what other information you need.  That's the

           5    only information you need.  So, the Type-I

           6    error, Type-II error feed into that.  If

           7    either of them are excessively high, your

           8    probability of wrong -- being wrong at the

           9    end of the day is going to be high.

          10              So I don't completely understand

          11    how you're on the one hand use these

          12    simulated operational characteristics, on the

          13    other hand use the posteriors.  I think there

          14    is an inherent conflict there that you're

          15    going to start confronting.  And as I said, I

          16    think that the answer is in doing things to

          17    better robustify inference with respect to

          18    the priors.

          19              I think that's what really is at

          20    stake, and that you ultimately are going to

          21    have to land in one camp or the other.

          22    You're going to take the posterior seriously




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            287


           1    and allow people to make inferences on them

           2    or not.  I don't think it's going to be

           3    possible to straddle that divide.

           4              The only other thing I'll say,

           5    which was touched on, which is that right

           6    now, Greg, you mentioned that only about, you

           7    know, five to ten percent of your work at

           8    most is Bayes and that means some smaller

           9    percentage of your total staff has to be

          10    comfortable with Bayesian methods.

          11              I think if this guidance is

          12    successful and that percentage increases, and

          13    I agree with Dr. D'Agustino, I don't think

          14    it's going to go, you know, to 80 percent or

          15    60 percent, you know, it might go to 40

          16    percent or whatever.  But it's going to go

          17    up, you know, it might go up three-fold or

          18    four-fold.  You have to make sure you have

          19    the staff to handle it.  And I think the --

          20    one of the biggest disasters could happen is

          21    when you wave this big invitation flag, and

          22    people come in and they do good work, and




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            288


           1    they are just shut down at the door and they

           2    have some very bad experiences.

           3              And, I've been on both sides of

           4    this and I know what it feels like, and you

           5    feel like you've been slapped in the face and

           6    betrayed, you know, you're invited to a party

           7    and then they, you know, throw mud at you

           8    when you get there.  So, you really have to

           9    be ready for the guests to come and I hope

          10    that planning is well underway.

          11              MR. CAMPBELL:  Well, we are ready

          12    for the guests to come, and funny thing you

          13    should mention that is that we're recruiting

          14    at the joint statistical meetings, and my mic

          15    isn't working or something.  Okay.

          16              MR. GOODMAN:  Use this one.

          17              MR. CAMPBELL:  So, it's not -- but

          18    let me just say for the record that we're

          19    recruiting at the joint statistical meetings

          20    for both Bayesians and non-Bayesians, and

          21    hopefully, we won't be shocked when the

          22    deluge occurs, but thanks very much.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            289


           1    Roseanne White.

           2              MS. WHITE:  I want to thank

           3    everybody for inviting me to this particular

           4    meeting, and to thank everybody for covering

           5    everything I was going to say, so I can make

           6    my comments really short.  We really feel we

           7    have our statistics-working group within

           8    AdvaMed and we discussed this guidance

           9    document and we really feel that this is a

          10    great guidance document, but we would like to

          11    see more details.

          12              And I think a lot of the

          13    presentations that FDA did this morning were

          14    really good.  And we'd like to see some of

          15    those details that were in the presentations

          16    incorporated into the document.  Couple of

          17    things that have come up during the course of

          18    the day that I would like to sort of point

          19    out is one of the cases is we have talked

          20    about missing data.  And the guidance

          21    document says oh, with Bayesian you can

          22    imputate missing data all over the place.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            290


           1              Well, I think there needs to be

           2    some warning that well, not 50 percent of

           3    your data can be missing even though Bayesian

           4    is really great at predicting it.  So I need

           5    some boundaries around where we're going to

           6    be looking at missing data.

           7              Another piece is talking about the

           8    surrogate endpoints.  I do know that the

           9    surrogate mentioned in the document is not

          10    the surrogate as according to the Prentice

          11    Criteria.  However, we'd like to know

          12    'because, you know, we're a very aggressive

          13    industry, we might like to do surrogate

          14    adaptive Bayesian trial where we're taking

          15    advantage of all the things that could

          16    happen.

          17              And so we need I think at least

          18    some guidance along the way in terms of using

          19    Bayesian and surrogate endpoints in terms of

          20    doing clinical trials.

          21              And we're -- also have the

          22    controversy about doing the operational




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            291


           1    characteristic curve, and is it really pure

           2    Bayesian to do it.  However, we sort of

           3    recognized that that is allows us to do some

           4    comfort -- have some comfort in terms of the

           5    Type-I and Type-II error.  But it also

           6    provides some of us in the industry a way of

           7    being able to explain it to our colleagues

           8    when they really don't understand the

           9    Bayesian, and until they become comfortable

          10    with Bayesian, it allows us to give them

          11    something that they are familiar with, which

          12    is the Type-I and the Type-II error.

          13              But, in conclusion, one of the

          14    things we do recognize is FDA needs to have

          15    the staff for Bayesian and also the industry

          16    needs to have the staff for Bayesian.  So I'd

          17    like to pose a challenge to my academic

          18    colleagues.  So, where are my new employees

          19    coming from?

          20              In the light of the Bayesian

          21    programs, it's nothing -- they don't even

          22    come close to anything that's like clinical




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            292


           1    trials.  Another, the many of the grey-

           2    haired Frequentist out there, which I

           3    consider myself one of them, it's nice to see

           4    that we'll be employed.  However, at some

           5    point, you have enough grey-haired

           6    Frequentist and you need to hire a Bayesian.

           7    And you would like them to have at least some

           8    idea of what a clinical trial is all about as

           9    opposed to applying Bayesian into areas that

          10    have absolutely no idea.

          11              So, help us.  Thank you.

          12              MR. CAMPBELL:  Thank you, Roseanne.

          13    And now, Dr. Bram Zuckerman.

          14              MR. ZUCKERMAN:  Okay.  This has

          15    been a very rich discussion and it's a very

          16    challenging area.  But I would like to get

          17    back to some general points.  Guidance

          18    documents only provide guidance and we're not

          19    going to get all the answers here.  And I

          20    think we've heard why we need continued

          21    interaction between industry, FDA, and

          22    academia for further development of this




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            293


           1    field.

           2              But, I do think the industry has

           3    embodied in Susan Alpert's first question has

           4    a legitimate question to ask, and that is

           5    right upfront, you know, why use Bayes, can

           6    you get a early read from the Agency as to

           7    whether that's going to be least burdensome

           8    or perhaps not the way to go.

           9              And in considering that question, I

          10    would suggest that the industry and

          11    consultants appreciate one thing that hasn't

          12    been brought up today.  In considering

          13    technology, I should say, in a lot of detail,

          14    in considering technology maturation, we

          15    frequently go quickly from one device

          16    generation to the next.  And the bare metal

          17    stent example that Ralph pointed out is a

          18    great one.

          19              In quirk of the Agency's

          20    willingness to consider Bayesian methodology

          21    is really going to be dependent on convincing

          22    our engineering staff that we're talking




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            294


           1    about minor changes versus major changes.

           2              In other words, Bayesian can't be a

           3    substitute for good sound science.  And, the

           4    example for -- the example that Ralph points

           5    to, the bare metal stent example, is a great

           6    one in that our center was willing to adopt a

           7    Bayesian approach and go to a one arm OPC

           8    type trial with adjustment for covariates

           9    after we saw enough data that convinced us

          10    that a stent struck change wasn't a big deal

          11    for hemodynamics, et cetera.

          12              What that really meant was that all

          13    the three parties in this room had to work

          14    together to develop a data set that was

          15    appropriate for modeling.  You know, we heard

          16    Don Berry and others state today that the

          17    Bayesian methodology is great for modeling,

          18    which is true.  But, the major problem with

          19    device technology development right now is

          20    that, individual manufacturers are generally

          21    doing modest trials.

          22              And for us to develop the modeling




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            295


           1    that would be appropriate for many areas of

           2    cardiovascular development, to more optimally

           3    employ the Bayesian methodology, there needs

           4    to be this sort of cooperation that was

           5    exhibited in the bare metal stent example, in

           6    an ideal world.  And I would encourage Greg

           7    to put that reference into the guidance.

           8              So what's the analog of, say the

           9    bare metal stent example, right now in

          10    cardiovascular?  It's probably the embolic

          11    protection devices, where you have a

          12    technology maturation of sophisticated guide

          13    wires, so that you have low event rates, with

          14    equivalence trials right now being done in

          15    the frequency domain of 800 to 1000 patients.

          16              Well, it's difficult to predict a

          17    priori what type of results you're going to

          18    see without a rather sizable dataset.  That

          19    apparently according to this week's

          20    circulation has been analyzed again at HCRI

          21    et cetera.  It's that sort of cooperative

          22    effort that makes things easier for the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            296


           1    Agency to accept this type of methodology.

           2              So to sum it up, number one, I do

           3    think you need to present early to the

           4    Agency, convince our engineering staff that

           5    we're talking about combinable data, i.e.,

           6    minor engineering changes, when appropriate.

           7              And two, we're looking at better

           8    ways to model data through cooperative

           9    efforts.  Thank you.

          10                   (Applause)

          11              MR. CAMPBELL:  Okay.  So the

          12    microphones are open, and my microphone's

          13    working again, thanks to Larry.  So, if you

          14    have a question for the panel, or a comment,

          15    this is the time to come up to the

          16    microphone, and while people are gathering at

          17    the microphone, let me give an opportunity of

          18    the panel members to respond to each other if

          19    they so choose.

          20              Anyone have any comments from the

          21    panel?  If not, our first -- please identify

          22    yourself and your affiliation and ask your




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            297


           1    question or make your comment.

           2              MS. ZHANG:  This is Xiaozheng Zhang

           3    from Edwards Lifesciences.  Dr. Zuckerman

           4    just mentioned the use of Bayesian approach

           5    for retrospective analysis.  I thought it's

           6    very interesting.  And since Dr. Zuckerman is

           7    here, you know, I'm thinking about, say for

           8    example, some long- term durability analysis,

           9    and we -- very difficult to conclude it at

          10    end because three year after some of the

          11    patient died.

          12              You know, you are never going to

          13    get a 40-sample size follow that norm.  So,

          14    is that something, you know, your panel can

          15    address, a little bit more?  See, if there is

          16    possibility we are using those approaches.

          17    The way you look at the, say for example

          18    heart valve, you know, follow up, you know,

          19    three year and we got more then 80 percent

          20    die, and maybe in the early year time we can

          21    somehow use the Bayesian approach to really

          22    conclude some of the durability issues.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            298


           1              MR. ZUCKERMAN:  Okay, maybe I can

           2    rephrase that so that we can get better panel

           3    response.  I think you're talking about a

           4    general problem that we have, which is time

           5    predictive modeling and whether we can use

           6    early results to better predict long-term

           7    results.

           8              In cardiovascular, we're, you know,

           9    worried about long-term durability, rare

          10    failures.  I think the spinal cage example

          11    that Dr. Berry and others referred to is one

          12    example of that type of approach.  Comments?

          13              MR. D'AGOSTINO:  Give it to Brad.

          14              MR. GOODMAN:  Maybe I should give

          15    it to Don.

          16              MR. CARLIN:  Predictive

          17    distributions as Don has been arguing for is

          18    a long -- been a long time, around a long

          19    time, and I think it's another kind of almost

          20    a design issue.  As long as you, sort of,

          21    say, well, we're going to use this predictive

          22    to make our decision, down the road, we're




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            299


           1    going to use this posterior, you know, we can

           2    decide any stopping criteria we like and then

           3    as long as we stick to the program, yeah I

           4    agree with you, it's perfectly legitimate to

           5    do it that way.

           6              MR. GOODMAN:  Yeah, I think, if I'm

           7    not mistaken, part of what you're saying is

           8    can we -- can the endpoints evolve as we

           9    learn about the relationship between the

          10    early markers and the final endpoints.  And

          11    there are a number of examples of that is, we

          12    just pointed out the spinal implant was one

          13    of them.  And Don's been doing a lot of this.

          14    I don't know if he wants to speak to that.

          15              But, you start basically using the

          16    initial sample to measure the relationship

          17    between the -- what we might call the

          18    surrogate, we won't call that the more

          19    proximal endpoint and the final endpoint.

          20    And to the extent that as evidence

          21    accumulates, it's shown to be either a good

          22    or bad predictor of the final endpoint.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            300


           1              You can start using that more and

           2    more if it's good, and less and less, if it's

           3    bad as the study goes on.  But I'll defer to

           4    Don.

           5              MR. BERRY:  What you say is exactly

           6    right.  And what you gain is the ability to

           7    make decisions sooner if it's an adaptive

           8    trial in a sense of modifications other than

           9    stopping, then you can do that.  Of course,

          10    to get back to Ralph's point, you have to

          11    prospectively define this so that you can

          12    address the implications of it.  This is

          13    something that we really have to be doing in

          14    many diseases.

          15              You don't have to do it in

          16    migraine.  Migraine, you get the endpoint in

          17    two hours.  In the spinal implant setting,

          18    the endpoint decided by the panel and the FDA

          19    is two years.  But there's lots of

          20    information that accrues over the course of

          21    that two years and you ought to be able to

          22    take advantage of it.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            301


           1              SPEAKER:  In cancer, the endpoint

           2    is (off mike) plus we've learned about

           3    progressive -- progression, we have learned

           4    about tumor response, we've learned about

           5    biomarkers.  With the Critical Path

           6    Initiative, the FDA has two focuses.  Who

           7    died -- the Critical Path Initiative has two

           8    focuses; one is using biomarkers, other is

           9    innovative product design.

          10              And they come together in a very

          11    important way in this kind of picture, and it

          12    requires some modeling, but as you say,

          13    Steve, it's a learning about the relationship

          14    and to hit at some of Ralph's important

          15    points, it's confirming what that

          16    relationship is.

          17              MR. GOODMAN:  What is unique about

          18    Bayesian to do that?

          19              MR. BERRY:  Nothing.

          20              MR. GOODMAN:  Nothing.

          21              MR. BERRY:  Except for it's a

          22    natural --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            302


           1              MR. GOODMAN:  Yeah, we're doing

           2    cardiovascular medicine all the time and

           3    things like that, but --

           4              MR. BERRY:  Yeah, well --

           5              MR. GOODMAN:  But it's not -- I'm

           6    familiar with your work, not as you do it,

           7    but, you know, it's just one method for

           8    getting out of this is there something you

           9    see that's unique about Bayesian that will --

          10              MR. BERRY:  Something that is

          11    unique about Bayesian is the modeling as a

          12    natural thing, and then using evidence is a

          13    natural thing.  You can do predictive

          14    distributions in a very natural way.  In the

          15    cardiovascular setting that I, you know much

          16    better than I, but you model these things,

          17    but it's typical of other kind of diseases

          18    where you -- there is a Phase-Two endpoint

          19    and a Phase-Three endpoint, and you look at

          20    them individually, but not together.

          21              MR. GOODMAN:  Yeah, you --

          22              MR. BERRY:  We're very bad in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            303


           1    medicine generally, modeling the course of

           2    disease.

           3              MR. GOODMAN:  You keep stopping and

           4    starting in the methods.  I'm talking about

           5    yours -- yours would have some sort of flow

           6    to it.

           7              MR. BERRY:  Yes, yes, and

           8    incorporating the longitudinal aspect of the

           9    disease with possibly the same marker, so in

          10    the case of the orthopedic application in

          11    spinal implants, it's the same measure, just

          12    at different times.  But we've got to do more

          13    than that, we've got to incorporate other

          14    kinds of possibly correlated measures

          15    including biomarkers, micro arrays et cetera.

          16    And so both pieces of the critical path come

          17    together in this application.

          18              MR. ZUCKERMAN:  Okay, and if I

          19    could just add one thing.  One point has been

          20    that, you know, the lack of examples in the

          21    literature, the spinal cage is a great

          22    example article.  We are seeing examples in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            304


           1    cardiovascular right now, where companies are

           2    trying to apply the same principle, and we

           3    would encourage that our Bayesian

           4    statisticians are extremely helpful with this

           5    problem.

           6              MR. MEHTA:  Yes, Cyrus Mehta from

           7    Cytel.  As one who has spent most of my

           8    software development career, and teaching,

           9    and research career in the Frequentist

          10    domain, I'm very interesting in following up

          11    on Ralph's comment -- Ralph D'Agostino who

          12    asked the question, "Well, what do we really

          13    gain by using the Frequentist approach?"  And

          14    so yes, you get a more elegant way of looking

          15    at the data for sure.  And then you have a

          16    new magical computational tool; the Markov

          17    Chain Monte Carlo.  But none of that really

          18    carries weight in a pharmaceutical company.

          19              What really would strike someone if

          20    -- from a drug company or a device company

          21    here today would be where one of the earlier

          22    presentations said, "Well, you can reduce the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            305


           1    sample size."  Now, that really is what -- is

           2    the sweet spot.  Can you really reduce the

           3    sample --

           4              Nobody cares about how elegant the

           5    presentation is except someone who is working

           6    in statistics.  But, you know, can you reduce

           7    the sample -- and how can you reduce the

           8    sample size.  It seems to me the only way you

           9    can reduce the sample size is through the

          10    choice of the prior.  And the choice of the

          11    prior is the key here to getting these

          12    methods adopted broadly.

          13              If you had to go out and consult

          14    and tell our clients "Well, we can reduce the

          15    sample size; we need access to priors," who

          16    has access to priors?  It's the CDRH that has

          17    access to priors.  They have access to all

          18    the priors because they get all the

          19    submissions from everybody.  And, you know, I

          20    wonder if it's ever going to be possible that

          21    they'll be available in some public

          22    repository appropriately classified by




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            306


           1    medical device and we can have access to

           2    them.  And that's how science will progress.

           3              MR. ZUCKERMAN:  Okay.  If I could

           4    just briefly respond to that.  The data as

           5    you pointed out is extremely important to

           6    construct the prior.  That's principle number

           7    1 that we've all agreed to, but we live in a

           8    regulatory environment, and this is why the

           9    bare metal stent example that Ralph points to

          10    is a great one.

          11              In that the industry, FDA, and

          12    academia were able to get the data together

          13    to model it, to come up with appropriate

          14    priors and a methodology that would allow for

          15    one-arm trials, but in the end, it's not FDA

          16    who holds the final card regarding use of

          17    these data.  It's the industry.  And, you

          18    know, the real challenge is to see how we can

          19    convince the industry to develop these

          20    available datasets.  Perhaps someone from the

          21    industry wants to respond.

          22              MS. WHITE:  Since I'm the only one




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            307


           1    from the industry, I guess I have to respond.

           2    One of the challenges again, in trying to get

           3    co-operation in terms of datasets is, each of

           4    us feel as our data is our proprietary

           5    ownership, and gives us a competitive

           6    advantage.  And we don't want to be able to

           7    lose that competitive advantage by

           8    necessarily sharing the data with everybody

           9    else.

          10              At the same time, we also do

          11    recognize that there are times in which it's

          12    important to have some sort of way of being

          13    able to share information when we get to a

          14    point where there is so much evidence there

          15    that will allow more diversity in the

          16    industry of smaller companies to be able to

          17    compete.  And so that is the challenge that

          18    we face is particularly early on -- we really

          19    see our data as a competitive advantage.

          20              Once we get to a point where there

          21    is so many for example, the stents, there are

          22    so many stents out there, really what we want




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            308


           1    to do is to make sure there is diversity in

           2    the marketplace.  Then that's a point at

           3    which we would probably want to share the

           4    data so that smaller companies could come and

           5    compete and be able to use these Bayesian

           6    activities or OPCs et cetera.

           7              MR. GOODMAN:  Yeah.  This is

           8    something that came up in the workshop a few

           9    years ago, and something I honestly had never

          10    thought of; the idea that for one company to

          11    make their data available to another company

          12    is in a sense gifting the second company with

          13    patients they don't have to pay for because

          14    now, the first study's investment then goes

          15    to benefit the second company.  You know, I

          16    can see that's a very real issue.

          17              I don't know that statisticians

          18    have the answer to that.  I think it is, you

          19    know, when you talk about borrowing strength,

          20    you know, in this case it's borrowing dollars

          21    essentially.  And then in a competitive

          22    marketplace, I think these are going to be




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            309


           1    some very, very real challenges that are

           2    going to have to be negotiated both within

           3    the industry, and between the industry and

           4    FDA.  I think it's very true.

           5              I -- and particularly in the -- as

           6    you say in the early developmental stages,

           7    which is exactly when the FDA is most

           8    involved, almost all the information is

           9    proprietary.  It's not out there, you know,

          10    for -- to access with systematic reviews and

          11    things like that.

          12              MR. CAMPBELL:  It is true, I should

          13    say, that in terms of examples that people

          14    want to see examples of Bayesian submissions

          15    that had been successful in CDRH.  CDRH

          16    website contains the summary of safety and

          17    effectiveness for PM -- original PMAs and

          18    some supplements.  And so you can go to the

          19    website, and I think the guidance document

          20    gives a number of examples of that sort.

          21              One of the interesting things is

          22    although there is a fair amount of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            310


           1    information there, the data is still owned by

           2    the company.  And so the notion that someone

           3    could, for example, take that information and

           4    use it for another submission and a different

           5    company is verboten, it's forbidden.  So it's

           6    sort of one of those things about where you

           7    can borrow strength legally and where you

           8    cannot.  I hope the guidance document makes

           9    that pretty clear that you need to have legal

          10    access to the data.

          11              One way to do that is for companies

          12    to share the data, but that's pretty hard

          13    particularly if the technology is fairly new.

          14    But there have been examples where they have

          15    done it.  The drug alluding stents is

          16    certainly one example, another example is

          17    heart valves that was mentioned in the first

          18    questioner where the various heart valve

          19    manufacturers got together, and saw it in

          20    their best interest to have a statistician

          21    analyze the data, and use that statistical

          22    analysis to form a guidance document.  It




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            311


           1    wasn't in a Bayesian format, but it was

           2    successful nonetheless.

           3              MR. GOODMAN:  I just want to add

           4    one footnote.  The one place where you're

           5    going to get -- that where you won't run into

           6    as much trouble here, is when the purpose of

           7    the application is to extend the indication

           8    to a new population, and the information on

           9    the other population was already out there.

          10              So when we were extending

          11    indications from the young to the old, from

          12    the old to the young, from men to women,

          13    whatever, this issue will not be as pressing,

          14    and it only makes sense that you should be

          15    able to use information from adults for

          16    example for children.

          17              It would seem crazy to have the

          18    same trial of 700 children when you just had

          19    one in adults, and clearly, the adults want

          20    to account for something.  So in cases like

          21    that you're going to have a very, very strong

          22    case for the Bayesian methods, and the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            312


           1    availability of the data.

           2              MR. MEHTA:  It just sounds to me

           3    like the advantage will be with the big --

           4              COURT REPORTER:  Use the mic.

           5              MR. MEHTA:  It just sounds to me

           6    like the advantage will be with the big guys

           7    because they'd be running a whole series of

           8    trials, and they'll get the prior -- they'll

           9    get the data from one, as the prior for the

          10    next.  And so Medtronics and others will

          11    benefit, but the smaller companies -- the

          12    smaller device companies may have a hard time

          13    having access to a good prior.  Just an

          14    observation, and it may not --

          15              MR. CARLIN:  Before you just tack

          16    on a Bayesian add here, it's important to

          17    remember that borrowing doesn't just refer to

          18    borrowing from other previous studies, it

          19    also refers to borrowing among the current

          20    dataset.  Right, you may easily have a design

          21    that involves multiple clinical centers or

          22    multiple docs reading films, but you don't




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            313


           1    want to have -- you're going to have -- you

           2    getting into a hierarchical model pretty

           3    quickly.

           4              And so the Bayesian engine allows

           5    full posterior inference for very complicated

           6    models.  You throw in some of Don's designs,

           7    and things get really hard.  I'm not sure how

           8    I'd even do it as a Frequentist.

           9              Many of us have become Bayesian not

          10    because we read De Finetti and got struck

          11    blind off a horse like Jay Kadane did, but

          12    because we just had hard models we want to

          13    fit.  Right, I mean, it got too hard to be a

          14    Frequentist.  So we're trying to blend this

          15    into a complicated regulatory environment,

          16    and there is a couple of things we need to

          17    work out.

          18              MR. ZUCKERMAN:  Okay.  But could I

          19    extend that point that you just made?

          20              MR. CARLIN:  Sure.

          21              MR. ZUCKERMAN:  Bradley, what is

          22    the regulatory environment right now?  It's




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            314


           1    usually that clinical trials are performed

           2    first in Europe, then to the U.S., and then

           3    to Japan.  And certainly one thing that we

           4    would consider, it goes back to Don Berry's

           5    point about the prospective worldwide global

           6    clinical trial development where initial

           7    European clinical data could be utilized as a

           8    prior for the U.S. and then to Japan, what

           9    have you.

          10              The problem that we frequently face

          11    is, number 1, we're not learning about the

          12    global development prospectively.  Two, our

          13    companies don't come into FDA early when they

          14    are in Europe.  So clinical trial design

          15    methodology for follow-up is often

          16    problematic.  But I think there is a

          17    potential ability to increase efficiency as

          18    companies are developing a worldwide market

          19    now.

          20              MR. CAMPBELL:  Next question.

          21              MR. ZHANG:  I'm Jerry Zhang from

          22    Becton, Dickinson.  So my comments actually




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            315


           1    enclose similar comments from Roseanne and

           2    the other panelists.  But since I've been

           3    standing here for a while, so I'm going to go

           4    ahead and make the comments.  In the

           5    beginning of the documents, it says that,

           6    "The changes often is minor so making the

           7    effects local and non-systemic."

           8              Speaking from personal experience,

           9    and this looks like a obvious assumption, but

          10    personally, I've seen that many times that

          11    obvious assumptions came back to bite us at

          12    the rear end.  So it's, you know, those

          13    obvious assumptions, when they're not true,

          14    traditionally if you run a traditional trial

          15    of course, you will let the trial go to the

          16    end, and then you take those bites with a red

          17    face and then try to find out what's wrong.

          18              But with the Bayesian -- so the

          19    question is whether you can either find out

          20    it's a wrong assumption or when can you find

          21    out it is a wrong assumption?  So in the --

          22    how much information can be contained in the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            316


           1    prior.  But the guidance doesn't really

           2    provide much of the guidance there in the

           3    details.  I mean it does have -- I mean I've

           4    heard it loud and clear throughout the day

           5    about go talk to FDA early and often, but

           6    personally, I would like to see more of

           7    details in how much information can be in the

           8    guidance -- sorry in the prior, you know,

           9    just saying, overwhelming is bad, it's not

          10    really much of a guidance.  Thank you.

          11              MR. CARLIN:  There is lots of great

          12    Bayesian textbooks you can refer to.  I'll be

          13    happy to give you one.  There is certainly a

          14    lot of stuff in Gelman's book, in my book, in

          15    O'Hagan's book and lots of others about model

          16    checking.  Bayesians do model checking.  We

          17    do model choice, we do diagnostics, we do

          18    residual analysis, we do all the things that

          19    Cook and Weisberg taught you to do in Stat

          20    100.  And so there is nothing new under the

          21    sun to being a good data analysis guy.

          22              And so no one is suggesting that




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            317


           1    you pick a prior and hug it close to your

           2    soul, and never consider anything else.  I

           3    mean you will check the prior, you will check

           4    the likelihood, you know, there is -- that

           5    this is fundamentals of Bayesian data

           6    analysis.

           7              So if Telba abused a prior on 0 to

           8    0.1 and the right answer should have been

           9    0.3, your posterior would be piling up as a

          10    point mass right at point 1.  The posterior

          11    would be screaming at you "I wish I could be

          12    over there at point 3 where you won't let me

          13    be."  And you would say, "Silly me, I didn't

          14    mean point 1."  So I mean -- I don't think

          15    anybody at FDA is going to say you must use

          16    an obviously stupid prior when you do the

          17    analysis and something stupid happens, and

          18    model checking is key -- the key --

          19              MR. ZHANG:  I -- excuse me, that's

          20    correct.  But in a lot of trials that we do

          21    we actually don't have that big a size of

          22    trial.  I mean obviously if you have




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            318


           1    thousands of patients then you will have a

           2    lot of opportunities to check these models.

           3    When you have to run a smaller size of trials

           4    then the prior plays a bigger role, as

           5    mentioned in the guidance.  So it could be a

           6    problem for us.

           7              MR. CARLIN:  Well, right now, we

           8    have this system where we're doing

           9    Frequentist evaluations on top of this.  So

          10    there is couple of -- a little bit of

          11    redundancy here.  The document says, "You

          12    can't start out with too much in your prior."

          13    And somebody also said, "Well, that's kind of

          14    relative, isn't it?"  But then if you also --

          15    if you had too much in the prior, what might

          16    happen is that it's a Type-I error, it would

          17    get out of control.  Right, when you simulate

          18    a Type-I error, you'd be making a lot of

          19    mistakes.

          20              MR. ZHANG:  Right.

          21              MR. CARLIN:  So I think these

          22    things are -- there is enough protection in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            319


           1    the document at the moment that I don't

           2    foresee a disaster happening all the time.

           3              MR. ZHANG:  Right.  So -- okay, so

           4    it comes down to the burdensome, you know,

           5    how much saving can you get, you know, so --

           6    we have to pick at issues for that one.

           7              MR. CARLIN:  It comes down to doing

           8    careful work before you do the trial, and

           9    then modeling as best you can.  But you are

          10    right, small datasets are harder than large

          11    datasets.  And I'm -- that's true for

          12    Bayesian, it's true for Frequentist.

          13              MR. ZUCKERMAN:  Okay.  If I could

          14    perhaps, address this very good practical

          15    question from a regulatory perspective, I

          16    think the question that would be asked is

          17    from the CLIN STAT Engineering perspective,

          18    which is, can the model account for effects

          19    of covariance and that are judged to be

          20    important, and usually with these small

          21    datasets, the answer is "No," or the sponsor

          22    comes in with just a point estimate, and the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            320


           1    answer is "No."

           2              So given what you've just pointed

           3    out and the difficulty that individual

           4    companies face when dealing with this

           5    situation where we don't have access to

           6    larger datasets, another paradigm that some

           7    companies utilize is again, prospectively,

           8    looking at how they're going to build

           9    generations of devices.  And with generation

          10    device I trial, they tried to do a major

          11    pivotal trial, and utilize that data for

          12    subsequent generations as priors.

          13              Now, of course, here I think the

          14    practical issue is getting our engineers

          15    inclinations to buy into the concept of minor

          16    modification as we progress along

          17    generations, which is an area that needs

          18    further development.

          19              MR. CAMPBELL:  Okay, next.

          20              MS. AJONGWEN:  I'm Patience

          21    Ajongwen, Ortho Clinical Diagnostics.  I'm

          22    going to change the tempo a bit.  I want to




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            321


           1    come from a patient perspective, and this is

           2    going to labeling.  I think somebody

           3    mentioned that before.  It's a great idea,

           4    but I'm like, okay, how is the labeling going

           5    to be simplified so that the patient can make

           6    informed decision on their care?  And I just

           7    wanted, you know, to bring that out so that

           8    somebody can -- you know, I know that

           9    somebody mentioned about the physician using

          10    the information with -- together with the

          11    patients making an informed decision on their

          12    care.

          13              It's a good idea.  It's different

          14    from the Frequentist point of view that most

          15    of the public knows how are we going to make

          16    labeling easy?  Because I don't think there's

          17    anything in the guidance about labeling so

          18    that a patient that can -- for example, I'd

          19    like to make an informed decision about my

          20    care.  I'd like to know what I'm taking, why

          21    I'm taking, anything.  How easy would that

          22    be?




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            322


           1              MR. CAMPBELL:  Okay.  Great

           2    question about labeling; the panel.

           3              MR. ZUCKERMAN:  In the examples

           4    that I'm familiar with, Greg, the indications

           5    for use are going to state what the device

           6    does on the intended target patient

           7    population such that if you do a Bayesian

           8    trial or Frequentist trial, the IFU will be

           9    interpretable.  Now, I think where the

          10    question -- this one and others have come

          11    from is how do we display results in the

          12    clinical trials section?

          13              And there I think we just indicate

          14    credible intervals and design of the trial.

          15    Roseanne, you've had experience with one of

          16    these trials.

          17              MS. WHITE:  Labeling in Bayesian is

          18    one of the great challenges in terms of how

          19    you go about labeling and in such a way that

          20    it's accessible to the patients.  And it's

          21    difficult in terms of trying to figure out

          22    what the probability is.  So often, what we




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            323


           1    do is, not only provide the Bayesian, but

           2    maybe a Frequentist s point estimate as well

           3    to go along with it, which sometimes helps.

           4              But it also then challenges why did

           5    we do Bayesian in the first place?  So we're

           6    trying to figure that out.  Right now, what

           7    we've done is indicated exactly what the

           8    hypothesis test was, and what was the

           9    probability that it was right, at least.  And

          10    that seems to have been a good way of

          11    communicating the information.

          12              MR. CAMPBELL:  Yeah.

          13              MS. AJONGWEN:  Thank you.

          14              MR. CAMPBELL:  There are examples

          15    of labels of Bayesian-approved devices, and

          16    those would be in the -- on the websites that

          17    I mentioned earlier.  But I think it's a

          18    great question.  There are people who would

          19    say that an advantage of the Bayesian

          20    approach is that it has the ability to bring

          21    the focus more to the patient perhaps than

          22    one might see in a Frequentist situation.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            324


           1              It's certainly possible to build

           2    very complicated models that take into

           3    account all the covariates, but how you

           4    report that in the label is I think quite a

           5    challenge, and I think that's what you heard

           6    from the panel here.

           7              MR. GOODMAN:  Yes.  But I think

           8    however hard it is to express the result of

           9    one of these analysis, it cannot be easier in

          10    a Frequentist framework to explain to

          11    somebody what a p-value means, or even what a

          12    confidence interval means.  I mean if you

          13    really try to -- the only reason I think they

          14    understand the Frequentist reports is because

          15    they don't understand them, so.

          16                   (Laughter)

          17              MR. CAMPBELL:  Okay.  Next.

          18              MR. BERRY:  My name is Scott Berry

          19    at Berry Consultants.  I'm a little surprised

          20    by the -- in the document itself and the

          21    discussion today so much is centered on the

          22    prior itself.  I've been involved in a number




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            325


           1    of submissions to the FDA and this is not the

           2    selling point of the Bayesian things most of

           3    the time.  Sometimes it's really nice, and

           4    it's a great feature to be able to use prior

           5    information.

           6              But the ability to be adaptive to

           7    use predictive probabilities in a

           8    time-to-event study, you've looked at 300

           9    patients, you're still accruing, and to

          10    address the probability of success at 12

          11    months which is something incredibly

          12    difficult, and doesn't even mean anything

          13    from a Frequentist perspective to the

          14    adaptive nature and to do innovative things

          15    that powering up to 500 and looking at the

          16    data when you're done, you can't do it.

          17              The question of patient advocate

          18    came up and patient rights to stop a study

          19    that's not likely to be successful, or to

          20    stop it sooner because it's very likely to be

          21    successful using Bayesian tools, treats

          22    patients in the study better, they are more




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            326


           1    likely to be contributing to science or

           2    treated more ethically, you get the therapies

           3    out to the subject sooner as well.

           4              The comment that the only advantage

           5    to Bayesian statistics is a prior I just

           6    couldn't agree more -- I couldn't disagree

           7    more.  It's -- the whole philosophy, the

           8    whole scientific nature of doing device

           9    discovery is the advantage.  The prior is

          10    great sometimes when you have it.  But if you

          11    don't have it -- and I've seen comments that

          12    if you don't have a subjective prior or other

          13    data you shouldn't use Bayesian statistics

          14    and I just couldn't disagree more.

          15              And so discussions about the

          16    adaptive nature -- maybe that wasn't meant to

          17    be discussed in the document, but the ability

          18    to be adaptive, to stop early, and to get the

          19    right sample size as opposed to the fixed

          20    sample size I think is the real selling point

          21    though.

          22              MR. D'AGOSTINO:  But I guess one of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            327


           1    the responses to it is you can do adaptive

           2    designs in a non- Bayesian context.  And I'm

           3    still not impressed by the Bayesian being

           4    better and adaptive designs than the non-

           5    Bayesian, but that may be just my lack of

           6    knowledge.  As far as the priors go, I think

           7    the issue is that that's where you begin

           8    quite often the discussion, and it creates a

           9    problem for non-Bayesians, it creates a

          10    problem for clinicians -- what are you

          11    talking about with this prior?

          12              So I don't say that it -- there are

          13    ways of getting around it and after a

          14    discussion they -- some understand this sort

          15    of non-informative and you don't have to --

          16    it's not going to win the day and so forth.

          17    But it's will you be -- where at least what

          18    I'm dealing with is, where you begin the

          19    discussion, and it just creates a confusion.

          20              So I think one of the issues when

          21    I'm talking to non-Bayesians, talking to

          22    clinical trialists who are quite good at




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            328


           1    clinical trials, and not less about the

           2    Bayesian, you hear that at the beginning.

           3    And when you're talking about some of the

           4    advisory committees -- I've had a lot of

           5    experience on advisory committees.  Many of

           6    these clinicians come in knowing a lot about

           7    the field, and when you start talking about

           8    priors they may all be saying the same thing,

           9    but it doesn't sound like they're saying the

          10    same thing.

          11              And you get a lot of fire going on,

          12    a lot of discussion going on about the prior

          13    whether it ultimately is going to drive the

          14    analysis is, in some sense, almost

          15    immaterial.  But it's where a lot of the

          16    discussions come up, or at least start.

          17              MS. WHITE:  Another reason may be

          18    the focus is so much on the prior.  What

          19    makes the device industry so useful for

          20    Bayesian is because you have quick turnover

          21    iterations, and you accrue patients very fast

          22    and you get to endpoints.  Well, there is a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            329


           1    challenge between, well, can I wait?  You

           2    know, I accrued my patients in a couple of

           3    months; my endpoint is six months off.

           4              Do I want to wait till the end of

           5    that time to figure out how many more

           6    patients or would I rather just have a fixed

           7    sample size, get it over with quickly,

           8    because it's only going to mean maybe another

           9    month or two of enrolment anyway.  So why not

          10    go ahead and just do a fixed sample size?

          11              So where the sample size reduction

          12    comes in is the ability to use the priors,

          13    because we really are not set up in a way

          14    where it's financially beneficial to really

          15    do the adaptive type of trials.

          16              MR. D'AGOSTINO:  Yeah.  I just want

          17    to support that.  In a number of trials that

          18    I'm involved in putting together, we have the

          19    debate over and over again about doing an

          20    interim analysis, and sometimes, by the time

          21    we get our first Data Safety Monitoring

          22    Committee together to look at safety, this




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            330


           1    recruitment is all done, the stents have been

           2    put in, and the patients are now rolling

           3    along.

           4              MS. WHITE:  Uh-huh.

           5              MR. BERRY:  Well, I can only -- I

           6    just -- I find that frustrating.  And I can

           7    only say that that's -- those are the type of

           8    companies that come to me and say, "We did a

           9    fixed sample size and gees, we just missed.

          10    I wish we had done something.  Can you help

          11    me now?"  And then it's too late.  And so I

          12    find the reasoning that, "Boy, this is easier

          13    to do," is disappointing.

          14              MR. D'AGOSTINO:  Yeah.  I hear what

          15    you're saying, but the study -- it's almost

          16    like you have to restart the study because

          17    you've recruited all your patients.  And when

          18    you do an interim and when you do the sample

          19    and hold off for an interim analysis, you --

          20    if it says increase your sample size it may

          21    be then I now have to get all the centers or

          22    many of the centers activated against --




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            331


           1    there is a logistic issue going on here.

           2              I'm sure that in the examples you

           3    must have in mind, the recruitment is not as

           4    fast as what we're talking about, and so the

           5    adaptive comes in.  I mean I'm a big advocate

           6    of adaptive in other settings, you know.

           7              MR. BERRY:  A play between that.

           8              MR. D'AGOSTINO:  Yeah.

           9              MR. BERRY:  If you had created a

          10    rate for stent, you can't do anything.

          11              MR. CAMPBELL:  I think Scott what

          12    you're saying is that there are different

          13    ways to gain efficiency.  One is by saying

          14    there's information available upfront.  And

          15    then the other is to use the information from

          16    the trial when it's available, when the

          17    timeframe works out more efficiently than we

          18    tend to now.

          19              Is that a fair summary so that the

          20    savings even in sample size and cost and

          21    ethics doesn't necessarily require upfront

          22    information if you're smart about how to use




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            332


           1    the internal information if it's available?

           2              MR. D'AGOSTINO:  Scott.

           3              MR. GOODMAN:  I agree.

           4              MS. WHITE:  I think what all of

           5    this is sort of pointing to is the need for

           6    more examples, examples of what you're

           7    talking about, where it's -- where you can do

           8    the adoptive in cases where you can't.  I

           9    really think what would be useful is if we

          10    could have a series of white papers maybe to

          11    follow up the guidance document where we look

          12    at examples maybe in invasive device, a

          13    combination product, and in vitro diagnostic

          14    where you can look at these potential

          15    situations where you do fixed sample size

          16    versus adaptive in Bayesian.

          17              And I think that would help a lot

          18    to help educate everybody so that we can take

          19    advantage of some of the things that you're

          20    talking about.

          21              MR. CAMPBELL:  Great.  Barb.

          22              MS. BUCH:  I think he was waiting




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            333


           1    longer.

           2              MR. CAMPBELL:  Oh.  Okay.

           3              MR. LEWIS:  Roger Lewis,

           4    Harbor-UCLA.  And unfortunately, I'm

           5    following Scott right after he said that

           6    priors aren't that important, and I have a

           7    comment about priors.  So I'm going to make

           8    two very quick comments.  The first bring --

           9    coming back to the question of the situation

          10    in which proprietary data is owned by one

          11    company that would provide a useful prior for

          12    another company.

          13              In that setting, if that prior

          14    allows for reduction in the expected sample

          15    size, then there is a natural metric for the

          16    value of that proprietary data, because the

          17    company B planning a trial certainly knows or

          18    has a projection of how much it would cost

          19    them to independently acquire that much of

          20    digital information.

          21              So I'd just like to suggest that

          22    this doesn't necessarily need to be sharing,




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            334


           1    but if there's an actual metric for the

           2    value, and I would expect that information to

           3    be purchased.  And the difference between

           4    purchasing it and doing that many more

           5    patients as you purchase it and you have it

           6    immediately, and that does lead to an

           7    efficiency if you can agree on the price.

           8    Obviously, I'm not an economist, and I don't

           9    work in the industry.

          10                   (Laughter)

          11              MR. LEWIS:  My second comment; this

          12    has to do with labeling -- couple of places

          13    where they talk about labeling and the

          14    communication of the information to the

          15    patient.  In my role as a practicing

          16    physician, it is my impression that labeling

          17    has no effect really on the decisions made by

          18    patients, and that its primary purpose --

          19    maybe dangerous to say this here, is to

          20    provide a fence around which the sales force

          21    can form their arguments.

          22              So I would state that the important




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            335


           1    audience for the labeling information is

           2    really the physicians to whom the sales force

           3    will be devoting most of their attention.

           4              MR. CAMPBELL:  Well, I'm going to

           5    touch only the first one and not the second

           6    one, and say that early on in the -- when we

           7    first got started Don Malec we borrowed from

           8    NCHS who is a Bayesian statistician, and he

           9    and I worked together.  And one of the things

          10    we talked about was basically this notion in

          11    a hierarchical modeling setting of using

          12    effective sample size to characterize this --

          13    the amount of information that you have in

          14    the prior in terms of number of patients.

          15              And so I think that's an

          16    interesting way, Roger, that you frame it.

          17    And there are probably monetary implications

          18    associated with that.  Other comments?

          19              MS. WHITE:  I sort of have the

          20    experience of the monetary implications.  In

          21    the case of Guidant, it was $80 per share in

          22    which Boston Scientific and Abbott Vascular




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            336


           1    bought us somewhat for our data.

           2              MR. GOODMAN:  On this issue of how

           3    to express things, I think it's really

           4    critical.  We've talked a lot today about how

           5    complex things are.  There's -- actually many

           6    things can be reduced to as you say number of

           7    patients and a lot of what patients do is

           8    equivalent to meta-analysis.

           9              So -- and these are things that

          10    folks who don't know anything about patient

          11    methods do have some intuition about the

          12    strengths and weaknesses.  I think it's very

          13    important that we demystify a lot of what the

          14    technology is all about, and emphasize the

          15    commonality, and teach how a lot of the

          16    judgments that they're very familiar with

          17    making in the Frequentist realm are really

          18    identical to the kinds of judgments we're

          19    making in the Bayesian realm just with a

          20    somewhat different language and a different

          21    technology.

          22              And I think, ultimately, it's that




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            337


           1    demystification and not the emphasis on the

           2    complex models, even though they are often

           3    necessary to get some of the advantages that

           4    will make people feel comfortable with it.

           5              MS. WHITE:  I think one of the

           6    other issues in terms of the labeling is your

           7    primary endpoint is Bayesian, do you make

           8    your secondary endpoints which is also in the

           9    labeling also Bayesian?  And I think what

          10    gets to be confusing in the labeling and this

          11    has been our experience is where the primary

          12    endpoint is reported as a Bayesian endpoint.

          13              And then you see all the secondary

          14    endpoints with their traditional confidence

          15    intervals, and then nobody can relate the

          16    two.  So part of the demystification is

          17    providing some consistency on how we do the

          18    labeling for both the primary and the

          19    secondary endpoints on the labels.

          20              MR. CAMPBELL:  Good point.  Barb,

          21    your turn.

          22              MS. BUCH:  I agree with that last




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            338


           1    point, by the way.

           2              MR. CAMPBELL:  Barbara Buch.

           3              MS. BUCH:  Barbara Buch with FDA.

           4    One of the things I want to point out at this

           5    meeting is that we've had a lot of really

           6    good academic discussion about how to define

           7    Bayesian as an approach.  And I just want to

           8    take this to the people that are writing this

           9    document as well as others in the room that

          10    at the end of the day this document is really

          11    being used in a regulatory sense.

          12              And we have to keep that in mind.

          13    And a lot of the academic discussions are

          14    very interesting, and they promote and make

          15    -- bring these things to light and bring

          16    these things forward.  But at the end of the

          17    day, we do have regulatory restraints that

          18    are involved in this particular guidance.

          19              And so a lot of the very emotional

          20    discussions that have come forth are valid,

          21    but they may not fit into the framework that

          22    we are restricted by.  And so I just want




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            339


           1    everybody in the room to keep that in mind is

           2    that the reason we're working here is to put

           3    something that's maybe a little bit more

           4    progressive out there.

           5              But we do realize that we have

           6    these restraints, we do have to follow things

           7    in the order of the law.  And so everything

           8    that you've said is being heard, but you also

           9    have to think of that as a caveat.  And then

          10    I also want to say that in labeling, one of

          11    the -- we do have resources that help us with

          12    these things, and there is a whole set of

          13    other guidelines for labeling.

          14              And I think the important message

          15    has come out today is the clarity of what the

          16    terminology is that we're using the

          17    vocabulary has to be understood by anybody

          18    who reads that.  And I think that's a point

          19    that ought to be put across to the authors of

          20    the guidance.

          21              MR. CAMPBELL:  Great.  Thank you.

          22    Yes.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            340


           1              MR. ZHENG:  Hongjie Zheng from

           2    Forest Research Institute.  I'm sorry for --

           3    to be the last person to ask question, but my

           4    question will be very short.  The question is

           5    about why are you doing the Type-I error

           6    assessment while you are using a Bayesian

           7    approach?  I think the answer is important in

           8    the public health's question, which is if the

           9    device doesn't work then what's the chance

          10    that you will approve that?

          11              I think this question can be

          12    answered perfectly by the Frequentist

          13    framework, because there your so-called

          14    intervention effect, trimming effect is

          15    fixed, so you can fix that at zero that means

          16    there is no effect then what's the chance you

          17    will recheck your new hypothesis?

          18              So in Bayesian framework the

          19    trimming effect is no longer a fixed

          20    quantity.  It's a random variable.  So you

          21    make your decision based on the posterior

          22    probability of, for example, is theta, which




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            341


           1    is, then zero; if that probability is more

           2    than 95 percent, you are (off mike) the

           3    device.

           4              To me that means, that the

           5    probability of theta less than or equal to

           6    zero means that the device has no work.

           7    That's only 5 percent.  So in a sense, you

           8    already contribute a Type-I error, even

           9    though it is not exactly a Type-I error, it's

          10    like a (off mike) error.

          11              Am I missing something here?  Why

          12    people, you know, are doing all the

          13    Frequentist assessment for Bayesian approach?

          14              MR. CARLIN:  This seems to be the

          15    big-ticket item here today.  Don Rubin wrote

          16    a paper in 1984 talking about

          17    Bayesianly-justifiable frequency

          18    calculations.  Bayesians don't view

          19    Frequentist evaluation of Bayesian procedures

          20    as inherently evil.  I hope you haven't

          21    gotten that impression from me or anybody

          22    else.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            342


           1              It's perfectly all right to

           2    evaluate the Frequentist behavior of a MLE or

           3    a confidence interval or a Bayesian posterior

           4    or any procedure you want to write down.  And

           5    so that's fine.  The question is only how

           6    much of that do you want to drive the

           7    analysis?  If you're a Frequentist, you would

           8    think that that's sort of the most important

           9    thing.

          10              And in fact you're willing to use

          11    p-values to make decisions even though as

          12    Steve points out, nobody understands what a

          13    p-value is.  And most people think a p- value

          14    is the chance to know all this is true.

          15    After all that's why we are rejecting the

          16    null, right, because the p-value is so small.

          17    Well, no that's not what a p-value is

          18    exactly.  And by the time you get down

          19    explaining it, the doctor that you are

          20    talking too is fast asleep and you say,

          21    "Yeah, that's sort of what it is."

          22                   (Laughter)




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            343


           1              MR. CARLIN:  So, you know, you just

           2    can't explain it.  So what a Bayesian would

           3    say I think is we don't really wan to make

           4    decision that way.  We may want to think of,

           5    you know, Type-I error or something like that

           6    as being a secondary characteristic.  A good

           7    characteristic of a Bayesian procedure would

           8    be that it has a low Type-I error.  That will

           9    be good thing if it did that.

          10              And FDA is a little more enthused

          11    about that than I would like them to be or

          12    maybe Steve or Don would like them to be.

          13    But again I think it's all to the good.  It's

          14    fine to evaluate these things.  And again the

          15    only thing I was hoping to add to the mix was

          16    to actually use the prior in that calculation

          17    too to try to sample values that are

          18    consistent with what you think is really

          19    going to happen rather than fixing theta

          20    equals zero or theta equals 1.5.  After all,

          21    that's just a guess.

          22              And you've got to guess, remember




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            344


           1    it's called the prior.  So sample from that

           2    and then see exactly what's the chance that

           3    you are going win the game.  You can simulate

           4    the probability that theta is going to be

           5    bigger than zero at the end of the trial.

           6    And so, now you are simulating I guess what

           7    I'd call Bayesian power.

           8              You are simulating exactly the

           9    probability that this trial is likely to end

          10    in success.  That's after all -- shouldn't

          11    that be the main thing you care about?  So

          12    what's the chance we are going to win?

          13    Sorry, but that's my take in it.  I am sure

          14    the panelists have different views.

          15              MR. CAMPBELL:  Let me just remind

          16    everybody although there's been a lot of

          17    discussion, the frequentist evaluation is at

          18    the planning stage.  I mean, we are trying to

          19    understand the frequentist properties of the

          20    Bayesian trial.  When the data comes in and

          21    the analysis is done, the analysis is

          22    Bayesian.  It's -- the -- no one cares




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            345


           1    anymore about what -- nobody care much

           2    anymore about what the frequentist power is.

           3    It's -- the emphasis now is on the posterior

           4    distribution, and --

           5              MR. GOODMAN:  Yeah, but the problem

           6    is that you can have two different monitoring

           7    plans with two different roots --

           8              MR. CAMPBELL:  Absolutely.

           9              MR. GOODMAN:  With the same

          10    posterior.

          11              MR. CAMPBELL:  That's right.

          12              MR. GOODMAN:  And that's where I

          13    was saying even in the planning, there is a

          14    little bit of schizophrenia in the document.

          15    It says how you are going to choose between

          16    those two designs.  The Bayesian and the

          17    Bayesian sphere of the document says it

          18    shouldn't make a difference.

          19              MR. CAMPBELL:  Right.

          20              MR. GOODMAN:  And yet you're saying

          21    we'll measure these, report them, and then

          22    well how is that number going to be used in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            346


           1    the, you know, initial exploratory

           2    discussions, I am still little unclear on

           3    that.

           4              MR. CARLIN:  If -- can I just give

           5    it -- try this one more time?

           6              MR. CAMPBELL:  Okay.

           7              MR. CARLIN:  So the bottom line for

           8    me is if the data -- you said that when the

           9    data comes in and whatever it happens we are

          10    going to be Bayesian about it, and that's

          11    going to be the answer.  What happens if when

          12    if the data comes in, I didn't use exactly

          13    the monitoring plan that I said I was going

          14    to use?  Are you going to have a problem with

          15    that?

          16              How far can I get if I peek at 45

          17    and 75 and 115, instead of 30, 60 and 90?

          18    Are you going to view my analysis with any

          19    sort of discomfort, because I didn't do

          20    exactly what my little calculation did or --

          21              MR. CAMPBELL:  Your point is well

          22    taken, and we'll see what we can do.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            347


           1                   (Laughter)

           2              MR. CARLIN:  If I were you, sir,

           3    sitting there I could smell, I'd want to

           4    know.

           5              MR. CAMPBELL:  Yeah.

           6              MR. CARLIN:  Are they going to have

           7    a problem with this, because it's never going

           8    to come out exactly like I said it was going

           9    to, you know.  It was just a simulation after

          10    all, so --

          11              MR. MEHTA:  But the frequentist can

          12    solve that problem through a special --

          13              SPEAKER:  Use the microphone,

          14    please.

          15              MR. MEHTA:  Oh yeah.

          16              MR. CAMPBELL:  He said that the

          17    frequentist cannot solve that problem.

          18              SPEAKER:  That's not what he said.

          19                   (Laughter)

          20              MR. CAMPBELL:  The virtues of being

          21    near a microphone, Cyrus.

          22              MR. CARLIN:  You want to give a




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            348


           1    lecture on land of the maths right now, or --

           2    go ahead.

           3              MR. BERRY:  Actually I did want to

           4    address the same point and I was going to

           5    make the point that Greg did to try to

           6    resolve what I am going to now call Goodman's

           7    Conundrum.  And when we do the operating

           8    characteristic, it's at the planning stage,

           9    it's just that power, power is a pre-notion

          10    of precision.  It's before you get the data.

          11              After you get the data, the data

          12    should speak for themselves.  So that sort of

          13    philosophy would mean at the end of the day,

          14    you present the data, the posterior

          15    distribution, but then you are a little --

          16    you are still stuck with the Goodman's

          17    Conundrum.

          18              For -- let me give you an example,

          19    you've designed something.  So you are

          20    looking at the data, and you are doing some

          21    things, and your criterion for success is

          22    that you've got a posterior probability of




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            349


           1    efficacy of 98 percent say.  That's your

           2    criterion throughout the trial.  And at the

           3    end of the day, you end up with 96 percent.

           4              Okay.  So there is -- all of these

           5    trials have upper bounds in the sample size.

           6    So you end up at the end of the day, 96

           7    percent.  So you present to the FDA and the

           8    panel, if it's a panel, "this is our

           9    posterior distribution, the posterior

          10    probability of efficacy is 96 percent."  Our

          11    criterion for success was 98 percent.  But

          12    this is what the data showed.

          13              Likelihood principle interpreted by

          14    many Bayesians would be 96 percent is the

          15    answer.  I submit that in that circumstance

          16    what you do is, you know, you tell everything

          17    that you did you present the data, you talk

          18    about the efficacy issues, you talk about the

          19    safety issues, you talk about, you know, is

          20    this an approvable device under the

          21    circumstances.

          22              Scott might say had you the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            350


           1    opportunity to look at the data one more time

           2    and to extend it, you might have done that,

           3    but you didn't, this was the end of the

           4    trial.  Steve, would this partially resolve

           5    your conundrum?  Separating out the design

           6    and the analysis points and then you present

           7    what the data are consistent with the

           8    likelihood principle.

           9              MR. GOODMAN:  I am not -- I mean, I

          10    have no problem with what you just described.

          11              MR. BERRY:  What would you do if

          12    you had a design like that and the data

          13    presented you -- you are on a panel, the data

          14    presented to you.  My criterion was 98

          15    percent. I ended up with 96 percent.  Here

          16    are the data.  What would you do?

          17              MR. GOODMAN:  Yeah, but the issue

          18    is, of course, why have that criterion in the

          19    first place.  It might have just been out of

          20    convenience.

          21              MR. BERRY:  You had that criterion

          22    so that you could get a false positive rate




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            351


           1    of 5 percent.

           2              MR. GOODMAN:  Oh, okay.  Yeah,

           3    okay, I didn't understand what the --

           4              MR. BERRY:  I am sorry.  I missed

           5    the critical point.

           6              MR. GOODMAN:  So if the criterion

           7    was based on some utility calculation or

           8    whatever --

           9              MR. BERRY:  No.

          10              MR. GOODMAN:  -- then, you know,

          11    the 96 percent might well justify saying it's

          12    not good enough.  But if it was solely based

          13    on a frequentist, you know, Type-I error

          14    issue, and 96 percent is good enough

          15    considering the -- you know, the losses of a

          16    false positive and a false negative, then I

          17    mean I think you just conclude on the basis

          18    of the 96 percent, not the pre-specification.

          19    But that's --

          20              MR. BERRY:  That's -- I think

          21    that's consistent with what Greg was

          22    suggesting.




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            352


           1              MR. CAMPBELL:  Okay.  I think we

           2    have time for one more question.

           3              MR. BREITER:  So, I am the last.

           4    Okay.  I am David Breiter with Boston,

           5    Scientific CRM.  First a comment, I want to

           6    commend the FDA for this guidance document.

           7    I think it was stated earlier that as

           8    industry statistician and sponsor

           9    statisticians one of the great difficulties

          10    we have is education internally of the folks

          11    who ultimately are making the decisions as to

          12    whether to move forward with the trial or

          13    not.

          14              And the language of this document,

          15    I think, is going to be helpful, or the

          16    language that it's written in, in terms of

          17    the lay language is going to be helpful in

          18    that regard.  My question or questions, I

          19    should say, are more of logistical questions.

          20              Section 3.9 of the document talks

          21    about if there are changes -- I may read it

          22    here, "A change in prior information or the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            353


           1    model may jeopardize or impair the scientific

           2    validity of the trail.  For this reason,

           3    formal agreement probably would be a good

           4    idea."

           5              As sponsors or as industry

           6    statisticians up to this point, we probably

           7    figured that if I am going on with a

           8    non-informative prior, I don't need to think

           9    about having an agreement.  But should we be

          10    thinking about having an agreement to guard

          11    against the possibility of pessimistic prior

          12    based on changing information being I would

          13    say thrust upon us say at a later point.

          14              So that's first part of my

          15    question.  Second part, I am going to

          16    apologize ahead of time if my memory isn't

          17    serving me correctly, but I believe in the

          18    FDAMA regulations under the agreement

          19    meetings states that the agreement meeting

          20    it's -- or the agreements itself can be

          21    amended if there is a significant change to

          22    the scientific information that's out there




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            354


           1    about a therapy or a device.

           2              So my question would be -- and I

           3    don't think that gave anyone heartburn

           4    before, because from a frequentist

           5    perspective there is really, you know, sort

           6    of abstract as to how you might even deal

           7    with that.  There is no formal approach as to

           8    how you would incorporate new scientific

           9    information from a trial.

          10              Now with Bayesian, there is a

          11    formal approach that prior could be changed.

          12    Does the Bayesian trial design require its

          13    own specific language around agreement

          14    meetings as to when it's possible they might

          15    be broken, and or there might be changes

          16    required and how those changes might be

          17    instituted?

          18              MR. CAMPBELL:  Okay.  That's a very

          19    regulatory question, Bram.  So it's you and

          20    me.  So how about you?

          21              MR. ZUCKERMAN:  Okay.  This might

          22    be simplistic.  But I wouldn't be so




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            355


           1    pessimistic upfront.  If the Agency gives a

           2    conditional approval to your IDE with a

           3    Bayesian design and impart three or section

           4    three of the letter, there is no future

           5    concern underlining your point about the

           6    choice of the prior.

           7              I think that we make every effort

           8    to honor the original design, and you have to

           9    appreciate that.  Again we are very fortunate

          10    at the Agency to have excellent Bayesian

          11    statisticians who are extremely interested in

          12    this area, and in developing a proper trial

          13    design upfront.

          14              So I do think even without an

          15    agreement letter, if you read the IDE -- in

          16    agreement meeting, if you read the IDE letter

          17    carefully, there shouldn't be that much cause

          18    for concern.  Now, you've, however, pointed

          19    out there's always the worst case scenario.

          20    Murphy's Law is true or what have you.

          21              And the Agency does take these

          22    factors into effect regardless of whether




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            356


           1    there's an agreement meeting upfront, and we

           2    try to act fairly, and I think that Greg is

           3    quite sensitive to these sorts of problems in

           4    particular.  Greg?

           5              MR. CAMPBELL:  Yeah, let me just

           6    give a scenario where I think this could come

           7    up.  For example, suppose there was an

           8    agreement that these were the studies for

           9    which a hierarchical model would be build.

          10    And then a new study comes out that maybe has

          11    a different spin, or maybe it's very

          12    different, and you say, "Well, you know, is

          13    the Agency going to force me now to use this

          14    study and include it in the hierarchical

          15    model?"

          16              With an agreement meeting, I can

          17    assure you that would not happen.  I mean the

          18    agreement -- the FDA will stand by the

          19    agreement.  The fact that there is new

          20    information may be part of the discussion

          21    that might occur if it were to come to an

          22    advisory committee panel, but the FDA would




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            357


           1    follow through on its agreement.

           2              As to whether you should have an

           3    agreement for non-informative prior or not, I

           4    guess I'd encourage companies to come and

           5    talk to the FDA, and figure out if they think

           6    that it will be worth that effort or not.  In

           7    many cases, it may not.  Okay.  Please join

           8    me in thanking the panel for an interesting

           9    time.

          10                   (Applause)

          11              MR. CAMPBELL:  What we are going to

          12    do is -- please don't go, we have just -- we

          13    are going to wrap-up very quickly.  So in

          14    terms of the wrap-up, I just wanted to make a

          15    few comments, and then call on Gary Grey

          16    who's been taking notes this afternoon to try

          17    and summarize for everybody what some of the

          18    comments are.

          19              What I want to emphasize is that

          20    this is a great opportunity to find out about

          21    the document today, but that it's really

          22    important that you think about making an




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            358


           1    official comment to the docket.  And the

           2    docket number is 2006D-0191, and on every

           3    page of your notes, you know how to do that.

           4              So, I want to thank all our

           5    panelist who've come today for what I think

           6    has been a very interesting series of panel

           7    discussions, and for people who traveled far

           8    and wide.  I also want to thank all the

           9    people who've helped this meeting happen,

          10    including Cindy Garrison, Bridgette Stein who

          11    were out front-manning the tables, including

          12    all the statisticians who made presentations.

          13    There were other statisticians who helped in

          14    planning this meeting, Kay Barrick, the

          15    Deputy Director of the Division, and Yunling

          16    Xu, and many others as well.  But I'd like at

          17    this point to ask Gary Gray to come forward

          18    and give us a little recounting of what we

          19    thought we heard in terms of messages.

          20              MR. GRAY:  Thanks Greg.  Like the

          21    mic's on.  Okay, I am going to try.  It's

          22    4:50.  So we have maybe 10 minutes to do




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            359


           1    this, and I think I can cover it.  As we were

           2    finishing up our discussion here today, we

           3    put our heads together down here and talked

           4    about what we thought we heard, and we came

           5    up with a few, I think generic kind of

           6    issues, or topics that we might use to

           7    summarize the discussion today.

           8              It turns out, of course, that a lot

           9    of them don't pertain directly to our

          10    document.  They are more like issues of the

          11    Bayesian initiative in general, or how to use

          12    Bayesian statistics in medical device

          13    clinical trials.  The first one was pretty

          14    much centered on education.  We need to

          15    educate the FDA reviewers.

          16              We need to educate the companies.

          17    We need to educate advisory committee

          18    members.  Everyone has to have a pretty firm

          19    understanding of the -- a common

          20    understanding of the kinds of -- of the terms

          21    we are using and what they mean to be able to

          22    accept the use of Bayesian statistics in




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            360


           1    medical device clinical trails.

           2              We heard a little bit about

           3    staffing.  This is sort of related.  Where

           4    all Bayesian statisticians going to come

           5    from?  I can assure you that we are all

           6    looking hard for them ourselves.  But the FDA

           7    and companies are going to -- if we are going

           8    to use these techniques, we need to have this

           9    staffing to do so.

          10              There was a little bit of

          11    discussion about priors in terms of practical

          12    issues about how to share data, how do

          13    companies get together, how can companies get

          14    together to combine data to create useful

          15    priors maybe using the European data or data

          16    from previous generations of the device in a

          17    planned way through the product lifecycle.

          18              And then there was even some

          19    mention about maybe creating what you might

          20    call a marketplace for priors.  And then in

          21    terms of the document itself, we heard a lot

          22    of discussion that maybe we need to have more




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            361


           1    about what works and what doesn't.  How do

           2    people decide when and how to use different

           3    Bayesian methodologies, how should they think

           4    about potentially using an adaptive design

           5    versus a fixed sample size, that kind of

           6    issue.

           7              Along with that, there were some

           8    notions and some good ideas about creating,

           9    for example, a library some sort of a

          10    resource repository where people could go and

          11    look and see some worked examples of Bayesian

          12    trials in this area, and maybe even making

          13    code available that people could use.

          14              I think that one of the things that

          15    make BUGS as successful is that the example,

          16    there is this giant half the manual that's

          17    just examples, and that kind of a set of

          18    examples that were useful in medical device

          19    clinical trails could prove useful.

          20              Finally, there were a whole bunch

          21    of -- I don't know how to describe them

          22    expect for, we need more details in the




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            362


           1    document about all kinds of issues, how we

           2    should evaluate priors, issues of maybe

           3    making it more clear how to judge levels of

           4    evidence, why -- how to -- what kind of

           5    software resources are available, how we are

           6    going to validate that, all that kind of

           7    stuff.  And I am hoping that most of that

           8    stuff or all that stuff gets covered in your

           9    comments to the dockets.

          10              And finally the last things that we

          11    heard time and time again is may be it's time

          12    to let go.  By that I mean let go of that old

          13    paradigm and then start to move forward, to

          14    try to resolve the inconsistency and

          15    insisting on frequentist evaluation of

          16    Bayesian designs.  And somehow I think that's

          17    an issue that's quite a bit larger than this

          18    document may be.  But I am sure those kinds

          19    of discussion are going to have to go in when

          20    we come to the next version.

          21              So with that, I'd like to thank you

          22    all for coming.  Thanks to the panel members




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382






                                                            363


           1    for being here, and we look forward to your

           2    comments on the docket.  Thank you.

           3                   (Applause)

           4                   (Whereupon, at 4:45 p.m., the

           5                   PROCEEDINGS were adjourned.)

           6                       *  *  *  *  *

           7

           8

           9

          10

          11

          12

          13

          14

          15

          16

          17

          18

          19

          20

          21

          22




                                BETA COURT REPORTING
                                www.betareporting.com
                          (202) 464-2400     800-522-2382