!ill
                       .   III
                        *EOO I I(
                            i@1116 iii(ilill






       Proceedings


                                                    REGIONAL
                                                    MEDICAL
                                                    PROGRAMS


            National Conference and Workshop
                           on EVALUATION,, 1970








                        U.S. DEPARTMENT of HEALTH, EDUCATION, AND WELFARE
                                            Public Health Service
                                Health Services and Mental Health Administration
                                       Regional Medical Programs Service







                    Regional Medical Programs
       National Conference and Workshop on Evaluation




           Chicago, Illinois - September 28-30, 1970

















                     Center for Continuing Education
                          University of Chicago









                                                                 PREFACE


                              The Regional Medical Programs Conference and Workshop on Evaluation, h
                                                                                                                   eld   in
                        September  1970,  marks  the  first  time  that  coordinators  and  staff  members  from   all   55
                        Regional  Medical  Programs  met  to  exchange  views  on  evaluation  and  to  assess  their   own
                        activities and programs.
                              A  number  of  factors  and  circumstances  prompted  the  idea   of   such   a   conference-
                        workshop. Most of the  55  Programs  were  at  least  three  or  even  four  years  o  .  t  was  a
                        natural time for stocktaking. Changes had been slowly taking place within                   rograms
                        and were subtly emerging; goals and objectives, and means and methods              for  achiq,  ing
                        these  ends,  were  being  examined;  and  national  priorities  and  budgetary  restrictions  were
                        leading  the  Congress  and  the  Administration  to  scrutinize  federal  programs  more   closely
                        than ever. This current of events emphasized the need for greater self-assessment.
                              The impetus  for  the  Conference  lay  largely  in  the  Regions  themselves,  and  most  of
                        the  Conference  planning  and  development  was  undertaken  by   the   Regions.   Moreover,   the
                        content of  presentations  and  discussions  were  drawn  directly  from  the  evaluative  work  of
                        the Regions. This fact illustrates  more  clearly  than  anything  else  the  considerable  strides
                        that Regional Medical Programs have made  in  the  past  several  years  -  not  only  in  building
                        up their evaluation capability, but also in putting it to good use.
                              The  Conference  was  significant  in  its  purpose,  development  and   content.   Some   of
                        the issues posed were broad and  generic  to  the  program  itself,  such  as  is  "change"  really
                        the mandate? Others  were  more  specific  to  evaluation,  e.g.,  how  much  should  be  spent  on
                        evaluation?  Still  others  were  directed  to  specific  aspects  of  the  Regional  Medical  Pro-
                                   at is the Regional Advisory Group's role in evaluation?
                        grams: Wh
                              if there was a  central  issue  posed  by  the  Conference-Workshops,  it  must,  I  believe,
                        have  been  capsulated  by  Dr.  Donald  Schon's  presentation.  If  the  whole  Regional   Medical
                        Program is greater than the sum of its parts, those  specific  activities  supported  by  it  -  as
                        its  proponents  have  long  argued  -  then  the  total  program  must  be  a  primary  object  of
                        evaluation or assessment.
                              The  Conference-Workshops  provided  few  solutions  to  the  great  gamut  of   issues   and
                        problems  that  were  raised.  It  did,  however,  make  more  explicit  than  ever  before   those
                        questions that had to be  answered.  That  in  itself  is  a  considerable  accomplishment  and  an
                        auspicious beginning.
                              Any measure of  the  relative  success  (or  failure)  of  a  conference  such  as  this  one
                        must of course be deferred. Its major impact, its final contr      ibution,  will  only  emerge  in
                        the actions and changes which will follow.
                              I  hope  these  "Proceedings"  will  be  useful  to  those  many  persons  who  are  concerned
                        with, and who will carry  out  and  evaluate,  the  Regional  Medical  Programs,  their  activities
                        and their efforts. This volume itself provides a fair index of the  range  of  both  the  interests
                        and work of the Regional Medical Programs to date.





                                                                                HAROLD MARGULIES, M.D.
                                                                                Acting Director
                                                                                Regional Medical Programs Service


                                                                       iii







                                                         CONTENTS


                                                                                                          Page

                    An Approach to Evaluation for The Regional Medical Program           .............      I
                          Donald A. Schon
                    How Others See Regional Medical Programs and Evaluation
                          Alexander M. Schmidt                    ....................................    21
                          Peter Fox   ...........................................                         23
                          Richard S. Wilbur                      .....................................    25
                          John M. Blamphin                       .....................................    27
                    Luncheon Remarks                          ........................................    31
                          Harold Margulies
                    HSMHA - An Instrument for Improvement in Health Services              ............    35
                          Vernon    Wilson


                                                        Workshop Sessions


                    WORKSHOP ON DATA
                          Participants   .........................................                        41
                          The Values and Limitations of National Data              ...................    41
                             Charles A. Metzner
                          Data for Ambulatory Care Planning                  .........................    44
                             J. William Gavett
                          Information Systems to Meet Common Data Needs of Health Agencies                46
                             Katherine G. Bauer
                          The  Northern  New  England  Regional  Medical   Program   Health   Planning
                          Data Base   ....................................                 ;    ......    51
                             John E. Wennberg


                    WORKSHOP ON MEASURING CHANGES IN BEHAVIOR
                          Participants   .........................................                        55
                          Measuring Changes in Knowledge                   ...........................    55
                             William R. Crawford
                          Measuring Changes in Clinical Performance             ......................    55
                             Barbara J. Andrew


                    WORKSHOP ON THE EVALUATION OF CHANGING HEALTH STATUS
                          Participants   .........................................                        61
                          Remarks by Moderator                      ..................................    61
                             Robert R. Carpenter


                                                                v







                                                                                                               Page

                                 The Value of Health Status Measures                                           62
                                    Sam Shapiro                      ........................................... I .
                                    Maureen M. Henderson                        ................................ 67
                                    Discussion                          ........................................ 72

                                 How to Measure Health Status
                                    Howard R. Kelman    ..........                     ......................... 78
                                    Discussion                          ........................................ 83
                                 The Relation of Process and End-Result Evaluation                             85
                                    Charles E. Lewis  ...................................
                                    Discussion  ................................                 I........     90

                          WORKSHOP ON PROGRAM EVALUATION
                                                                                                               -99
                                 Participants                         ..........................................
                                                                                                               99
                                 Approaches to Program Evaluation                     ..........................
                                    Harold W. Keairnes
                                                                                                               104
                                 Program Evaluation Workshop - A Case Study                    .................
                                    Harold W. Keairnes


                          WORKSHOP ON RESOURCE ALLOCATION/ECONOMICS
                                 Participants ............................................                     109
                                 Cost-Benefit and Cost-Effectiveness Analyses in the Health Field         ......I
                                    John Glasgow
                                 Role of Social and Behavioral Scientists in RMP Evaluation    .......         I116
                                    Michael Zubkoff
                                 The Application of Economic Analysis to Regional Medical Programs             1 17
                                    James R. Jeffers
                                 Summary of Remarks                         .................................... 122
                                    John E. Wennberg
                                 Social Scientists and the Process of Evaluation             ................... 122
                                    Conrad Seipp
                                 Cost  Benefit  -  Cost  Effectiveness  Studies,  and   Their   Application   to
                                 Allocation of Resources                       ................................. 126
                                    Robert L. Berg
                                 Accountability  and  Decision-Making  in  the  Iowa  Regional  Medical   Pro-.,
                                                                                                 .........     130
                                 gram   .....................................
                                    Charles W. Caldwell
                                 Resource Allocation and the Evaluation Process                ................. 141
                                    Charles L. Joiner







                                                                       vi







                                                                                                          Page

                                                    Special Interest Meetings

                                                                 S RESEARCH
                   STATISTICAL      MODELS      AND      OPERATION
                         Participants                             ......................................... 147
                         A 'Weighted Aggregate' Approach to R & D Selection                   ............. 147
                            David H.  Gustafson,  Pai  and  Kramer
                         Comments on an Evaluation Model for the Regional Medical Program               ... 157
                            Vernon E. Weckwerth
                         On Evaluation: A Tool or a Tyranny      .........................                159
                            Vernon E. Weckwerth


                    EVALUATION OF CORONARY CARE TRAINING
                          Participants                            ......................................... 167
                          Evaluation  of  Coronary  Care  Training:  Some  Direct  Observations   of   Per-
                          formance in Hospital Practice                       ............................. 167
                             Rodger Shepherd
                          Report on Xerox Study of Eleven 'National Coronary tare            -      Training
                          Centers                            .............................................. 167
                             Daniele Deverin
                          Evaluation  of  Coronary  Care   Unit   Nurse   Education   in   Washington   and
                          Alaska   ........................................                   I       ..... 170
                             Mariella Larter
                     A SYSTEMS APPROACH TO CORONARY CARE EVALUATION
                           Participants                           ......................................... 175
                           A Study of Coronary Care Unit Effectiveness                  ................... 175
                              M. A. Rockwell
                     EVALUATION OF INSTRUCTIONAL TECHNOLOGY PROJECTS
                           Participants                         ........................................... 177
                           Summary of Session                         ..................................... 177
                              Cecelia Conrath


                     EVALUATION OF PHYSICIAN EDUCATION
                           Participants                           ......................................... 179
                            Summary of Session                        ..................................... 179
                               William B. Munier


                      EVALUATION OF MULTIPHASIC SCREENING
                            Participants                         .......................................... 181

                                                                 via






                                                                                                                  page
                                    Evaluation of Multiphasic Health Testing                 ...................... @ 181
                                       Donald N. Logsdon

                              EVALUATION OF STROKE - REHABILITATION PROJECTS
                                    Participants                          ......................................... 189
                                    Evaluating Stroke and Rehabilitation Programs: An Overview            ......... 189
                                       Charles M. Wylie
                                    An Evaluation of a Stroke Program in California               ................. 192
                                       Bertram L. Tesman
                                    North  Carolina  Comprehensive  Stroke   Program             .................. 194
                                       B. Lionel Truscott


                              EVALUATION OF CANCER REGISTRIES
                                    Participants                          ......................................... 197
                                    Use and Evaluation of Cancer Registries                 ....................... 197
                                       Abraham Fingel
                                    Alternative Methodologies for Evaluation of Registries            ............. 198
                                       George Linden
                                    Methodologies for Evaluating Effectiveness and Value of Registries        ..... 200
                                       Charles R. Smart


                              EVALUATION OF REGIONAL ADVISORY GROUPS
                                    Participants                          ......................................... 207
                                    Regional Advisory Groups as a Factor in the Regionalization Process         ... 207
                                       David A. Pearson
                                    Bases for  Regional  Advisory  Group  Evaluation             .................. 211
                                       Paul E. White and Van Hove


                              MEDICAL CARE EVALUATION
                                    Participants                          ......................................... 219
                                    ABCD Strategy of Patient Care Assessment       ....................           219
                                       John W. Wflllamson


                              EVALUATION OF NEW CATEGORIES OF MANPOWER
                                    Participants   ......................................                   I   ... 225
                                    Evaluation of New Categories of Manpower       ....................           225
                                       Harriet Kitzman





                                                                          viii




                                                                          Page


              TRAINING FOR EVALUATORS
                   Participants .........................................  227
                   Summary of Session ....................................  227
                     Marion Leach




               ATTENDANCE BY ORGANIZATIONAL AFFILIATION   ..............   229

























                                                ix






                                     AN APPROACH TO EVALUATION FOR THE
                                             REGIONAL MEDICAL PROGRAM
                                                   DONALD A. SCHON, President
                                         Organization for Social and Technical Innovation
   Introduction                                                             Within  this  framework  evaluation  serves   three   distinct
        The  questions  in   which   we   are   primarily   interested  purposes:
   are these:                                                               Justification:  to  defend  what's   planned   or   what   has
        * What  are  the  criteria,   methods,   and   measures   per-      been  done.  We  justify  in  order  to   assign   reward   or
          tinent  to  evaluation  of  the  activities   of   the   Re-      punishment   (as   in   "grading"),   to   decide   what   re-
          gional Medical Program?                                           sources  to  commit  to  an  activity,  or  simply  to   place
        * How  can  evaluation   be   linked   most   effectively   to      an activity on a scale of  excellence.  In  any  case,  justi-
          the planning process?                                             fication   concerns   itself   with   identifying   what   has
        * What  are  the  appropriate  roles  for  those  engaged   in      been  done,  or  what   is   proposed,   and   appraising   it
          evaluation  at  project,  regional,  and  national   levels?      against some standard.
        These  questions  have  a  deceptively   simple   ring.   They      Control:  to  monitor  an  ongoing  activity   in   order   to
     raise,  in  fact,  not  only  the  special   problems   stemn-dng      make it conform to standard.
     from  the  nature,  context  and  history  of  RNT  but   several
     more   fundamental   questions   of   theory    concerning    the      Learning: to  change  activity,  to  do  it  better.  Learning
     evaluation of any activity.                                            may  be  limited  to  the  selection  of  means   to   achieve
                                                                            goals  or  to  conform   to   standards,   or   it   may   en-
                                                                            compass   change   in   the   goals   and   standards    them-
     Section I                                                              selves.
     Toward a General Theory
     of Evaluation                                                        For  any  program  such  as  RMP,  there  are   always   demands
        Evaluation is an  essential  part  of  intelligent  individual    for  justification,  control  and  learning.  But  it   is   not
     and organizational behavior.                                         always   recognized   that   these   several    purposes    have
     It  is  the  process  through  which  individuals  or   OTganiza-    different  implications  for'methods  and   systems   of   eval-
                                                                          uation.
     tions  perceive  the  consequences  of   action,   assess   their
     meaning  for   future   action,   and   reformulate   plans   and       We  are   accustomed   to   think   about   evaluation   from
     policies.                                                            the  point  of  view  of  a  rational  manager  who   supervises


                                                               Action by Individual

                                                                      (Work) @
                                                        (implementation in organization)



            Perception of Consequences by Individual                                          Reformulation of Action by Individual
                                                                                                            (Planning)
                             (Judging)
               (Evaluation by and of organization)                                             (Policy formulation by management
                                                                                                         of organization)




                        the business of an  organization  or  program.  The  rational     formance  of  the  components  the  information  is   intended
                        manager takes as his  reference  point  a  systems  rationale     to characterize.
                        - that  is,  a  set  of  formal  objectives,  operations  for       AU  variants  of  the  rational  manhger's  model   and   the
                        achieving  them,  and  methods  for  appraising  the   effec-     evaluation systems  that  flow  from  it  suffer  in  practice
                        tiveness  of  operations  in  achieving  objectives.   In   a     from an  overriding  constraint.  Characteristically,  systems
                        business firm,  the  systems  rationale  makes  reference  to     do  not  behave  as  they  are  supposed  to.  Even  the  most
                        profits and return on  investment;  in  the  'public  housing     bounded organized activities result  in  social  systems  that
                        system, to the provision  of  standard  housing  for  persons     do not behave  exclusively  in  terms  of  the  rational  pur-
                        cut off from  access  to  the  market;  in  the  health  care     poses  assigned  to  them.  As  distinct  from  the   rational
                        system,  to  improvement   in   people's   health,   in   the     manager's model, there is. always  a  real  system  of  actors
                        quality of care. or in equitable access to care.                  and  agencies  which  interact  with  one   another   in   the
                           According t.- - the rational manager's  model  of  evalua-     ways they are  found  to  do  and  with  the  interests   they
                        tion, the systems rationale is fixed  and  given.  Justifica-     are found to   have.   Their   discovered   interactions   and

                                        Level 1                                                               Top Management



                                        Level 2


                                                                                     oductio              ales                 Finance

                                        Level 3
                         tion consists, then, in  assessing  the  impact  of  past  or    interests may have li ttle to do wi th  the  interactions   and
                         proposed activity  on  established  systems  objectives.  How    interests  imputed  to  them  under  the  systems   rationale.
                         effective are these activities  in  meeting  objectives?  How       The  "discovered  systems"  of   organizations   and   pro-
                         efficiently  do  they  use  resources?  Control  consists  in    grams  tend  to  have  certain  features  in  common.  Regard-
                         monitoring  ongoing  activity  to  make  it  -   conform   to    less  of  systems  rationale,  individuals  tend  to  be   in-
                         established standards. Learning is limited to  the  selection    terested in:
                         of means for achieving objectives.                                  o  their own survival in their positions;
                                                                                             o  independence of action;
                            The  evaluation  process  appropriate  to   the   rational       o  local  conditions  and  needs  (as  opposed  to  '.'cen-
                         manager's  model  depends  on  the  assumption  that   every-          t@al's" view of them);
                         body in the system is to  some  extent  a  rational  manager.          protecting and extending territory;
                         People's accountabflities for activities  within  the  system          maintaining stability.
                         are supposed to mirror the systems rationale.                    These  interests  characterize   the   informal,   homeostatic
                                                                                          structure  of  organizations  and  programs.  But   discovered
                                                                                                                                   I
                            Within the organization or program, as        within   the    systems  tend  also  to   be   open-ended,   associated   with
                         systems rationale, activities are  organized  hierarchically.    emergent obj  ectives  and  swift  changes  in   goals   which
                         Each person is accountable for the  activities  of  his  com-    correspond to  individual  interests  in  creativity  and  re-
                         ponent, whose goals are keyed, in  turn,  to  the  objectives    sponsiveness.  Often  the  rational   manager's   model   con-
                         of  the  system.  The  job  of  evaluation  is   to   compare    strains  creativity,  responsiveness  and  freedom  of  action
                         accountabflities  with  the  actual  behavior  of  individual    in  ways  that  run  directly  counter  to  the  interests  of
                         components  within  the  system.  Evaluation   tends,   then,    actors and agencies within the system
                         to  become  an  auditing  process  in  which  a  third  party       Within  any  on-goirig  program,  the  rational   manager's
                         assesses behavior in terms  of  the  systems  rationale,  and    model  and  the,  discovered  system  always   co-exist.   The
                         sends information toward  the  top  of  the  system.  On  the    state  of  their  relationship  critically   detern-dnes   the
                         basis  of  this  information,  decisions  flow  downward   to    nature of evaluation.
                         influence  the  behavior  of   the   components   below.   At       When  the  two  systems  have  little  overlap  and  little
                         each successive step of the way, the primary  use  of  infor-    interaction, evaluation is lin-dted to retrospective  justifi-
                         mation is in justifying and  then  in  controlling  the  per-    cation,

                         2





                                                   Performative                                     Retrospective


                  The System of the
                  Rational Manager







                  The    Discovered
                  System


                                                   stem       produces     3.  The  discovered  system  and   the   rational   manager's
           In this condition, the evaluation sy                            system may fight  one  another  more  or  less  openly  until
           statements believed neither by  the  producer  nor  by  the                                               of  view   of   the
           consumer, which are generated ritualistically  in  response     they reach a compromise. From the point
           to formal  demand.  Rational  managers  produce  justifying     discovered system,  this  is  paying  a  price.  Those  in  the
           statements at regular intervals, expressed in the  language     system  do  some  of  what  the  rational  manager  wants  in
           of the systems rationale, and resources  continue  to  flow     order  to  preserve  considerable  abflity  to  satisfy   the
           into  the  system.  Evaluation  processes  have  no   other     interests  of  the  discovered  system.  From  the  point  of
           output  than  justification.  They  are  used  neither   to     view of  the  rational  manager,  the  discovered  system  is
           modify the systems rationale nor to force the  real  social     merely distorting system objectives in the direction  of  its
           system to conform to it.                                        own 1 terests; but he has to  put  up  with  it  to  get  any
           Where there is little overlap,  but  the  rational  manager     response at all.
           seeks to impose  a  systems  rationale  on  the  discovered
           system, several things may happen:                              In none of these dissociated  cases  is  there  any  interest
           1.   The   discovered   system   may    respond    verbally     in producing or using information that r.
           without other  changes  in  behavior,  by  offeringprofonna .   the  strategy  of  evaluation  as  justification.  Where  the
           retrospecdv-e-j'-ustificatio.n long  on  language  but  short  on systems  are  operating  in  parallel-   but   without   much
           substance, a process  generally  known  as  "conning."  The     contact,  there  is  common  interest  in  avoiding  informa-
           two systems operate substantially in parallel.                  tion that threatens dissociation. In  the  other  two  cases,
           2. The  discovered  system  may  respond  to  the  controls     there  is  common  interest  in  information  that   supports
           that the rational manager seeks to impose  by  adapting  to     the systems  rationale;  since  justification  rests  on  the
           the evaluation measures he  prescribes  but  continuing  to     systems rationale, and resource allocation rests on  justifi-
           operate as much as possible as  before.  Measures  of  per-     cation.  The  discovered  system  is  content   to   generate
           formance are  always  different  from  performance  itself.     information  that  conceals  how  great  the  discrepancy  is
           For example, in an effort to  control'expenditures  of  the     between  the  goals  of  the  rational  system  and  the  be-
           vocational  rehabilitation  system,  Congress  demanded  to     havior of the discovered  system  in  order  to  protect  the
           know  how  many  "rehabilitations  per  yeae'  the   agency     resource allocation they  need  to  continue  doing  more  or
           effected for a  given  investment.  "Rehabflitations"  were     less what it is they want to do.
           defined as job placements lasting  three  months  or  more.       However,  where  the  whole  activity  is  conceived  as  a
           As a  consequence,  the  vocational  rehabilitation  system     learning  system,  then  relationships  between  rational.and
           began to "cream" its clientele for  those  most  likely  to     discovered  systems  can  be  fundamentally  different   from
           graduate to job status leaving  out  those  who  were  most     those  just  sketched.  The  opportunity  for   learning   is
           in need and least able to qualify; to select low4evel  jobs     primarily  in   the   discovered   system.   The   discovered
           for graduates so as to facilitate entry; systematically  to     system  offers  the  most  vital  basis   for   reformulating
           avoid distinguishing between a  "case"  and  a  person,  so     systems   objectives   and   redesigning   systems    theory.
           that a graduate who had achieved job status,  lost  it  and     Discrepancies  between  the  rational  manager's  system  and
           returned  to  training,  could  be   counted   as   another     the  discovered  system  as  perceived  by  its   inhabitants
           rehabilitation";  and  systematically  to  avoid  follow-up     become  the  basis  for-progressive   modification   of   the
           of clients after three months.                                  system's  rationale,  of  modifying  the  real  interests  of
                                                                                                                                       3






               individual  participants,  and  of   developing   relationships    manager's  efforts  to  control   it,   may   mean   that   the
               between the total activity and its constituencies.                 rational   manager   is   simply   precluded   from    learning
                  It is critical that an  evaluation  system  aspiting  to  an    what's  actually  happening  in  the  discovered  system.   But
               i.mportant   role    in    intelligent    management    recognize  the  rational  manager  may  be  able  to  bargain   for   this
               rather  than  bury  discrepancies  between  systems   rationale    information   by   exchanging   information   about   resources
               and  the  discovered  system.  The  evaluation  system   itself    and  ongoing  administrative  changes  to  which  he  is  privy
               must  become  a  vehicle   for   continuing   interaction   and    for  accurate  information  about   what's   really   happening
               continuing  mutual  'influence  of  the  two.  Its  ability  to    in  the  social  system.  Even  more  powerful,  when   central
               support   intelligent,   direct   interactions   between    the    rational   management   gains   some    freedom    to    modify
               rational   manager's   system   and   the   discovered   system    systems rationale to  take  account  of  real  local  interests
               becomes  @  central  function  and  a  central   criterion   of    and activities, the basis for  withholding  or  distorting  in-
               adequacy  in  an  evaluation  system  oriented   to   learning.    formation  may  disappear.  The  way  may  then  be  clear  for
               While  these  considerations  are  important  at   all   times,    central  rational  management  and  local  people  to   bargain
               they  become  essential  in  a   period   of   development   or    effectively  and  directly  over  changes  in  systems  ration-
               mstabflity,  when  new  kinds  of  activity  must  be   devised    ale,  local  behavior  modification,  and   information   flow.
               to  meet  established  objectives  more  effectively  and  when    As in all such cases, the  bargaining  will  depend  on  estab-
               program  environment  changes  so  as  to  lead  to  shifts  in    lishing and maintaining good faith.
               objectives, as well.                                                  Several   additional   consequences   for   the   evaluation
                                                                                  system flow from these considerations:
               Learning-Oriented Evaluation                                             Information   intended   to    modify    behavior    must
                                               m
               Discovered Systetns Hooked                                               flow  upward  to  influence  systems  rationale  as  well
               to Rational Systetns                                                     as  downward  to  bring  the   discovered   system   into
                                                                                        line with pre-existing systems rationale.
                 When  planning  begins  to   incorporate   a   mutual   modi-          The  evaluation  information  that  is  gathered   should
               fication of  objectives  and  activities,  evaluation  includes          be  limited  to   amounts,   complexities,   and   preci-
               much  more   than   mere   measurement   of   the   extent   to          sions  determined  by  the  capability  and   willingness
               which  activities  conform  to  specification.  The  evaluation          of  actors  within  the  system  to  learn  from  it,  as
               system that is  oriented  to  learning  has  special  features:
                                                                                        experienced   in   actual   practice.   Nobody   in   the
                    The  'conceptual   framework   for   evaluation   has   to          system  should  be  presented   with   more   information
                    include  a  description  of  the  discovered   system   as          than  he  can  handle,  nor  information  laid   out   in
                    well  as  the  rational  manager's  statement  of  systems          more  precision  or  complexity  than  he   can   respond
                    rationale. This  includes  a  description  of  key  actors          to.  Analyses  should   not   present   actors   with   a
                    and  agencies,   actual   relationships   and   modes   of          greater  breadth  of  alternatives  than  are  real   for
                    interaction  among  them,  and   the   several   interests          them.  As  a  corollary,  the  evaluation  system   needs
                    of all of them. It must  include  also  a  description  of          to  be  able  to  detect  the  changing  capability   and
                    the  real  (if  informal)  evaluation   system   as   dis-          willingness   of   actors   to   use   information,   and
                    covered   -   the   information   that   actors   in   the          should  itself  be  capable  of  responsive  modification
                    system  in  fact   produce,   are   interested   in   pro-          in turn.
                    ducing, and how they use it.                                        The  evaluation   process   shoul.d   be   structured    to
                    An  analysis  of  discrepancies   and   overlaps   between          accomodate  the   different   kinds   of   learning   ap-
                    the  systems  rationale  and  the  behavior  of  the  dis-          propriate  to  different  roles  and  levels  within  the
                    covered  system.  This  analysis  takes  account  of   the          system  (rational  managers,  project  pushers,   evalua-
                    differing perspectives of actors in the system.                     tors, planners, etc.).
                    Strategies  for  responding   to   discrepancies   between
                    the  discovered  system   and   the   rational   manager's          The  learning  objective  should   also   determine   the
                    system.  Mere  analysis  is  not  enough;  learning   must          content,  extensiveness,  duration,   and   accessibility
                    be capable of application.                                          of  information  in   die   evaluation   system   memory.
                 These  factors  focus  on  gathering   accurate   information          This  requirement  places  high  priority   on   accessi-
               about  the   discovered   system.   The   discrepancy   between          bflity  and  retrieval  capability  on  behalf  of   many
               the  rational  system  and  the  discovered  system,   or   the          different  levels  Nvithin  the  system  in  addition  to
               response   of   the   discovered   system   to   the   rational          that of the rational manager.


               4





             Since the  learning  derived  from  evaluation  may  be          attei-npt  through  its  own  evaluation  processes   to
             applied  to  evaluation   processes   themselves,   the          accomplish justification,  control,  and  learning  that
             conceptual  framework  for  evaluation  may  itself  be          is  downgraded.  Accordingly,  the   evaluation   infor-
             expected  to  change  (sometimes  rather  rapidly);  so          mation  flowing  to  central  from  the  local   regions
             information  needs  to  be  gathered   and   formulated          normally reflects the nature  of  the  processes  devel-
             in ways that make it more or  less  equally  usable  in          oped for  raising  and  answering  evaluative  questions
             terms of a broad range of  systems  rationales.  Prior-          in  the  localities  rather  than  the  answers  to  any
             ities should be given  to  those  bits  of  information          specific  questions  thought  up  by   central   manage-
             that are likely  to  retain  high  relevance  across  a          ment.
             range   of   manager's   rationale    and    discovered          Central also takes  on  the  role  of  building  a  net-
             systems.                                                         work   learning   system,   facilitating    information-
                                                                              transfer  from  locality  to  locality  and  encouraging
                                                                              specific local experiments.
       Cases in Which There is
       No Explicit Systems Rationale
         What if the activity  to  be  evaluated  is  itself  recog-    Section 2
       nized as so diverse,  diffuse,  swiftly  changing,  and  open    RMP in the Context of
       that no overall systems rationale is  credible?  This  situa-    Evaluation Theory
       tion may occur  with  respect  to  public  problems  urgently       To place  the  Regional  Medical  Program  in  the  evalua-
       requiring solution but for which there are  no  clear  policy    tion context  developed  in  the  previous  section,  some  of
       answers,  where  national  willingness  to  devote  resources    RMP's principal characteristics should be recited.
       to  their  solution  is  high,  though  the  credibility   of       1.  There  is  no  single  organization  corresponding   to
       proposed  rational  solutions  may  be  low.   Agencies   may    RMP.  RMP  is  a   broad-aimed   Federal   program   concerned
       be -funded  to  work  on.  such  problems,  constrained  only    with   introducing   changes   of   various   kinds   into   a
       within very  broad  limits  as  to  what  their  work  should    number of  more  or  less  interconnected  systems  of  actors
       be like.  What  are  the  implications  here  for  evaluation    and  agencies  involved   in   health   care.   Within   these
       systems?                                                         systems, RMP attempts to  play  a  variety  of  related  roles
             Each region or  subregion  (or  other  entity)  saddled    with respect  to  other  actors  and  agencies;  but  for  the
             with a whole  problem  becomes  a  center  of  its  own    most  part  it  cannot  directly  control   them.   RMP   does
             problem-solving  process.  The  number   and   location    not,  therefore,  have  to   do   with   a   single   rational
             will  depend  on  the  number  of  centers  that   turn    "system," in  the  sense  used  earlier,  and  its  boundaries
             out to  be  capable  of  functioning  under  their  own    are vague and shifting.
             individually  developed  systems  rationales.  In  this       From the  point  of  view  of  evaluation,  this  assertion
             situation  the   distance   between   information   and    has  several  implications.  RMP's  scope  and  turf  do   not
             analysis  is  minimized  ,   and   responsibility   for    have  sharp  boundaries.  We   cannot   go   about   analyzing
             designing and  conducting  the  evaluation  process  is    RMP as  though  it  were  a  unitary  organization,  like  the
             very close  to  the  actors  who  are  accountable  for    Veterans'  Administration,  for   example.   And   while   RMP
             the activities under evaluation.                           has  formulated  broad  objectives  for  itself,  its   funda-
             In this  case  central  management's  evaluation  func-    mental activity in  relation  to  these  objectives  rhust  be
             tion is changed with respect to that  of  the  regions.    understood for the most  part  as  "influencing"  or  "facili-
             Central   management   may   now    impose    on    the    tating" rather than direct control.
             localities criteria for the evaluation process, but  it       2.  There  is  no  single,  established  systems  rationale
             is no longer in  a  position  to  impose  criteria  for    either for the health care syste m as  a  whole  or  for   RMP
             substantive  evaluation  of  concrete  activities.  For    in particular. There are various rationales, held  at  various
             example,  central  management  can  still  ask  whether    times and in various  contexts  by  different  actors  in  the
             regional evaluation  processes  are  differentiated  in    system.
             terms of  justification,  control,  and  learning;  but       3.  The  larger  health  care  system  and  the   RMP   are
             the central evaluator will accord just  as  high  marks    changeable. They are not in  a  stable  state.  The  character
             to  a  region   displaying   one   workable   form   of    and functions of  these  systems  are  themselves  in  process
             differentiation  as  to  a  region  displaying  another    of  constant  change.  Within  them,  the   key   actors   are
             form. It is only the region that  does  not  explicitly    often unsure of their principal,  functions  or  of  how  best





                to  carry  them  out,  and  they  tend  to  shift  behavior   as        The   authorizing   legislation   made    no    attempt    to
                they learn and as the system around them changes.                   rationalize  these  elements  or  to   resolve   potential   con-
                    4. Ne-v-ertheless,  as  a  federal  program  RMP  is  locked    flicts  among  them.  It  was   understood   by   many   of   the
                into  a  structure  of  controls  and  demands  for   justifica-    key  actors  that,  as  the   program   matured,   the   specific
                tion. At  the  national  level  these  include  regular  reviews    meaning  of  its  legislative  provisions   would   develop   and
                by  the  Congress,  the  Bureau   of   the   Budget,   and   the    clarify.
                Department   of   HEW.   These   demands    for    justification        It is  not  surprising,  then,  that  there  have  been  per-
                and  for  controls  over  the  expenditure  of  funds  are,   of    ceptible  shifts  over  time  in  the  dominant  systems  ration-
                course, passed on to the regional program le@el.                    ale  for  RMP,   even   though   no   element   originally   con-
                    The  problem  of  devising  approaches  to  evaluation   for    sidered as the  legislation  evolved  has  altogether  ceased  to
                RMP  is  essentially  that  of  meeting   what   may   well   be    exert some influence.
                conflicting  requirements  for  learning,  on  the   one   hand,        Let us  be  explicit  about  an  evaluation  scheme  that  is
                and  for  justification  and  control,   on   the   other.   The    generally  accepted  as  appropriate  to  one  of  the   simplest
                vagueness  and  changeableness  of  objectives,  lack  of   pro-    and  accordingly   most   easily   rationalized   interpretations
                gram   control   over   components   to   be   influenced,   and    of  RMP.  We  refer  to  the   center-periphery   regionalization
                sources  of  methodological  uncertainty   all   argue   for   a    model  based  on  the  diffusion   of   technology   and   infor-
                flexible,   process-oriented    approach    to    evaluation-as-    mation that  is  assumed  to  be  stored  in  the  great  medical
                learning;  whereas  the  agents   of   rational   administrative    centers. In this instance, it  is  seen  as  desirable  to  judge
                control  tend  to  press  for  firm,  quantitative  measures  of    the  program   initially,   at   both   national   and   regional
                  rogram impact.                                                    levels, by its  effectiveness  in  reducing  rates  of  mortality
                p
                    Like most   broad-gauged  federal  programs,  the   legisla-    and  morbidity   for   heart   disease,   cancer,   stroke,   and
                tion  establishing  RMP  represented   a   series   of   compro-    related diseases.  Individual  projects  are  seen  as  means  to
                mises  among  the  diverse  interests   of   various   concerned    these  ends,  and  fall  basically  into  the   following   cate-
                  oups.  The  authorizing  legislation  is,  therefore,  a  kind    gories:  deployment  of  new  facilities  (for   example,   coro-
                gr
                of  mosaic  of  objectives,  values,  and  constraints.   Ajnong    nary  care  units);  establishment  of   new   linkages   between
                the more important elements of the mosaic are these:                medical   centers   and   peripheral    care-providing    centers
                       Emphasis  on  the   provision   of   means   to   improve    (for   example,   exchange   of   personnel);   the   development
                       the treatment  of  the  three  "categorical"  diseases  -    of  new  working   relationships   (for   example,   changes   in
                       heart disease, cancer and stroke.                            referral   patterns);   continuing   education   (for    example,
                       Emphasis  on   the   transmission   of   advanced   tech-    training  of  physicians  and  other  medical   personnel);   and
                       niques and knowledge relating to these diseases.             information   dissemination   (for   example,    DIAL    access).
                    *  Emphasis  on   the   method   of   continuing   education        The  major  kinds  of   evaluative   questions   under   this
                       as a device for  this  transmission;  and  on  the  major    interpretation  of   the'RMP system are these:
                       academic  medical  center  as  the  principal  source  of        1.  What  are  the  kinds  of  baseline  data  and   measures
                       expertise.                                                   of  performance  by  which   the   impact   of   diffusion   pro-
                                                                                    jects on mortality and morbidity can be assessed?
                    *  Emphasis   on  maintaining  or  improving   the   quality
                       of medical care.                                                 2.  What  is  the  relative   effectiveness   and,-efficiency
                                                                                    in relation to cost    of  the  various  technologies   diffused,
                    *  Concern  with  the  region  as  the  principal  unit   of    seen  as  means  of  achieving  reductions  in  rates   of   mor-
                       activity; concern,  that  is,  that  the  program  be   a    bidity and mortality?
                       regional  one,  with   regional   centers   of   activity        3.  What  is  the  related  effectiveness,   for   particular
                       throughout  the   country;   concern   with   recognition    technologies and  for  particular  regional  situations,  of  the
                       of  regional  diversity  of   problems   and   resources;    various  methods  of   diffusion?   This   question   leads,   in
                       and  concern  with  "regionalization"  as  a  process  of    turn,  to  questions  about  the  optimal  "regions"  for  diffu-
                       knitting  together  or  building  regional  resources  to    sion, the forms  of  greatest  "diffusion  impact"  for  a  given
                       realize the purposes of the Act.                             investment  of  dollars  and   other   resources,   patterns   of
                       Emphasis    on    the    establishment    of    voluntary    utilization of new facilities and the like.
                       arrangements   among   regional   institutions   as   the        Other  aspects  of  the   activities   within   the   center-
                       dominant mode of program activity.                           periphery   model   of   RMP   -   for   example,   the   manage-
                       Specific   warning   against   "interference    in    the    ment  of  new  institutional   arrangements   at   the   regional
                       interface between patient and doctor."                       level - must  be  judged  in  terms  of  their  effectiveness  in

                6



                                     f   the   quality   Of   care    through     The   effects   of   substantial   investment   in   Medicare    and
       leading to enhancement o                                                   Medicaid   have   begun    to    convince    observers    that    no
                                                        d technoloaN
       the   more   effective   diffusion   of    advance            -      I     amount  of  investment  in  payment  for  care   will   suffice   to
       with  the  ultimate  effect,  of   course,   of   reducing   mortality                                                   vider          system.
                                                        tified      diseases.     introduce  necessary  changes in the  pro ,
       and  morbidity   from   the   categorically   iden                         There is clea rly  need  for  some  forms   of   interIvention         on
               In  the  minds  of  many  key  actors   in   Wasliinaton   and
       in  the  regions,   the   DeBakey   model   came   to   dominate   the     the provider side as well.   r   to   be    overriding    objections
       conceptual  climate  of   the   early   phases   of   RNIP.   But   it     There continue to appea
       was   not   always   or    everywhere    the    dominant    ,iew    of     either   to   the   development   of   nationalized    systems    of
       RMP  activity.  In   the   discovered   systems   of   some   of   the     care   or   to   such   decentralized   solutions   as    community-
       regions,   regional   coordinators   and   other   key   actors   took     based   group   practice,   on   a   large   scale.   Shortages   of
       as  primary  the  sorts  of   changes   in   institutional   arraizoe-     scarce    resources    of    medical    manpower    suggest     that
       rnents  which,   from   the   point   of   view   of   the   DeBakeli)     changes  in  the  system   will   have   to   work   with   existing
       model, figured only as secondary means to an end.                          personnel   and,   very   largely,   with   existing   institutions.
               In this interpretation:                                            This  means.  to  a   great   extent,   attempting   to   facilitate
               aRjIIP's   central   concern   may   be    expressed    through    voluntary re-arrangements,of existing institutions.
               categorical  diseases  or   @vith   the   diffusion   of   ad-     Of    the    available     program     instruments     (Neighborhood
               vanced medical technology, but RMP                 consciously     Health    Centers,    Comprehensive    Health     Planning,     Com-
               concerns  itself   with   overall   improvement   in   quality     munity   Mental   Health   Centers),   kMP   presents   itself    as
               of care and  equity  of  access  to  care.                         perhaps   the   most   promising    candidate    for    intervention
               &But  these  sorts   of   improvements   require   changes   in    of  this  kind.  What  RMP  has  been  doing,  initially  en   route
               the  structure   and   modes   of   interactio?is   of   care-     to   the   DeBakey   model   in   some   regions   or    in    other
               providing   institutions   which   no   single   agency   con-     regions  as  a   matter   of   primary   though   informal   agenda,
               trols  -  changes  that  can   be   generally   described   as     now  is  emerging  as   a   more   dominant   (though   not   exclu-
               knitting   together   components   of    the    system    that     sive)  rationale  for  the  program  as  a   whole.   It   must   be
               are  now  fragmented  so   as   to   permit   more   effective     added,  or  course,  that   by   no   means   all   regions   regard
               and rationalized planning and action.                              themselves   as   primarily   involved   in   systems    transforma-
               #These   systems   changes   are   necessary   conditions   for
               improvement    in    quality    or    equi.ty  of   care.   They   tion.  Some  RMP's  still  regard  themselves  as   solicitors   and
               must    precede    any    significant    improvement     along     screeners  of  proposals,  and  do  not  yet   conceive   of   them-
               these lines.                                                       selves  as  "programs"  in  any  sense  other  than   as   clearing-
               In  the  past  year,  systems   transformation*   has   be,,un     houses for projects.  And  in  nearly  all  regions,  there  is  the
                                                                            0     residue  of  the  view  of  RMP  as  a  conglomerate   of   projects
               to   dotwnate   among   competing   systems   rationales   for     centering   around    continuing    education,    training,    coro-
               RMP  (without,  of   course,   completely   displacing   other
               views) at national as well as some regional levels. NVhile  it     nary  care  units,  and  the  like.  At  the   very   least,   then,
               is to some extent a  subject  for  guesswork  why  this  shift     coordinators  face,  as  part  of  the  task   of   systems   trans-
               has occurred, certain factors suggest themselves.                  formation,  the  problem  of  what  to   make   of   and   what   to
               There  has   been   a   movement   into   good   currency   of     do  with   the   projects   initiated   under   earlier   views   of
               certain basic concerns about  the  national  system  for  pro-     RMP.
               viding  medical  care  -   concerns   about   rising   medical       Under a systems transformation model for RMP:
               costs, about the effective  exclusion  from  the  health  care           The   primary   unit   for   evaluation   becomes   the   pro-
               system  of  large  numbers  of  disadvantaged  people,   about           gram;  and  since  RMP  is   conceived   as   an   essentially
               shortages of  medical  manpower,  about  the  difficulties  of           regional   enterprise,   this   means   the   regional    pro-
               negotiating  the  medical  care  system  even   for   ordinary           gram.  It  will   be   necessary   to   reach   both   "above"
               middle class people.                                                     this  level  to  the   national   program   and   "below"   it
                                                                                        to  the  project;   but   the   regional   program   is   pri-
                                                                                        mary.
               *"RMP as  process,"  "RMP  as  facilitator,"  "RMP  as  oppor-           Every  element  of  RMP   takes   on   a   dual   aspect.   As
               tunistic change agent" were expressions heard as early as 1967           we   seek   to   assess   projects,   regional   program   and
               and conveyed the underlying idea behind systems transformation           national   program,   we   must   ask    both    about    sub-
               before this rationale became as significant as it now is. Recent         stantive  changes  in  the  provision  of   care   -   changes
               legislative proposals convey the idea even more explicitly.              in   the    quality    and    configuration    of    services,
                                                                                                                                                     7





                                                                      e
                       changes in access to  services,  changes  in  h  alth  -      levels of change in health, access to health  care,  quality
                       and  about  systems  transformation.  Seen  as   systems      of care, configuration  of  health  resources,  as  well  as
                       transformation,   RMP    functions    in    two    ways:      changes  in  the  institutional  arrangements,  interactions
                       through  the  direct  efforts  of   the   regional   co-      and attitudes cl:a:acte'ristic of the  health  care  system.
                       ordinator and those  he  works  with  to  knit  together      The issue of justification raises  sharply  the  problem  of
                       or  otherwise  influence   elements   of   the   medical      what it is possible to know  about  these  matters,  and  at
                       care system  of  his  region,  and  through  the.shaping      what level of generality it is possible to know it.
                       and  selection  of  projects  which   become   occasions       The  remainder  of  this  paper  "411  be  taken  up   with
                       to effect systems transformation.                             questions (1) and (2), above. We  will  focus  on  the  view
                       Evaluation must  take  account  of  regional  diversity.      of  RMP  as  systems  transformation  and  will  attempt  to
                       The starting conditions of  the  region,  the  array  of      spell out the bases on which, in spite  of  regional  diver-
                       resources, the problems to be  attacked,  the  level  of      sity  and  open-endedness,judgments  about   regional   per-
                       development,  the  regional  strategy  -  there  may  be      formance  may   be   made   and   learning   about   systems
                       as  many  of  these  as  there  are  regions.  From  the      transformation may be fostered.
                       point of view  of  evaluation,  therefore,  the  content
                       of  regional  programs  should   be   expected   to   be
                       different. There  is  no  'model"  of  a  regional  pro-      Section 3
                       gram  to  be  applied  to  all  regions,   although   we      The Central-Regional Dialogue
                       should  be  able  to  develop  a  conceptual   framework      There  is  a  conceptual  framework  for  systems  transfor-
                       which  will  allow  assessment   of   diverse   regional      mation  in  RMP  from  which  we  can  derive  criteria  and
                       models.                                                       questions  useful  in  undertaking  and  assessing   systems
                       Evaluation  must  not  only   take   account   of   this      transformation,  without  violating   regional   differences
                       regional diversity; it must also  take  account  of  the      and  without  second-guessing  particular  regional  answers
                       fact  that  regional  programs  are  in  critical   ways      to the substantive questions of medical care.
                       open-ended.                                                   The  essential  elements  to   which   attention   must   be
                  Regional   programs    undertake    systems    transformation      paid are these:
              by  engaging  the  emerging  issues  of  medical  care   in   the      9  Starting conditions (What is to be changed?).
              region. These  are  only  partly,  if  at  all,  within  the  co-      9  Ends-in-view (Changed to what end?).
              ordinator's control;  to  be  effective  he  must  use  them  and      o   Processes   and   techniques   (How   can   change    be
              build  on   them.   Evaluation   must   take   account   of   the         accomplished?).
              open-ended  or  existential  character  of   regional   activity;      Broad  regional  strategies   for   systems   transformation
              except  within  a  very  broad  range,  it  cannot   second-guess      express  directions  for  the  process  through  which   the
              the issues  to  be  encountered  in  a  particular  region  at  a      region may be brought  to  move  from  its  starting  condi-
              particular  time;  and  it  must  not  impose  on  the  region  a      tions (as they are conceived in a  particular  instance)  to
              model  of  sequential  activities  independent  of   the   issues      particular ends-in-view. Characteristically, such a  process
              of medical care which in fact arise.                                   proceeds in stages of:
                 The   central   questions    of    evaluation    now    become      o  Diagnosis (getting started, casing the region).
              these:                                                                 o  Involvement  (engaging  these   individuals   and   agen-
                 1.  How  can  we  facilitate  learning  about  systems  trans-         cies whose interaction is taken to be critical).
              formation,  at  all  three  levels,  but  with  emphasis  on  the      9  Planning  and  goal-clarification  (discovering  feasible
              regional program?                                                         processes and  choosing  and  testing  specific  ends-in-
                 2.  Given   regional   diversity   and   open-eiidedness,   on         view).
              what  basis  can  we  control   regional   activities   or   hold      These stages are apt to  be  cyclical  rather  than  sequen-
              them to standard?                                                      tial. The  passage  from  diagnosis  through  implementation
                 3. Given the  several  levels  of  change  relevant  to  eval-      leads to a  revised  picture  of  starting  conditions,  and
              uation  of  RMP,  how  can  we  go  about  the  justification  of      through  the  cycle  again.  Because  several   streams   of
              past or projected regional activity?                                   activity often proceed concurrently, the  region  may  at  a
                The  questions  of   justification   demand   separate   treat-      given time engage  simultaneously  in  all  stages.  As  the
              inent.  Given  the  multiple  impacts  of  RMP  activity,  justi-      region  moves  through  stages  of  systems  transformation,
              fication  requires  methods  for   identifying   baseline   data,      in its developmental cycle, it  may  extend  the  scope  and
              ends-in-view,  and  indicators   of   change   at   the   several      depth of the issues it tackles.

              8




         Given  this  skeletal  view  of  systems   transformation,   an          All  likely  emphases  have  been   tried   out   by   the
        evaluative  process   oriented   toward   learning   must   take          evaluator  in  an  effort  to  test  and  understand   how
        the  form  of  a  dialogues  continuin-   process   of   inquiry          the starting conditions fit together dynamically.
        in  which  the  regional  coordinator*  and  RMPS   both   raise   An   adequate   response   constitutes   a   diagnosis   of   the
        and  respond  to  questions.  This  is  for   several   reasons.   regional  health  care  system.  It  also  furnishes  the   eval-
        Given   the   open-endedness   of   '@systems   transformation,"   uator    with    some    beginning    hypotheses    about     how
        prospective  systems  rationales  for  RMP  must   be   inferred   skillful the regional core staff is in casing the region.
        from   (rather   than   imposed   on)    regional    activities.       When well  explored  and  laid  out  in  the  dialogues,  the
        Systems  rationales   and   systems   activities   must   modify   diagnosis includes  the  data  crucial  to  workina  out  strate-
        one  another.  The  evaluative  process  must  detect   discrep-   gies  of  systems  transformation,  both   those   which   define
        ancies    between    systems    rationales    and     discovered   health  issues  and  health  needs   and   those   which   define
        systei,ns,  and  tactics  for  responding  to   those   discrep-   the  organizational  and  political  character  of   the   health
        ancies.  Moreover,  project  and  program   goals   shift   over   care system.
        time. That is often a  si-n  of  progress;  and  the  evaluative
                                C,                                             9  What is  the  character  of  the  principal  health  prob-
        process  should  help  to  discover  whether  it  is,   and   in          lenis of the reaion? What is their distribution?
        appropriate cases both reflect and encourage it.
                                                     z'                        o  What  is  the  character  of  the  present   configuration
          In  what  follows,  we  list  guidelines  for  the  kinds  of,          of health care  facilities  and  resources?  What  is  the
        questions  to  be  raised  in  such  a  dialogue;  criteria  for          nature of  the  health  care  delivery  systems  that  are
        systems   transformation,    from    which    these    questions          dominant in the region?
        flow; and, in some instances, illustrations of response.               0  What are  the  patterns  of  access  to  care  among   the
          One  test  of  the  dialogue  is  that  both  coordinator  and          principal population groups?
        central"  become  able,  on  the  basis  of  it,  to  form  con-
                                                                           The    foregoing   questions   aim   at   establishin-   starting
        tinning,   grounded   judgments   of   regional   program   per-                                                      C,
        formance.  A  second  test  is  that  as  a  by-product  of  the   conditions  at  the  level  of  health,  access  to   care,   and
                                                    re   proficient   at   configuration of care-providing resources.
        dialogue the coordinator becomes mo
        designing  and  carrying  out  the  process  of  systems  trans-          Who  are  the   key   actors   and   powers   within   the
        formation. A third is that the  national  staff  is  enabled  to          health  care  system  of   the   region?   How   do   they
        formulate   progressively   more   adequate   "systems   ration-          relate to the power  structure  and  to  the  politics  of
        ales" for RMP.                                                            the region as a whole?
          The  dialogue  follows  what  we  have   identified   as   the          What  is  the  nature  of  the  linkages,  the   relation-
        three   main   elements   of   systems    transformation    and,          ships,  the  patterns  of  referral,  the   tensions   and
        within them, the stages of development.                                   conflicts, among these key actors?
           1.   Starting   conditions.   The   coordinator   should   be          What  do  the  central  actors  perceive  as   the   major
        capable  of  articulating  a  regional  diagnosis  which   holds          issues  of  health  care  for  the  region-whether   these
        water,  at7d  which  provides  the  basis  for  the  formulation          are identified in  disease-specific  terms,  in  terms  of
        of directions of systems transformation.                                  access to care, quality of care, or  in  terms  of  costs,
        The  subject  has  to  be  probed  to  the   point   that   both          manpower,   patterns   of    dominance    and    distribu-
        participants in the dialogue are convinced that:                          tion, or other facets of the health care system?
           *  The   evaluator   understands   the    spokesman's    view    Ths   next set of questions aims at an understand       Iing   of
              of  the  region  and  has  stated  enough  of  it  clearly    the "political"  forces  that  can  be  used  or  that  must  be
              enough to reassure himself and the spokesman.                 dealt with in any strategy for systems transformation.
           *  The  spokesman  has  stated  whether  he   believes   this       Out  of  responses   to   these   questions   come   regional
              particular  array  of  starting   conditions   is   tough,
              average, or a  bit  simpler  to  deal  with  than  average    diagnoses   which   provide   the   material    for    designing
              (assuming  for   the   moment   the   accuracy   of   what    strategies of systems transformation for the region.
              the spokesman has said).                                         At  the  point  of   establishing   agreement   on   starting
                                                                            conditions, the evaluative dialogue has to involve:
                                                                               o  Feed-back to a widening circle.
              We will use "regional coordinator" as shorthand for  those       o  Testing  the  perceptions  of  those  who  first  describe
        agents involved in formulating and carrying out RMP strategy  at          starting  conditions,  strategies,  or  other  aspects  of
        the regional level.                                                       RMP and the territory in which it functions.
                                                                                                                                           9





                          Some appraisal  (i.e.,  development  of  a  more  or  less         RMP  has  to  make  its  way  among  a  number  of   giants,
                          acceptable  description)  of  the  way   the   local   RMP     all zealous  defenders  of  quality  medical  care,  each   with
                          went about data selection and gathering.                       its own  tradition  of  constructive   innovation,   each   with
                          Gradual  clarification,  through  the   dialogue   itself,     its own  considerable  institutional  inertia   and   sense   of
                          of  the  specifics  on  which  detailed   information   is     independence.
                          needed.                                                            Y  Region.  In  the  region's  largest  city  there  is  one
                       The  following   are   excerpts   from   regional   diagnoses     large  medical  school  and  one   large   community   hospital.
                    which  illustrate  something  of   the   variety   of   starting     The  region   consists  of  five   quite   different   counties.
                    conditions to be discovered.                                         Three  counties  made   common   cause   with   RMP   from   the
                                                                                         outset. Two are left.  In  one,  a  private  physician  has  his
                       X  Region.  X  is   a   prosperous,   relatively   homogenous     own   comprehensive   health   plan;   prepaid   medical    care
                    society. Good  medicine  is  practiced  here,  and  the  profes-     has  been  attempted  under  his  auspices;   success   is   be-
                    sion is in relatively good  repute  with  the  local  political-     lieved to be uncertain;  critics  prophesy  failure.  The  other
                    social  establishment.   As   yet   medicine   and   the   other     county is simply cut off  and  disinterested.  It  is  difficult
                    health  professions  are   facing   only   tentative   questions     to  get  medical  or  consumer   representatives   from   either
                    about  the   "relevance"   of   where   sub-specialization   and     county  even  to  meet  for  reasons  that  pre-date  RMP,   but
                    bigger-better hospitals take us.  But  something  very  real  is     embrace  it:  several  of  the  major  counties  are  joined  in
                    brewing  in  the  state  leoislature's   effort   to   force   a
                                                 0                                       uneasy alliance, with  many  rivalries,  all  felt  particularly
                    "Family   ,   Practice"   Department   on   the    distinguished     strongly in the smaller cities.
                    specialists  of  the  University  medical  faculty.   Additional         Z  Region.  The  major  hospitals  and  associated   medical
                    intimations exist  in  the  reluctance  and  opposition  of  the     schools are all in the  major  city  and  dominate  the  region.
                    Academy  of  General   Practice   to   the   way   the   medical     These  are  set  against  the   smaller   community   hospitals,
                    faculty  had  first  planned  to  go   about   teaching   family     each of which  in  turn  is  trying  to  be  a  medical  center.
                    medicine.                                                            Not surprisingly,  there  is  relatively  thin  patient  use  of
                       Layer  on  layer  of  competent,  skilled,   devoted   people     these expensive  facilities  in  suburban  hospitals.  Not  sur-
                    working in hospitals and  other  health  care  institutions  all     prisingly,   too,   there   are   parochial   and    compartmen-
                    over the  state,  all  of  which  tend  to  emulate  or  somehow     talized referral  patterns  disturbed  by  conflicts  among  the
                    react  or  respond  to  the  presence  of  the   internationally     several large medical  schools  and  hospitals.  There  tend  to
                    famous  institutions:  the  Central  Clinic,   the   University,     be  economic  and  social   distinctions   drawn   between   the
                    and   Rehabilitation   Foundation.   There   is   an    apparent     largest  and  the  other  medical   school   complexes,   though
                    shortage  of  manpower  willing  and   able   and   wanting   to     these  may  be  decreasing,   and   certainly   keep   changing.
                    perform health care services  on  the  level  of  ordinary  care     With  all,  the  distribution  of  physicians  to  patients   is
                    for  ordinary  conditions.  Town-gown  issues  are   real,   but     highly inequitably spread over the region.
                    because  "gown"  somehow   includes   Central   City   as   well         0  ghetto areas:       1/3000 to 115000
                    as  "The  U,"   and   because   "everybody"   was   trained   at         o  center city:        1/200
                    "The  U,"  the  issues  take  a  special  form.   Centralization         *  suburban:           1/700 to 1/800
                    of  the  Clinic   and   decentralization   of   the   University         0  rural:              1 /I 000 to 1/2000
                    complicates   their   association,   whenever   joint    commit-         The  5  medical  centers  have  limited   goals.   All   are
                    ments  are   required   or   contemplated.   Good   acute   care     under  great  financial  pressure,  pressure  relative  to   in-
                    general hospitals are  a  dime  a  dozen,  and  coming  to  view     come,  to  student  load,  and  pressure  to  pay  attention  to
                    one  another  as  competitive   whether   they   are   or   not.     the ghettoes. They  are  beginning  to  believe  that  is  where
                    Many  are  trying  to  become  referral  centers  both  in   big     the  money  is.  In  the  meantime,  the  cultural  institutions
                    specialist  consulting   staffs   and   many   high   technology     of  the  major  urban  center   continue   to   tend   to   turn
                    services.                                                            inward, there  is  very  little  that  can  happen  "unless  you
                       Generally    the    establishment,    medical    and     non-     own it."  So  the  tendency  is  rather  stronger  than  average
                    medical,   exhibits   a   touc-,h-minded,   "show    me"    con-     to want to turn  RNIP  and  training  dollars  to  the  enhance-
                    servatism,  tempered   b@,   a   very   active   consensus   and     ment of existing institutions and departments.
                    willingness  to  try  out  credible  ways   of   improving   the         Rivalry  conditions   all   attempts   to   regionalize   or
                    situation  (e.g.,  40%  of  X-State  private   physicians   have     otherwise  bring   about   constructive   associations   between
                    tried out group  practice.  They  and  their  patients  like  it     people  in  the  somewhat  depressed   cities   of   the   North
                    well enough to stick with it.)                                       and the rich primary city.


                    10




          2.   Preliminary   strategies.   -   Proponents    of    the    re-       both   for   changes   in   the   delivery   system   and    for
         gional  diagnosis  should  be   capable   of   meeting   chaflen(,es       changes   in   their   own   position   within    the    system9
         as  to  the  accuracy  or  relevance  of  their  analysis.  But  the       What   are   the   critical   "starting   issues,"    and    how
         analysis  need  be'  neither  exhaustive  nor   entirely   accurate.       might  these   be   used   to   move   toward   systems   trans-
         It  is  of  greater  importance  that  it  be  capable  of  shifting       fori-nation?
         in  response  to  challenge  and  that  there  be,  in  the  inquiry    But the   specific   forms   of   these   questions   must    Cot-ne
         undertaken   by   the   coordinator,   a   continual    source    of    from  the  regional  diagnoses,  and  must  elicit  the   ways   in
         challenge  to  be  met.  In  particular,  it   is   important   that    which  prelin-Linary  strategies  address   themselves,   or   fail
         judgments  about  major  issues   of   health   need,   quality   of    to  address  themselves,  to  the  issues  raised  in  these  diag-
         care  and  access   to   care,   facilities,   manpower,   cost   Of    noses.
         care,  and  the  political  and  organizational  structure  of   the    The   following   are   examples   of   some   of   the    prelimi-
         health care  system,  all  be  subject  to  the  continual  test  Of    nary  strategies  emergent  from   the   fragments   of   diagnoses
         the  multiple  perspectives  of  key  actors  in  the  health   care    listed  above,  and   questions   that   the   evaluator   can   or
         system.   Where   important   conflicts   of   perspective    arise,    should  raise  about  these  strategies,  to  push   the   dialogue
         they  should  be   confronted   explicitly   and   actively.   Where    a step further:
         they  cannot  be   resolved,   these   conflicts   of   view   them-
         selves become issues for continuing work and inquinr.                   XRegioti
              Based   on   the   regional    diagnosis,    the    coordinator
              should    have    formulated    preliminary    directions    of    The   primary   problem   is   the   isolation   of   many    small
              strategy   which    reflect    defensible    judgments    about    communities,    especially    rural    communities    from    which
              crucial substantive issues  of  health  care,  issues  relating    physicians  are  slo@vly  disappearing,   and   their   disinclina-
              to  the  political  and   organizational   structure   of   the    tion  to  collaborate.  Corollary  to  and   underlying   this   is
              health  care  system,  and  key  actors   and   initiators   of    the  past  success  of   medical   education   in   selecting   and
              innovation in the health care system.                              training   physicians   to   want   to   work   in    sophisticated
                                                                                 hospital  settin-s,  thus  creating  strong  impetus   for   hospi-
              While  the  coordinator   should   be   capable   of   ar,,W'ng                  C,
                                                                    0            tals  to  compete,  even  within  communities,   and   to   attract
         for  these  directions  of   movement,   on   the   basis   of   the    physicians   by   offering   ever   more   highly    differentiated
         regional   diagnosis,   these   preliminary   views   about   strat-    and   costly   services,   without   careful,   credible   investi-
         egy   should   remain   developmental,   in   two    senses.    They    gation  of  community   needs   and   how   they   are   satisfied.
         should  take  account  of  the   issues   they   do   not   address,       The   function   of   RMP   should   be   (and    is)    through
         and  there  should  be   some   thought   as   to   the   means   by
         which  these  other  issues  may   come   to   be   addressed.   And    projects,   membership   on   advisory   committees,   and    core-
         they   should   be   responsive   to   changes   in   the   regional    staff  activity  to   facilitate   connections,and   collaborations
         diagnosis   which   come   to   light   in   the   course   of   RMp    among  elements  of   the   medical   care   system,   particularly
         activity,                                                               among   small   communities   and   particularly    among    physi-
              The   basic   question   is   "How   have   you   gone    about    cians.   The   connections    and    collaborations    should    be
         formulating    preliminary    strategies    for    systems    trans-    multiple  and  small-scale,  so  as  not   to   ruffle   too   many
         formtion?"                                                              feathers.
              *   Through what process have you gone?                            I  So   RMP,   for   example,   should   serve   as   broker    and
              *   What  is  the  substance  of  the  strategy   as   so   far    supplier  of  seed  money  for   the   merger   of   hospitals   in
                  developed?                                                     adjoining   rural   market   towns;   should   support   short-term
              *   Why  this  far,  and  no  further  -  or  why  so  far   in    in-residence  programs  for  GPs  at   the   Clinic;   shotdd   dot
                  this direction?                                                coronary    care    programs    around    the     State;     should
          Often,  the  best  way  of  gettinc,  at  these   issues   in   the    promote   outreach   programs   from    the    Clinic    and    the
          dialogue is through questions such as these:                           University;  should   use   the   RAG   and   its   committees   to
              0   Where   are   the   outstanding   strengths    and    weak-    involve   all   elements   of   the   medical   care   system   and
                  nesses  among  key  agencies  and  actors  in   the   medi-    representatives   of   its   consumers,   in   order   to   connect
                  cal care system?                                               small   communities    with    one    another    and    with    the
              0   What  are  the  patterns  of  alliance  and  conflict,  and    centers.
                  how are these changing?                                           The  object  is  to  build  larger   movements   toward   colla-
              a   For  key  actors  in  the  system,  and  for   the   issues    boration    and    more    ambitious    ends-in-view    from    the
                  they  regard  as  critical,  what  are   the   ends-in-view    success  and  the  fallout  from  many  small-scale   efforts,   in
                                                                                                                                                   11






                the process of learning  what  is  feasible  and  helping  the        The  coordinator  should  have  found  ways   of   includ-
               various  interests  and  groups  involved  to  assume  as  con-        ing actors and eleniei2ts of  the  region's  medical  care
                structive leadership roles as possible.                               system  identified  as  key  in  the  regional  diagnosis;
                   Some questions:                                                    where some of these cannot  be  included  at  the  outset,
                      Will  the  small-scale  collaborations  ever   get   big        the problems  about  their  inclusion  should  be  explic-
                      enough  to  make  an  impact  on  medical  care   in   X        itly  confronted  and  strategies  developed   for   over-
                      Region, and will they  happen  so  slowly  that  one  is        coming these problems over time.
                      forgotten  before  the  next  happens?   What   is   the        "Inclusion"  may.  be  indicated  by  participation  in  a
                      threshold level of scale and pace  for  facilitation  if    range of  RMP-related  activities,  including  involvement  in
                      it is to have a building effect?                            RMP coi-nmittees, in  project  work,  or  in  ventures  initi-
                      Have   you   taken   into   account   what   needs    to    ated  or   supported   by   RMP.,   The   difference   between
                      happen in order  to  get  the  Clhiic  and  the  Univer-    significant and pro  fon?7a  inclusion  must  be  resolved  by
                      sity really involved in  the  medical  problems  of  the    tests that vary from case to case.
                      smaller   communities?    How    much    "involvei-nent"
                      do  you  want  and  why?  Can  you   do   that   without        What is to be appraised includes:
                      confronting  the  "family  practice"  issue,  and  help-          Whether there has (or  has  not)  been  a  real  attempt
                      ing to  attain  a  viable  resolution  to  the  conflict          to  arrange  for  specific  people  to  be  included  in
                      among  the  Academy  of  General  Practice,   the   Uni-          RMP.  (Was  the  labor   union   representative   really
                      versity  medical  faculty  department  heads,  and   the          invited to  RAG  meetings?  Did  he  feel  invited?  Was
                      legislature?  Would  sponsoring  more  activity   within          there anything for him to do?)
                      the  allied  health  manpower  field  force  or  encour-        o How  well  the  attempt  is  related  to  the   co-ordi-
                      age a i-nore valid solution  to  the  general  practice-          nator's sense of starting conditions  and  his  strategy
                      family  practice  problem  -  or   just   convince   the          and  objectives  (which  depends   on   having   learned
                      MD's that RMP is against doctors?                                 those things first).
                      How  do  you  propose  to  respond  to   the   conserva.        9 How  explicit  the  coordinator  can  be  about  who  is
                      tive  stand  of  many  GPs,  particularly  in   southern          not  to  be  included,  and  under  what   circumstances
                      areas who don't see  what  RM-P  has  in  it  for  them,          those persons would or should be included.
                      and  who  feel  threatened  by  or  disagree  with  what        o How  much  the  coordinator  and   core   staff   learns
                      they hear?                                                        about  the  process  of  including   people   from   the
                  o   What  stance  will  you  take  toward  groups  currently          experience of doing it. (If  they  had  it  to  do  over
                      left  out  of  the  strategy  -  for  example,  hospital          would  they  do  it  another  way?  Are  they   increas-
                      a dministrators,   dentists,   mental   health   practi-          ingly  imaginative  and  increasingly  direct  in  their
                      tioners? Are there  parts  of  the  State  in  which  it          approaches to people?)
                      would make sense to do so?                                      o The  ii-npact  on  others  of  the  co-ordiiiator's  at-
                  o   Does  the  current  mix  of  efforts  respond,  at   the          tempts  at  including   people   (clumsy   or   skilled,
                      level  required,  to  the  serious  problems  you   have          relevant  or  irrelevant,  useful   or   useless,   well
                      identified -  i.e.,  to  the  problems  of  rural  medi-          planned and well understood or otherwise).
                      cine, isolated  communities,  care  for  the  small  but        A case in point is the following:
                      clustered  populations  of  minorities,  the   deficien-
                      cies associated  with  the  (otherwise  desirable)  pro-    Y Regio,z
                      liferation  of  specialist  physicians  and   the   dis-
                      appearance   of   family   physicians,   both   in   the        The RMP has taken the position  that  it  is  a  clearing-
                      central parts of the large cities and  in  rural  areas?    house for projects; it  solicits  and  processes  applications
                      If  you  cannot  envisage  any  adequate   response   in    from elements all  over  the  region.  RMP  is,  therefore,  a
                      flrst-round  activities,  how  do  you  plan  to   build    conglomerate  of  projects;  how  can  it   have   a   program
                      toward  such   a   response?   If   manpower   shortages    strategy for systems transformation or anything else?
                      seem  to   you   the   central   questions   about   the        But there  is  the  sense  of  need  to  involve  the  two
                      response,  how  do  you  plan  to  attack  the  question    counties  currently   disengaged   from   the   program.   The
                      of manpower over time?                                      preliminary strategy  has  impacted  on  the  starting  condi-
                  Often  the  formulation  of   preliminary   strategies   de-    tions  in  a  way  that  permits,   encourages,   and   partly
               pends  upon  the  involvement  of  key  actors  and   agencies.    specifies a revision in approach.

              12




          One   county,   medically   under   the   leadership   of    a   from  the  interaction,   planning,   bargaining   and   negoti-
          strong  physician,  has  no  involement  in   the   RMP   pro-   ating of the key actors.
          grarn. And there are  250,000  people  there.  The  belief  in      These   ends-in-view   are   the   specific    rearrangements
          the county is that  the  big  city  always  wins,  and  that's   sought  in  systems  transformation.  They,   too,   have   many
          where the money is.                                              qualities that are subject t  o   evaluation.   The    emphasis,
          In  spite  of  its  apparent  role  as  a  "clearinghouse  for   again, is first to  discover  what  attempt  has  been  made  to
          projects," the @NIP turns out to  be  operating  on  a  strat-   identify  these  qualities,  and  to  deal  with  them.  Evalua-
          egy  which  says,   "Get   every   major   actor   and   every   tion of specific  content  makes  sense  only  after  its  clear
          county active  in  RMP."  Their  tactics  are  based  on  this   and  more  or  less  agreed  what  has   been   attempted,   and
          strategy.                                                        the context for attempting it.
          The  major  physician  in  the   isolated   county   is   con-      The  following  are  examples   of   appropriate   questions:
          cerned   about   diagnosis   of   cancer,   and   about    the          Have the issues earlier  identified  as  crucial  in  the
          100-mile  round  trip  required  to  get   specialized   diag-          region  found  their  way   into   the   formulation   of
          nostic  screening  in  the  large  city.  He  is   encouraged,          ends-in-view?
          therefore,  to  propose  the  establishment  of  a  diagnostic      This is an  illustration  of  what  such  a  list  of  issues
          center in his county.                                            might look like:
          Some  of  the  relevant  questions,   especially   appropriate          Guidance to  get  people  into  the  health  professions.
          to -early involvement phases:                                       -   Coordination   and   involvement   of    the    voluntary
          1.  Is  the  investment  worth  it?  How  much  does  it  take          agencies.
          to "purchase"  involvement?  as  a  percentage  of  the  over-      -   The urgent need for dental care in the north.
          all  budget?  compared  to  the  costs  of  confronting  other      -   The  lack  of  out-patient  care   centers   except   for
          urgent  health  care  issues?  Are  there  other  excluded  or          emergency rooms.
          isolated   elements   of   equal   importance    (geographical      -   Essentially  no  preventive  medicine  is  done  in   the
          areas, professions,  voluntary  associations,  health  depart-          State.
          ments,  medical  societies,  hospitals,  or  a   combination)?      -   Too   many   community   hospitals   trying   to   become
          What  are   the   potential   future   consequences   (enmity,          medical centers.
          retribution, etc.) of  failing  to  try  to  involve  somebody      --  There  is   no   weekend   and   almost   no   night-time
          now? How does an effort  to  include  Dr.  H.  relate  to  the          medical  coverage   now   in   a   major   rural   county
          regional diagnosis?                                                     area."
          2.  What  are  the  signs  that  investment  has   been   sue-   Is the  RMP  engaging  some  of   these   issues   through   the
          cessful in  involving  Dr.  H  and  his  county?  How  do  you   deliberations     and     interactions     stimulated      among
          distinguish  pro  forma  from  significant  involvement?   For   elements  of  the  health   care   system?   "Engaging"   means,
          example, visibility at  RMP  meetings?  Attitudes  of  Dr.  H.   here,  facilitating  the   formulation   of   ends-in-view   and
          toward  the  proposals  of  others?  Willingness   to   permit   strategies adapted to them.
          some "teaching  days"  in  the  area?  Other  projects  coming      9   Certain  general  criteria   cut   across   regions   and
          out of the county? Willingness of Dr.  H  and  others  in  the          across  possible  activities  within  regions.  Questions
          county to lend voices in  support  of  RMP  activities?  Will-          about "relevance". of  particular  activities  apply  not
          ingness of  Dr.  H.  to  share  his  emergent  strategies  for          only   to   the   match    between    ends-in-view    and
          development of medical  care  system  in  his  county,  or  to          judgments  about  issues,  but  to  the  need  for   some
          participate with others in formulating such strategies?                 attention to these criteria.
          3.  Ends-in-View.  -  Out  of  interactions  of  key   actors,          -  Costs  of  care,  particularly  for  hospitalizatiotij
          ends-in-view  should  have  been   established.   These   must             extended   care,   and   costs   as   experienced   by
          confront at least some of the key  issues  earlier  identified             lower-  and  lowe.r-middle  income  persons   as   wet
          as  crucial  in  the  region.  On  the  level  of  substantive             as others.
          health  care,  they  must  confront  at  least  some  of   the          -  Quality of  care,  and  the  distribution  of  quality
          constant  health  problem  themes,  or  emergent   issues   in             of care across the region.
          health care.                                                            -  Access to care, and equity of        access  to  care,
           At  a  zone  in  time,  attention  shifts  from  the  problem             across  socio-econon-dc  strata,  minority   and   mal
          of "getting all the key actors active in  RMP"  to  the  prob-             jority groups, and geographic subregions.
          lem  of  formulating  the  more  specific   ends-in-view   and          Have  the  processes  making  for  inclusion,   discussed
          the  strategies  for  achieving  them  which  are  to   emerge          earlier, extended beyond formal         membership      in
                                                                                                                                          1 3





                        RMP  activities,  to  formulation  of  ends-in-view  and           -  An  outreach  center,   as   a   way   of   involving
                        strategies for achieving them?                                     major hospital  and  medical  school  in  the  problem,5
                        How are  priorities  formulated?*  Are  priority  issues           of  an  adjacent  ghetto?  Who  will   make   it   work@
                        being  confronted  explicitly  at  all?  By   whom?   Do           Who wants it?
                        priority  considerations  enter  explicitly   into   the           -  A  joint  coronary  care  project   as   a   way   of
                        deliberations  and  interactions  of  elements  of   the           encouraging   collaboration   and   rationalization   of
                        medical  care  system,  or  are  they  handled  by   the           planning   among   a   set   of   community   hospitals?
                        coordinator  or  core  staff  alone,  or  ostensibly  or           What will make it transcend its original focus?
                        really  left  to  Washington?  If  there  are  conflicts    Questions  about  such  strategies  will  focus  on  a   number
                        among elements  judged  to  be  crucial  to  the  region    of dimensions:
                        -  for  example,  conflicts  between   major   hospitals           Adequacy   of   scale   of   the   "solution"   to   the
                        and   medical   schools,   between   town   and    gown,           problem.
                        between  professional  providers   and   representatives           Feasibility of the methods proposed.
                        of  users  -  are  these  conflicts  allowed   and   en-           Appropriateness of the strategy to objectives         on
                        couraged   to   enter   into   the    fon-nulation    of           multiple  levels  of  the  activity  (e.g.,  substantive
                        priorities?  Does  the  coordinator  intend  to  attempt           health  impact,  as  well  as   systems   transformation
                        to  build  clusters  of  these  elements  into   working           ends-in-view; clarification of ends-in-view as  well  as
                        groups,  through   explicit   confrontation   of   these           involvement).
                        questions? If he is not doing this, is it  a  matter  of           Appropriateness  of  the  strategy  to  the  constraints
                        deliberate intent?  Is  he  working  -  temporarily,  or           and problems  perceived  to  be  underlying  the  issue.
                        as a matter of continuing  strategy  -  on  a  model  of           One of the questions to  arise  at  this  point  is  the
                        compartmentalization,   in    which    conflicts    over           question of "teeth." Is the issue one  that  will  yield
                        priorities and ends-in-view  are  not  allowed  to  come           best,  or  at  all,  to  voluntary  involvement  on  the
                        up,  except  within  limited  subsets  of  elements?  Is           part  of  the  key  actors   concerned?   Or   does   it
                        he "sub-regionalizing" in this sense?  If  so,  does  it           require  some  forms   of   sanction   and   complusion?
                        make sense to do so?                                               This is a question of ideology,  strategy  and  legisla-
                        Is  conflict  of  ends-in-view  being   handled   as   a           tive  mandate  for  RMP,  as  well  as   of   propriety:
                        matter  of  "dividing  up  the  pie"   among   competing           possibly some other agency is more appropriate.
                        actors, or is there  also  an  attempt  to  relate  such  Where the focus is  on  learning,  attention  will  go  not  only
                        judgements    to    shared    judgements    about    the  to  questions  of  this  kind  but   to   questions   about   the
                        urgency of health issues, or about  the  usefulness  of.  ways     in which  the  development  of  strategies  is  handled:
                        issues  as  ways  into  systems  transformation  in  the     -     Is  there  evidence  of  the  active  consideration   of
                        region?                                                            alternative  ways  of  achieving   the   same   ends-in-
                        Major  themes  of  RMP  activity  should  be   developed           view?
                        and stated.  These  should  be  iiot  merely  a  reflec-     -     Does the deliberation  over  strategies  carry  with  it
                        tion of what is con7moii  to  ongoitig  activities,  but           consideration  of  effectiveness  of  the  strategy   in
                        a  source  of  guidance  for  the  gei7eration  of   new           relation to the costs  of  carrying  it  out,  and  con-
                        actii,ities. Questioiis of priorities  an7ong  eiids-in-           sideratioii of  the  cost/effectiveness  characteristics
                        view   should   have   beeii   confronted,   through   a           of alternative strategies?
                        process iii which key actors iii  the  region  work  oii     -     Are   there   timetables   for    accomplishment?    How
                        their conflicting interests i?ot only oil the  level  of           realistic are they?
                        ownership  of  RMP  resources  but  oil  the  level   of     -     Has  there  been  consideration  of   ways   of   deter-
                        substantive health issues at7d strategies.                         ininin-  over  time  how  effective  strategies  are  in
                        How  appropriate,  acceptable  and  feasible   are   the           achieving  ends-in-view?  Tests   for   their   achieve-
                        strategies  being  developed  for  achieving  the  ends-           inclit?
                        in-view adopted? For example,                             Where  the  focus  is  successfully  placed  on   learning,   the
                                                                                  impact  of  such  questions  will   not   be   to   "grade"   the
                   This may be the first  time  that  themes  of  RNIP  activity  strategies at  this  zone  in  time  where  emphasis  is  on  the
               become explicit and that  questions  of  priorities  become  real  development  of  specific   ends-iii-view,   but   to   influence
               issues (often first stimulated by  conflicts  over  ownership  of  their   development    positively,    by    "accelerating"    and
               limited funds).                                                    "enricliing."

              14





           4.    Implementation.    -    The    process    of    implementation            jttcl,,iiients  of  impact,  which  can  help   to   guide   per-
       should    be    characterized    by    involvement    of    implementers            foriiiance in the course of the activity?
       in   selection   of   ends-in.-view   and   strategies   for   achieving            What   is   the   relation   of    the    regional    coordinator
       them;  and  by  a   relationship   of   coordinator   or   core   staff'            and his staff to  the  activity?  If  it  is  not  their  activi-
       to     i@nplenienters     ivlzich     permits     continuing      mutual            ty,  do  they   have,   in   relation   to   it,   a   continuing
       modification   of   strategy    and    end-in-view    and    of    iiii-            monitoring,       leariiing-evaluative       contact        which
       plenientitig activity.                                                              allows    mutual     modification     of     the     ends-in-view
           The    implementation    of    strategies    toward     ends-in-view            and    the    strategies    by    which    the     attempt     at
       may  take  the  form  of   core   staff   activity,   of   the   conduct            implementation is being made?
       of  specific  RMP  projects,  or  of  the   activities   of   coi-nniit-            How   compartmentalized   is   the   activity?   Is    it    con-
       tees   or   ad   hoc   groups,   under   the   aegis   of    RMP.    The            iiected  to  analogous  activities   in   the   region,   or   to
       end-in-view   and   the   strategy   may   be   specific    enough    to            activities   which   are    parts    of    the    same    program
       lend  themselves   to   ordy   one   of   these   kinds   of   activity,            strategy,    so    that    both    learning     and     concerted
       and   to   a   well-defined   unit   of    implementation,    or    they            action may occur, where appropriate?
       may  lend  themselves   to   a   widespread   cluster   of   activities.            What   is   the    relationship    of    these    processes    of
           For example,                                                                    implementation   to   the   overall   strategies    of    systems
                                                                                           clianc,,c   held   by   the   coordinator   and/or    his    col-
              End-iti-vieiv                          l@nplet?zentation                     laborators?    Has    the    coordinator    attempted    to    be
       To  foster collaboration and        A  coronary  care  project   jointly            explicit  about   these?   Is   there   an   effort   to   relate
           rationalization  of  planning       granted  to  the  13  hospitals,            them      to     particular     strategies     for      achieving
           among   13   community   hos-       requiring the use of com-                   particular   ends-in-view?   For   example,    to    connect    a
           pitals.                             mon facilities.                             particular  activity  as  a  feature  of   a   "master   plan   ;
       To encourage multi-level            Brokerage functions by core                     to  identify   a   particular   negotiation   as   part   of   an
           collaboration   between   two       staff, RiNfP support of one                 overall   strategy   which   seeks   to   involve   key    actors
           hospitals in  adjacent  rural       hospital staff member                       in  a  process  of   negotiation   over   their   interests   and
           communities.                        charged with working out                    conflicts  in  relation  to   the   system   of   medical   care.
                                               details of the merger.                      Is   the   coordinator   able   to   use   the   experience    of
       To  increase  the  "power   base"                                                   particular   activities   to   learn   from   or   to   influence
           of  the   medical   community   A series of projects, funded in
           "on the other side of the           that area, linked to major                  his overall strategies of systems change?
           mountain."                          medical institutions.                   There   is   one   side   of   the   question   of    impact    which
                                               Brokerage  activities.  Use   of    should  be   treated   separately   here,   because   it   involves   the
                                               RMP committees to estab-            impact    of    the    process    of    implementation,     which     can
                                               lish relationships crossing         reflect   back   both   on   the   formulation   of   particular    ends-
                                               the mountains.                      in-view   and   on   the   region's   capabilities   for   carrying   out
                                                                                   further   systei-ns    transformation    activities.    This    is    the
       Some of the relevant questions are these:                                   process   through   which    the    definition    of    accepted    ends-
           *  Are  initiators  and   leaders   of   the   activity   aware   of    in-view may shift.
              the   ends-in-view,   and   the   processes   leading    up    to        9   The    connections    established    and    reinforced    in    a
              their    formulation,    on    the    basis    of    which    the            particular    activity    may    lay    the    groundwork     for
              activity   actually   came   to    be    undertaken    by    RMP?            new  forms   of   collaboration,   e.g.,   the   joint   planning
           o  What    are    the    patterns    of    access    to    resources            of  a  coronary  care   unit   which   leads   to   joint   plan-
              required   for   implementation"   Is   there   a    basis    for            ning  of   a   range   of   common   facilities;   the   diagnos-
              judgments  to  be  made,   on   a   continuing   basis,   as   to            tic   screening   project   in   a    County    previously    cut
              the adequacy of resources to the task?                                       off   from   the   medical   system   of   the   region,    which
           o  Is   attention   given   to   the   possibility    of    shifting            leads   to   a   series   of   boundary-crossings.   Are    these
              definitions   of   ends-in-view   as   more   of   the    reality            things    happening?    Are    there     attempts     to     make
              of   the   discovered   system   comes   to   light?    Is    the            them happen?
              project  or   activity   leader   locked   into   a   potentially            Learning   from    an    implementation    process    can    lead
              stultifying view of what constitutes "success"?                              to  changes   which   facilitate   new   processes,   e.g.,   the
           0  What    constitutes    progress?    Are     there     operational            cumbersomeness     of     a     process     of     review     and
              tests   of   performance,   short   of    more    nearly    final            monitoring   can   lead    to    simplifications    which    make

                                                                                                                                                          15





                          it easier and more attractive  for  others  to  enter  the          The   evaluative   processes   adopted    by    coordinator
                          orbit of RMP activity.                                              and  core   staff   should   be   conducive   to   learning
                          Processes  of  implementation  can   display   or   enable          across   sub-regional    boundaries,    so    that    those
                          development  of  "role   models"   which   influence   the          engaged  in   analogous   activities   (continuing   educa-
                          character  of  new  activities   undertaken,   e.g.,   the          tion  for  GP's,  for   example)   can   learn   from   one
                          impact  of  Jim  Musser  as  broker-facilitator  on  other          another's   experience,   and   those   whose    activities
                          key actors in  the  North  Carolina  region,  or  of  Paul          are elements of a  larger  strategy  can  interact  in  the
                          Ward  in  California,  e.g.,  the  influence  of  the  few          light of that strategy.
                          emerging   medical   care   corporations   in   California
                          on  similar,  varying  approaches  to   medical   corpora-      5 . The     Developmental     Cycle.     Regional      programs
                          tions.                                                       develop iteratively,  -if  at  all.  Cycle  succeeds  cycle,  each
                     Questions    about    impact    of    implementation,     then,   growing   out   of,   but   resembling,   its    predecessor.    A
                  need also to  be  addressed  to  the  impact  of  the  process  of   regional  program,   seen   as   systems   transformation,   moves
                  implementation itself.                                               through  its  cycle:  casing  the   region,   planning   and   im-
                                                                                       plementing.   Then   through   another    cycle    widening    and
                      At  this  point,  RMPS  criteria   for   systems   transforma-   deepening  its  rings  of  activity.  The   evaluative   questions
                  tion in  the  region  take  the  form  of  meta-criteria  for  the   of  any  one  phase  continue  to  be  relevant;  only,  new  sets
                  evaluation processes carried out in the region.                      of  questions  are  'also  relevant  to  established   activities,
                          Without specifying  evaluative  criteria  to  be  used  in   and  to  other  sets  of  activities.  The  process  of   bringing
                          assessing  the  impact  of  implementation   on   any   of   new  elements  into  RMP,   for   example,   continues   even   as
                          the  levels  of   change,   RMPS   should   require   that   the   ends-in-view   emerging   from    earlier    processes    of
                          such  criteria  be  developed  and  that  they  be  appro-   inclusion begin to be carried out.
                          priate  to  the  ends-in-view  and   strategies   adopted.      The   most   relevant   new   questions   help   uncover    the
                     *    These  criteria  should  not  be   limited   to   program-   directions  of  change  in  the  scope   and   purchase   of   the
                          matic  criteria  (e.g.,  how  many  nurses  trained?   how   whole   program   as   it   moves   through   successive    inter-
                          many  calls  received?)  but  should  attempt  to   assess   actions  of,  the  process.  These  questions   are   of   several
                          change  at  one  or  more  of  the   several   levels   of   kinds:
                          change in substantive health care.                                  Is the process increasing its scope?
                      *   In  each  instance,   consideration   should   have   been              Is it increasing in the  overall  volume  of  activity,
                          given  to  the  choice  of  level  at  which   change   is              as    measured    by    actors    involved,     dollars
                          assessed,  aiming  at  health  outcomes,  then  at  access              1-nobilized,  number  of  separate  activities   under-
                          to  delivered  care,  and  so  on.   There   should   have              taken?
                          been  review  of  the   definitions,   test-methods,   and
                          measures    appropriate    to    the    end-in-view    and              Is there  a  widening  range  of  parties  involved  in
                          strategy involved.                                                      interaction  and  negotiation?  Is  the  level  of  ag-
                      *   With  respect  to   the   process   of   evaluation,   the              gregation    of    the    parties    increasing?    For
                          evaluative   framework   should   have   been    developed              example,  is  the  interaction  beginning  to   involve
                          collaboratively   between   the   regional   center    and              clusters  of  community  hospitals  rather   than   in-
                          the   implementing   agency.   There    should    be    an              dividual  community  hospitals?   Is   the   level   of
                          openness  to  modification,   through   the   process   of              aggregation  also   decreasing?   For   exw-nple,   are
                          evaluation,  both  of  the   implementing   activity   and              individual  physicians  as  well  as  medical   society
                          of  the  original  choice  of  end-in-view  and  strategy.              representatives  coming   to   be   actively   involved
                          This  openness  should  be   evidenced   in   the   demon-              in a  way  that  extends  the  scope  of  the  program@
                          strated  capacity  of  evaluative  activity  to  influence
                          the  planning  of  the  implei-nenting  process,  and   in              Is  there  an  increase  in  the  number   of   hea   t
                          the  evolution  of  the   concept   of   end-in-view   and              issues  engaged?  Is   there   an   increase   in   the
                          strategy  during  the  course   of   implementation;   and              coverage   of   the   region   represented   by   those
                          the  frequency  and  pattern  of  contact   between   core              issues  and  by   die   ends-in-view   and   activities
                          staff  and  implementing  agency   should   be   such   as              generated?  Within  each  phase,   the   map   of   the
                          to make that kind of mutual influence feasible.                         issues   confronted   and   their   location   in   the

                   16





               re ion  should  reveal  changes  of  the  followiiio    expression of interest of the  parties  approached  who  at-
                9
               kind:                                                   tended the iiieetiii-
                                                                                        cl,
                                                                          As the pro-r@iiii has begun to expand, its  emphasis  has
                                Regional Location
                                                                       shifted  away  from  the  categorical  approach.  The   RAG,
      Issues                                                           which began  with  30  physicians,  has  begun   to   change
                    x                          x                       composition to include  laymen.  In  view  of  the  relative
                            x                                          weakness  of  other  institutions,  includin(,,  the   State
                                                                       Health  Depirtiiient,  KRMP  has  moved  toward  a  control-
                                                                       ling position for health planning for the State.
                                                                          Concentration  ,it  the  beginning  has  been   on   work
                                     Phase I                           with  individual   physicians   and   community   hospitals,
                                                                       with an emphasis on education, viewed  as  the  easiest  and
                                                                       least threatening way in.  At  the  same  time,  core  staff
                                Regional Location                      became  involved  in  project-writing  for  individual  hos-
      Issues                                                           pitals,  KRMP  his  now   withdrawn   from   CCU   proarams,
                    x   x                                     X        except  for  continuing  education.   However,   a   similar
                                                                  -    effort  based  on  the  earlier   experience   (establishina
                    x   XI                             I               facilities,  loanina  equipment  to  communities  who  could
                                                             -         not afford to buy it) is now being carried out for  respira-
                            x   x                   x                  tory pro-rar-ns.
                                                                          Dr. P. now realizes that in his region, which is poor  in
                    x                                    x             physicians and clear in its referral patterns and which  has
                                                                       one medical  school  and  not  much  institutional  rivalry,
                                     Phase 2                           the provision of  continuing  education  to  physicians  and
                                                                       others is not enouch. What is needed is the provision  of  a
           Is the process increasing in  depth  and  intensity?        system of  care  and  appropriate  facilities  within  which
           -   Is there an increase  over  time  in  the  perceived    the fruits of education can be realized.
               importance, urgency,  and  ambition  of  the  issues       Here, since the structure of the program as  a  whole  is
               engaged and the ends-in-view formulated?                built around  the  coordinator,  the  development  of  ends-
           -   Is  there  an  increase  in  the  connectedness  and    in-view  becomes  very  much  the  development  of  his  own
               "clout" brought to  bear  or,  the  issues  engaged?    views of the issues that  need  to  be  confronted  and  the
           -   Is the level  of  aggregation  of  the  parties  de-    ends-in-view adopted. Is the  process  characterized  by  an
               creasing?  Are  individual  physicians  as  well  as    evolution of  issues,  ends-in-view  and  strategies,  which
               medical  society  representatives   coming   to   be    reflects learnincl,9
               involved in a  way  that  deepens  the  program?                 -ional diagnosis of  the  coordinator,  the  issues
                                                                          The re.,
         We  can  provide  an  example   of   the   development   oi   he takes to be important, the  ends-in-view  and  strategies
      ends-in-view and strategies  in  a  regional  program  as  it    to which he  is  committed  -  in  short,  his  own  systems
      be-,ins to go through a succession of cycles:                    rationale - may shift in  response  to  now  perceptions  of
                                                                       the discovered system of the region, as regional  activities
      The  K   Region                                                  bring that system into focus.
         Dr.  P.,  the  coordinator,  came  from   a   program   of       This learninc., may take the  form  of  an  explosion  of
      continuing education in  the  one  large  medical  school,  a    "rational" plans for the building of the  health  care  sys-
      program  of  continuing  education  for  GPs  which,  by  his    tem, by contact with  the  political  interests  and  powers
      own present  view,  was  not  too  successful.  He  began  by    of the real-world actors in the  system.  It  may  take  the
      seeing the creation  of  RMP  as  an  opportunity  to  expand    form of a  shift  in  priorities  about  health  issues,  as
      his  own  educational  program,  and  ob,tained  a   planning    previously "hidden issues"  -  for  example,  the  depth  of
      grant to create K-RMP. He  visited  local  medical  societies    inadequacy  of  health  care  in  ghettos  -  come  to   the
      over the region  and  with  them  set  up  a  program  around    surface. It may take the form of perceiving  the  extent  to
      tumor  re  stry,  coronary   care   units,   and   continuing    which  the  needs  of  physicians  and  community  hospitals
           I    gi  .
      ed                                                               in "have not" areas  are  inadequately  served  b  diffusion
         ucation. Boundaries of the  region  were  set  up  by  the                                                    y

                                                                                                                                 17





                of  the  technologies  and  research  findings  generated  at       Judgements  about   a   region's   progress   in   systems
                the major medical center.                                        transformation may be made on the basis  of  its  ability  to
                   In each instance, the   discrepancies   between    systems    meet critetia within any  given  stage  of'development,-  its
                rationale and  discovered  system,  at  the  regional  level,    rate  of  movement  from  stage  to  stage,  given  the  con-
                may lead to  the  refon-nulation  of  regional  diagnosis  as    straitits under which it is operating,-  and  the  level  of.
                well as of  ends-in-view  and  the  strategies  corresponding    scope, depth and learning  evidenced  by  its  overall  cycle
                to them.                                                         of development.
                   It is not reasonable to  set  uniform  standards  for  the       In point of fact,  most  of  the  RMP  regions  are  still
                periods  of   time   within   which   regions   should   have    primarily involved  in  the  problems  of  inclusion  of  key
                reached certain levels of  maturity  in  their  developmental    elements of the medical  care  system  in  RMP  activity  and
                cycles, just  as  it  is  not  reasonable  to  apply  uniform    on the formulation of   Ipreliminary  directions   of   move-
                standards  across  regions  to  the   time   periods   within    iiient and strategies. In  spite  of  the  number  of  opera-
                which  the  various   stages   of   development   should   be    tional projects, most regions are  only  beginning  the  work
                completed. On both  levels,  the  time  intervals  will  vary    of fitting projects into strategies  for  achieving  specific
                with regional conditions. The key factors  here  are  not  so    ends-in-view. Most are  only  now  at  the  stage  where  the
                much the size of the region as its complexity,  its  internal    formulation  of  themes  of  RMP  activity   and   the   con-
                connectedness   or   disconnectedness,   the   nui-nber    of    froiitation  of  questions  of  priority  among  ends-in-view
                conflicting or  disconnected  elements  within  it,  and  the    become feasible tasks.
                seriousness  of  their  conflicts  or  isolation   from   one
                another.
                   Elements that affect the speed of motion include:             Conditions for the
                   - simplicity  of  the  politics  of   the   medical   care    Central-Regional Dialogue
                     system.   Few   elements   to    be    connected;    few
                     conflicts to be resolved.                                     Having  sketched   out   a   iiational-regioiial   dialogue
                   - relative weakness  of  other  elements  of  the  system,    aimed at fostering learning in  relation  to  systems  trans-
                     permit@-  RMP  to  function  from   the   be-inning   in    formation,  there  remain  questions  about  the   particular
                     dominant  or  unusually  significant   health   planning    vehicles  through  which  such  a  dialogue  may  be  brought
                     role.                                                       to reality and the conditions  under  which  it  can  be  ef-
                   - relatively   high   degree   of   connectedness    among    fective.
                     elements of the medical care system.                          9  The  two  parties  to  the  dialogue  must  begin   with
                   It may  be  possible  to  establish  a  typology   of   RMP   some  commitment  to  and  understanding  of  the  goals  and
                regions  in  terms   of   their   potential   for   movement,    methods of this kind  of  evaluative  process.  The  require-
                similarities  in  strategy,  and  characteristic   types   of    iiients here relate both to  the  theory  of  the  evaluative
                activities  chosen  to  carry  out  the  RMP  program.  There    process and the role of the dialogue within it,  and  to  the
                are, for example, many  instances  of  efforts  to  stimulate    particular skills and  techniques  involved  in  carrying  it
                collaboration  among  comi-nunity  hospitals  through   their    out.
                joint  involvement   in   some   program   of   approach   to      0 Although  we  have  used  simple  words  like   "central"
                categorical disease; to  establish  outreach  arms  of  major    or  "RMPS"  and  "coordinator,"  the  parties  to  the   dia-
                medical  centers;  to  reach  isolated  subregions   tlirouoi    logues will be complex. On the regional  side,  the  dialogue
                programs  using  paraprofessionals,   continuing   education,    will be carried on by  groups  of  varying  kinds,  depending
                and  the  secondary  support  of  specialists,  Regions   and    on the makeup of those involved  in  carrying  initiative  at
                subregions differ as to  the  constraints  they  put  in  the    the regional level. In one region, it may be  a  "strong  man
                way of these  kinds  of  activity,  but  they,  too,  can  be    coordinator," his key  assistants,  and  from  time  to  time
                                                                                 others  that  he  may  wish  to  bring  along  in  order   to
                grouped in terms of the  seriousness  of  those  constraints.    involve or  educate  them.  In  another  region,  it  may  be
                 The  purpose  of  such  a   typology   would   not   be   so    the team  the  coordinator  has  been  trying  to  assei-nble
                much  to  permit  judgements  of  the  effectiveness  of  one    out of core staff,  certain  RAG  members,  and  certain  key
                region against another as  to  provide  guidelines  both  for    actors in the medical care system of.the region.
                RMPS and  for  regional  coordinators  as  to  the  rates  of     o On the  side  of  the  national  staff,  there  is  a  key
                i-novement it is reasonable  to  expect  in  a  given  region    requirement  for   continuity   of   involvei-nent   in   the
                and for a given kind of activity.                                dialo-Lie with region over long periods of  time  -  ideally,


                18





     over  the  life  of   the   region's   development   under   RNIP.    region  are  able  to  establish  with  one  another,  and  on  the
     The    requirement    for    continuity    becomes    particularly    extent to  N@,liich  the  dialogue  isfound  to  facilitate  learn-
     critical,   given   the   diversity    and    open-endedness    of    ing.
     regional  approaches  to  systems  transformation;  it   is   only        The  dialogue  requires  a   certain   frequency   of   contact
     out  of  intimate  knowled   e   of   the   content   of   earlier    between  central  and  regional   groups.   Given   the   rate   of
                                   9
     stages  of  development  that  central   can   be   effective   hi    movement  in   most   regions,   once   a   year   is   not   often
     dialogue with the region.                                             enough.  Within  the  interval  of  a  year,  too   much   happens,
        But,  given  the  realities  of  life  in  both   central   and    and  too  many  decisions  are   made   which   lock   the   region
     regional  bureaucracies,  continuity  of  this  kind  is   to   be    into  patterns  of  activity.  Frequency  of  contact   should   be
     achieved  not  through  one   man   but   through   small   groups    determined  by  the  time   required   for   the   coordinator   to
     whose members overlap in the course of time.                          take  significant  steps,  or  for  the   regional   situation   to
        From  central's  point  of  view,  the  small   group   permits    shift  in  si,,nificant  ways  that   mark   important   milestones
     the  inclusion  of  the  varieties  of  competence   required   to    in   the   stages of   systems   transformation.    Intervals    are
     carry  out  effective   dialogue   with   the   region   -   corn-    likely  to  var@. over  the  course  of  the   region's   cycle   of
     petence  to  question  and  respond  on  issues   of   substantive    developi-nent.   For   example,   contacts    might    be    estab-
     medical  care   and   on   issues   of   systems   transformation,    lished     around key  events  such  as  the  first  formulation  of
     and skills in the  evaluative  process  of  the  dialogue  itself.    regional  dia2nosis,   the   establishment   of   themes   of   RMP
        There will be  no  need  to  distinguish  the  central-regional    activities ard the first  effort  at  establishing  priorities  for
     dialogue   from   funding   decisions,   and,   concurrently,   to    specific  ends-in-view,  or  the  first  phase  of  experience   in
     move   away   from   the   usual   mode    of    central-regiona-I    implementing   a   specific   strategy.   Within   the   range   of
     contact, in which  the  region  displays  its  wares  for  central    frequency   indicated   by   "oftener   than    once    a    year,"
     and  central  and  the  region  then   engage   in   a   game   of    there  should  be  provision  for  flexibility   increases   if   a
     attack  and  defense.  For  the   central-regional   relation   to    representative   of   central   and   the   regional    coordinator
     be solely  or  primarily  in  this  mode  prohibits  learning,  in    can   maintain   contact   during   intervals   between    meetings
     the  senses  outlined   above,   and   makes   it   difficult   or    of central and regional groups.
     impossible  for   central   even   to   gain   information   about        The   central-regional   dialogue   offers   another   perspec-
     regional activities.                                                  tive  on  the  role  and  conduct  of  regional  site  visits,  and
        On  the  other   hand,   the   dialogue   requires   that   the    on the proposed process of anniversary review.
     RMPS  staff  be  capable  of  being   tough   with   the   region,        The   central-regional   dialogue   could   become   the   main
     raising  issues  hard  enough  to   be   heard   and   challenging    function of  the  site  visit.  The  site  visit  team  would  then
     the  region   in   the   light   of   findings   and   commitments    become  central's  party  to   the   dialogue.   Such   a   concept
     which emerge from the dialogue over time.                             would   answer   some   of   the   problems   currently   reflected
        In order to make these things feasible,       there is first  a    in  regional  and  central  reactions  to  the  conduct   of   site
     need to  model  the  roles  involved  and  to  set  the  tone  for    visits  -  for  example,  the  pattern  of  regional  display   and
     such   a   dialogue,   and   concurrently   to   set   apart   and    of   attack-and-defense   which   make   it   difficult   or    im-
     formally    distinguish    the    funding-justification    process    possible  to  find  out   what   is   really   happening   in   the
     from   the   central-regional   dialogue.   The   dialogue    will    region; lack of continuity     in the  site  visit  team;  lack  of
     surely   feed   into   RMPS   judgments   about   regional   fund-    feed-back to the region;  inability  of  the  site  visit  team  to
     ing,  but  should   be   formally   and   operationally   separate    respond   to   the    region    by    clarifying    or    modifying
     from the funding process.                                             central's  "signals."  There  are   also   significant   potentials
        Will  such  a  distinction  be  feasible,  given  the  tendency    of  the  site  visit  as  a  vehicle  which  the   central-regional
     of  the  region  to   view   central   as   monolithic   and   the    dialogue  may  help   to   tap:   the   opportunity   for   on-site
     region's  knowledge  that   funding   decisions   will   be   made    contact   with   regional   actors   and    agencies,    and    the
     by  central?  This  problem   is   comparable   to   the   problem    presence  in  the  region  of  persons   regarded   as   peers   by
     of  the  regional  evaluator   in   establishing   his   "helping"    many of those undertaking regional activities.
     role, in spite of the  fact  that  his  findings  will  be  influ-        There  is  the  further  issue   of   the   manpower   require-
     ential   for   decisions   on   project   funding;   indeed,   the    ments  RNIPS  would   experience   if   it   took   seriously   the
     problem  is  central   to   any   process   of   good   management    conduct   of   central-regional   dialogues   with   all   of   its
     in  which  the  manager  seeks   both   to   facilitate   learning    regions.  The  site  visit  team  concept,   in   which   outsiders
     and to exercise  control.  The  feasibility  of  the  effort  will    are  mobilized  alongside   central   personnel,   would   provide'
     depend  ultimately  on  the  good  faith  that  central  and   the    a  crucial  extension   of   central   staff.   But   the   concept

                                                                                                                                            19





                 would also require intensive  efforts  at  internal  training     seen,  whatever  the  intent,  as  a  funding-justificati(
                 and team-buflding for the site visit teams.                       process. The site visit  team  would  then  come  to  play
                                                                                   critical role in the anniversary review  process,  and  ti
                     With respect to  Anniversary  Review,  that  event  would     results of earlier phases of the central-regional  dialogi
                 have a very different significance if it were to function  as     would then provide the  basis  for  the  inquiry  conduct(
                 the  yearly   culmination   of   central-regional   dialogue,     and  the  judgments  made  in  the  course  of  anniversai
                 rather than as an isolated contact  which  will  tend  to  be     review.


























































                20






                 HOW OTHERS SEE REGIONAL MEDICAL PROGRAMS AND EVALUATION

                ALEXANDER M. SCFMIDT, M.D.                              favorable. It has occurred to me  there  are  three  principal
                  Dean, Abraham Lincoln School                          reasons for our aversion to the subject of evaluation.
                            of Medicine,                                 First, there is  the  general  feeling,  expressed  over  and
                        University of Illinois                          over to me, that "seat of the pants flying," if  it  gets  you
        Having corm in late, I  was  sitting  in  the  back  of  the    there, can't really be all that bad.  Over  the  past  decade,
       room, rather than  here  on  the  platform;  and  I  am  very    through  trial  and  error,  in  both  education  and   health
       pleased to have been able  to  hear  the  elegant  discussion    service, we  have  evolved  methods  that  we  think  we  know
       on  Regional  Medical  Programs   and   systems   change   by    to be both good and effective. It is my  belief  that  we  are
       Dr. Schon, I was  late  arriving  this  mornin-  because  our    far too content with this type of reasoning.
       three upper classes are returning this  morning,  and  I  met     Secondly,   evaluation   turns   out   to   be   hard   work,
       with  them  -  about  625  strong  -  for  a   re-orientation    expensive, time-consuming and technically difficult.
       session. This is something new  for  us.  Change  now  is  so     Lastly,  it  is   now   apparent   that   evaluation   is   a
       great, and the rate of change is so rapid  that  we  are  not    discipline  all  by  itself,  and  not  many   disciples   are
       only  orienting  our   incoming   freshman   class   of   225    available. It seems also  true  that  the  discipline  is,  to
       students, but are re-orienting  students  who  have  been  on    some  extent,  quite  backward  in  its   development.   Thus,
       vacation.  The  need  for  such  sessions  was  made  evident    application of the discipline is even more difficult.
       by their questions,  I  thought  as  I  was  driving  to  the                                  Regional    Medical     Programs
                                                                          The great importance to
       meeting. Among the questions asked were:                         of  evaluation  was   recognized   early   by   the   National
          "Is Cook County Hospital still alive and well?"               Advisory  Council  and  Review   Comn-dttee.   Many   of   you
          "How  many  medical  schools   are   there   in   Illinois    will   recall   the   numerous   early   messages   from   the
       today?"                                                          Division  about  evaluation,  and   the   resulting   anguish,
          And  finally,  "How  many  people  have   you   added   to    frustration and even outright hostility felt in  some  of  the
       the university police force?"                                    regions.  In  retrospect,  I  don't  think  anyone   concerned
          And my answers were respectively:                             fully appreciated the three  reasons  I  have  given  for  the
          "Not very."                                                   initial negative feelings about  evaluation.
          'Ten.
                                                                          During the  early  years  of  the  programs,  the  case  for
          and "Plenty."                                                 evaluation  was  argued.  A  significant  amount  of  research
          I was also musing that it  was  only  a  couple  of  years
       ago that I gave a  talk  entitled,  "Is  Evaluation  a  Dirty    in  evaluation  techniques  was  supported  by  the   Division
       Word?"  The  response  from   the   audience   then   clearly    (wisely,  I  think)  -  as  well   as   trai'ning      programs,
       indicated that they thought it was.                              conferences,  seminars  and  the  like  -  all   designed   to
          In the ensuing  years,  however,  it  has  become.apparent    provide needed expertise. As  a  result,  while  we  are  much
       that the wo'                                                     better off today than we were four  or  five  years  ago,  the
                   rd  "evaluation,"  like  some  other   words   we
       are now  hearing  almost  daily,  has  had  the  shock  value    problem still remains. I'd  like  to  discuss  RMP  evaluation
       worn off,  as  more  and  more  people  have  used  the  word    as  I  now  see  it  in  1970,  from  the  perspective  of   a
       in open public.                                                  member  of  the  Review  Committee  and   a   medical   school
          It is really too bad that evaluation got off to  a  rather    dean.
       shaky start  in  Regional  Medical  Programs.  From  time  to       To  go  back  for  a  moment  to  the  first  of  my  three
       time I have  tried  to  figure  out  just  why  it  happened.    reasons for our  aversion  to  evaluation,  it  seems  obvious
       Certainly  from  the  viewpoint  of  the  administrator  (who    to me that the trouble with "seat  of  the  pants"  flying  i@
       hopefully is a good manager)  evaluation  is  a  very  power-    simply  that  technology  has  rendered  it  totally  obsolete
       ful friend.  Evaluation  ranks  along  with  cost  accounting    except    in    bush    country.    Any.one   flying   a   plane
       and  program  budgeting  (two  other  dirty  words),  as  one    nowadays,  almost  anywhere,  can   pinpoin.t   his   location
       of the  most  powerful  management  tools  we  have.  We  all    accurately in  seconds.  And,  if  he  is  approaching  O'Hare
       Probably know this, and  believe  in  at  least  the  theory,    Field  and  wants  to  survive,  he  must  do  so,  and   know
       yet  our  response  to  the  word  is  too  often  less  than    how to use  the  proper  technical  devices.

                                                                                                                                     21





                       In point  of  fact,  the  methods  we  have  developed  by    avoided by this ploy, but such tactics  do  delay  decentra.
                   trial and error over the past 50  years  in  both  the  educa-    ization of project review  and  approval.  Happily,  I  thin
                   tional and health  service  fields  simply  aren't  doing  the    we  are  now  rapidly  overcoming  these  difficulties   an@
                   job, and  we  must  now  very  accurately  and  scientifically    using  Dr.  Schon's  analogy,  I   would   agree   that   th
                   determine our  position,  and  plot  a  new  course.  We  must    metapliase is upon us, and  the  diagram  on  the  board  ti
                   assess our education and  health  service  systems,  and  plan    your  right  really  is  applicable  now,  if   the   nuclea
                   to  make  needed  changes.  I'm  absolutely   convinced   that    chroi-natin represents the  evaluation  and  review  of  mos
                   Regional  Medical  Programs  are,  as  Dr.  Schon  has   said,    activities within Regional Medical Programs.
                   the best mechanism that now is available for doing so.               I recently  have  discovered  that  most  regions  realiz
                       Since  we  are  trying  to  make  changes  in  a  lot   of    that the National Review  Committee  is  only  a  collection
                   "traffic"  @  when  surrounded  by  agencies   and   organiza-    of individuals drawn  from  regions.  Several  regions  hav,
                   tions and individual  citizens  (often  irate)  trying  to  do    begun  developing  their  own  specialized   review   bodie@
                   similar things, I think the  O'Hare  Field  analogy  is  quite    which often  for  specific  purposes  are  better  than  th,
                   appropriate.  Anyone  trying  to  get  a   program   off   the    National  Review  Committee.  On  two  recent  site  visits,
                   .ground today had  better  know  precisely  and  scientifical-    was  provided  with  sounder,  more  detailed  reviews   an4
                   ly where  he's  going,  how  he's  going  to  get  there,  and    critiques  of  projects  than  the  National   Review   Corn
                   very importantly,  when  to  land.  "By  guess  and  by  gosh"    mittee has had  the  time  to  develop.  Some  regions  havi
                   isn't  good  enough  anymore.  And  we   should   reject   the    mounted their own  project  site  visits,  using  both  thei
                   argument  that  intuition  tells  us  we're  being   good   or    own experts  and  consultants  from  other  regions.  Severa
                   successful in medicine as in flying airplanes.                    of these project reviews were  so  good  that  the  Division
                      The importance of  regional  capability  in  evaluation  is    sponsored site visits added little to the  understanding  o:
                   made evident  by  the  current  efforts  of  the  Division  to    the project or activity. I'll  add  parenthetically  that  .
                   decentralize  authority  and  thus  enhance  regional  autono-    have  noted  a  regrettable   reluctance   by   regions   t(
                   my. We are moving  to  the  anniversary  review  systerr.,  to    respond to the criticisms of their own  experts  and  reviev
                   local  project  review  and  approval  and  to   greatly   in-    bodies, so that the same deficiencies existed, both  at  th(
                   creased  overall  regional  autonomy.  In  theory,   this   is    time of the  Division-sponsored  site  review  and  the  sub
                   very, very good.  In  practice,  there  are  definite  dangers    sequent  Review  Committee   and   Council   meetings.   Bu-
                   and problems.                                                     of great importance is the growing  realization  by  region!
                      Early  in  the  program  development,  the   Review   Corn-    of the value of a sound  review  process,  of  good  project
                   niittee often found that regions  were  passing  the  buck  to    planning, and  of  good  evaluation  (of  both  program  an(
                   the    Review    Committee     when     theoretically     they    projects),  demonstrated  by  the  willingness  to  hire  oi
                   shouldn't  have  been  doing  so.  Two   reasons   were   com-    borrow the expertise necessary to do these jobs well.
                   monly given for this avoidance of local responsibility:             As for the future,  I  agree  almost  completely  with  Dr
                      First,  regions  were  new,  and  local  expertise   simply    Schon's estimation of what  will  be  important  for  us  t(
                   wasn't available to allow local  determination  of  the  value    accomplish.   Anyone   following   developments    in    th(
                   of  the  proposed  program.  The   Review   Cormnittee   early    health field today, for example,  realizes  the  probability
                   on saw literally dozens of  projects  with  no  stated  goals,    that private medicine is in danger of pricing itself out  ol
                   no hope of  evaluation  and  really  no  hope  of  accomplish-    existence. As one  result,  during  the  next  few  years  ,
                   ment. Yet, this was the best  the  region  could  do  at  that    great effort will  be  made  to  control,  however  possible
                   time, in that particular field  of  endeavor.  This  was  very    the cost of medical care.  This  may  well  involve  Regiona
                   understandable,  and  led  to   the   establishment   by   the    Medical  Programs.  For  example,  there  is   currently   ,
                   Division  of  the  research   and   training   programs   men-    great rhubarb, which interestingly  enough  is  pitting  th(
                   tione,d earlier.                                                  American  Hospital   Association   against   the   AMA   an@
                      More bothersome, really,  was  to  receive  a  proposal  of    others,  concerning  the  idea  of   creating   "Professioni
                   much  poorer  quality   than   one   might   expect   from   a    Standards    Review    Organizations."    They     represent
                   particular region. This was  often  justified  by  the  region    expanded,  more  powerful  utilization   review   committees
                   on the basis of political  expendience:  it  would  be  better    It has been  proposed  that  these  organizations  be  estab
                   for  the  National   Review   Committee   to   turn   a   poor    lished by local  medical  societies,  which  would  then  bc
                   project down than for  the  local  program  to  run  the  risk    charged   with   evaluating   medical   care   and    makini
                   of alienating  some  faction.  I'm  sure  that  early  in  the    decisions as to reimbursement for this  care.  The  conflict
                   program,  many  local  fights  and  much  hard  feeling   were    arises over who  should  have  this  degree  of  power.  But


                   22





      more  importantly,  the  question  to  be   asked   is   whether    without  question,  be  turned  to  for  the  process  and   the
      or  not  county  medical  societies  have  available   to   them    expertise to do thisjob.
      the expertise to do this job. If this legislation    passes,   I       An important key to success in all of these      'things   is
      @vould imagine that at least in some areas,             Regional    good  evaluation.  Regional  Medical  Programs  are  still   the
      Medical  Programs  will  very  suddenly  be  in   the   business    best  instrument  our  society  has  created  to  do  all  these
      of  evaluating  not  only  their  own  programs,  but  also  the    jobs,  and  we  must  develop  the  necessary  capabilities.  As
      extent and quality of medical care delivered          in   their    the  action  moves  to  the  regions,  whether  we  succeed   or
                                                                          fail will depend  on  how  well  we  manage  the  tasks.  If  we
      region.  This  should  be  a  sobering   thought   to   a   good
      many  of  us  here  today.  I  believe  that   our   traditional    know  what  we  want  to  do,  we  also   have   to   know   how
      involvement  with  the  providers  of  medical  care  will  soon    well we are doing it. And  evaluation  in  these  terms  is  the
      be  put  to  very  good  use,  indeed,  as  we  get   more   and    ordy  possible  way  to  manage  our  efforts.  I  believe  that
      more  directly  involved  in  the  problems   of   quality   and    the  climate  is  now  favorable  for  evaluation.   In   recent
      availability of health care.                                        years  we   have   seen   significant   fractions   of   Federal
         If you  have  also  followed  the  life  and  hard  times  of    agency  budgets  earmarked  for  evaluation.   It   has   become
      medical  education,  you  know   that   while   we   need   many    accepted   practice   in   Regional    Medical    Programs    to
      more  physicians,   simply   graduating   more   of   the   same    budget specifically for  the  costs  of  evaluation.  Thanks  to
      type of physicians  we  now  have  is  not  thought  a  solution    Regional  Medical  Programs   and   other   agencies   such   as
      to our health care  problems.  We  are  told  that  our  current    the  National  Center   for   Health   Services   Research   and
      graduates  are  not  able  to  solve   the   problems   of   our    Development,  growing  numbers  have   been   trained   in   the
      health  care  system,  that  our  curricula  are   too   narrow,    science  of  evaluation.  If  these  experts  are  not   locally
      and the training base,  largely  the  urban  specialized  teach-    available,  they  usually  can  be  brought  in  as  consultants
      ing   hospital,   is   irrelevant   to   much    of    community    for a time.
      medicine.  Thus,   there   is   now   general   agreement   that       I  suspect  that  as  we  mature  as  a   program,   national
      medical   education   must   be   geographically    distributed,    conferences such  as  this  will  diminish  in  number,  and  we
      for  one  thing.  Also,  medical   schools   must   assume   in-    will  have  regional   conferences   on   evaluation,   regional
      creasing   responsiblity    for    graduate    and    continuing    training  programs,  and  the  emergence  of  the  word   "eval-
      medical  education,  and  they   must   train   a   variety   of    uation"  as   a   very   friendly,   commonly   used,   everyday
      types of  physicians  to  practice  the  profession  in  totally    household word - safe even for young children.
      new  ways,  Medical  schools   must   engage   more   and   more
      in  health  services  research.  Finally,  the  new   physicians                     PETER D. FOX, Ph.D.
      must  stay  in  the  state  where  they  were  trained,  and  be           Senior Economist, Office of Management
      paragons of  virtue  and  excellence.  What  is  common  to  all                            and Budget
      these goals is  the  involvement  of  what  is  now  called  the
      "private   sector"   of   medicine.   Indeed,   what    we    in       I would like to  begin  by  discussing  some  of  the  trends
      medical  education  are  looking  for  is  some  way  to  create    in Federal  health  expenditures   as   background   to   under-
      a  brand  new  education/medical  care   system   out   of   the    standing   the   context   in   which   all   health   programs,
      old separate systems of education and care.                         including  Regional  Medical  Programs,   are   likely   to   be
         In  the   past,   some   Regional   Medical   Programs   have    evaluated  in  the  next  few  years.  Federal  health   expend-
      looked  to  medical  schools  to  provide  expertise  for  plan-    itures are large.  They  are  expected  to  exceed  $20  billion
      ning  and  for  projects   such   as   training   programs   for    for the first time  this  fiscal  year  and  represent  over  10
      coronary  care  nurses.  I'm  convinced  that  medical   schools    percent of the total Federal budget.
      should  now  be  looking  to  Regional  Medical   Programs   for       Many  of  the  health  programs  were  started   during   the
      help  in  creating  the  new  education  and  service  mix,  in-    1960's  and  carry  with  them   the   potential   for   tremen-
      corporating  most  or  all  practicing  physicians  into  a  new    dous  demand  for   increased   funding.   For   example,   some
      system   of   teaching,   learning   and   service.   Our    new    80  comprehensive  health  centers,   funded   by   the   Office
      graduates,  like  many  physicians  now,  must  all   assume   a    of  Econon-dc  Opportunity  and   HEW,   are   now   in   opera-
      lifelong responsibility  for  learning  and  teaching,  for  re-    tion,  and  each  center  receives  an  annual  Federal  contri-
      newing their own  talents  and  slfflls  and  those  of  others.    bution  of  roughly  $2  nifllion.  Few  of  these  centers  can
      If medical societies or the  profession  as  a  whole  is  given    be  self-supporting   without   Federal   project   funds,   and
      or assumes  the  responsibility  for  setting  and  keeping  its    estimates  of  the  number   of   centers   required   to   meet
      own  house   in   order,   Regional   Medical   Programs   will,    health needs in poverty areas run as high as 800.

                                                                                                                                        23





                     Similarly,   the   Federal   Government    has    supported        Most  of  RMP  expenditures  are  for   three   types   c
                  staff  of  community  mental  health   centers   on   a   seed    activities-support  of  the  efforts  of  core  staff,  demor
                  money   basis.   Federally    financed    centers    now    in    stration  projects,  and  continuing  education  and  trainin
                  operation provide services to less  than  20  percent  of  the    programs. Consequently,  the  questions  that  the  Office  o
                  country.  Already,  the  authorizing  legislation   has   been    Management  and  Budget  is  likely  to   ask   of   RMP   ii
                  changed to extend  the  time  limit  on  the  grants  from  51    future years will largely  be  directed  towards  the  output
                  months to 8 years because  i-nany  of  the  centers  have  not    of those activities.
                  become  self-supporting.  Whether  these   centers   will   be        First, with regard to core staff. Are their activities ii
                  self-supporting after eight  years  is  questionable,  and  in    fact promoting  new  patterns  of  medical  care?  Subsidiar,
                  the  meantime,  increases  in  budgets  are  required   merely    to tl-ds, one can ask whether these  activities  are  success
                  to support existing commitments.                                  ful in  rationalizing  the  relationships  among  the  variou
                     Similarly, pressures  exist  to  expand  the  Medicare  and    organizations in  the  region  that  deliver  health  care  o
                  Medicaid  programs.   Many   medical   schools,   rightly   or    otherwise  impact  on  the  local  health  care  system.  RMI
                  wrongly, say they face  insolvency  if  they  do  not  receive    should  prevent  wasteful  duplication  in  training  program
                  additional  Federal  support.  The   pressures   for   Federal    and health care facilities.  Core  staff  should  both  foste
                  support of health  research  are  strong.  Some  people  argue    the  acceptance  of  new  technology  and  promote   new   ap
                  that  health  services  research  is  underfunded.  And,  last    proaches to  health  care  delivery.  For  example,  traininj
                  but not least, I see  estimates  that  Regional  Medical  Pro-    programs  for  physicians  assistants  and  other  types   o@
                  grams requires at least twice its  current  level  of  funding    nonphysician  manpower  are  now   multiplying   in   an   un
                  to be fully operative.                                            coordinated  fashion.  The  problems  of  the   location   o@
                     I will not attempt  to  project  the  actual  size  of  the    training facilities, training content, career  mobility,  an(
                  Federal health budget in the  coming  years.  However,  it  is    physician  acceptance  of  new  forms  of   manpower   shoulc
                  clear that we must do  better  with  the  funds  that  we  are    be  concerns  of  RMP.  Is  RMP   successful   in   achieving
                  already  spending.  This  is  the  environment  in  which   we    solutions to  these  problems?  Similarly,  is  RMP  bringinj
                  live, and  it  is  a  considerably  tighter  environment  than    about  proper  coordination  among  health  care  facilities'
                  the  one  to  which  we  were  accustomed  during   the   last    Has it achieved an appropriate  level  of  coordination  witt
                  decade.                                                           other    goverment    programs    such    as     Neighborhoo@
                     What,   then,   does   the   Office   of   Management   and    Health Centers  and  Comprehensive  Health  Planning?  Is  ii
                  Budget  expect  RMP  to  contribute?  The  goals  of   Federal    providing  a  vehicle  for  physician   acceptance   of   nem
                  health  programs  in  general  include  improving  the  health    forms of i-nedical practice, such as prepaid  group  practic(
                  status of Americans,  increasing  the  efficiency  with  which    or improved  referral  patterns,  that  may  lead  to  highei
                  care  is  delivered,  and  fostering  equity  of   access   to    quality or less expensive care?
                  medical  care.  RMP  is  expected  to  assist   in   achieving
                  these  goals,  and  in  setting  budget   levels,   OMB   must        We  also  expect  RMP  to  fund  only  those   demonstra.
                  assess  whether  the  $97  million  currently  spent  on   RMP    tion projects, continuing  education  courses,  and  trainin@
                  could  have  higher  payoff  if  spent   on   other   programs    programs that  are  an  integral  part  of  a  well-conceive@
                  such  as  Comprehensive  Health  Planning  or   the   National    strategy to satisfy the  health  care  needs  of  the  regior
                  Center  for  Health   Services   Research   and   Development.    rather than their essentially  reflecting  discrete  and  un-
                  We also assess the alternative of  not  spending  these  funds    coordinated  proposals  that  are  simply  related   to   the
                  at all.                                                           interests  of  the  persons  applying  for  funds.  Much  ha!
                     Measuring directly  the  impact  of  RMP  on  the  achieve-    been  said  about  the  diversity  that  exists  among  RNW's
                  ment of  these  objectives  is  difficult,  and  one  must  be    Such diversity is comniendable if it  represents  a  response
                  content  with  proximate  measures.  These   include   changes    to local conditions and factors. It is  less  commendable  ii
                  in  decisionmaking  procedures,  in  decision  outcomes,   and    it  devolves  from  confusion  over  objectives  or  how   tc
                  in  attitudes.  For   example,   RMP   should   be   able   to    carry out these objectives.
                  demonstrate  that  it  has  promoted  sharing  of  health  re-        Core staff should also avoid funding projects  or  train.
                  sources in a  mariner  that  contributes  to  better  care  or    ing activities that the market place is likely  to  undertake
                  increased  efficiency.  The  commonly   used   argument   that    without Federal support.  Nor  should  it  engage  in  activi
                  RMP   has   achieved   better   communication   among    those    ties that do not result in  efficiency  increases  or  medica
                  concerned with the health  care  system  does  not  in  itself    care  improvements  that  are  of  sufficient  magnitude   tc
                  justify the current level of expenditures.                        justify the related expenditures.

                  24





         In  evaluating  demonstration  projects,   market   criteria                RICHARD S. WILBUR, M.D.
        seem very appropriate.  Funds  for  these  projects  are  in-              Deputv Executive Vice President,
        tended as seed  money.  This  implies  both  that  the  funds                American Nie(lical Association
        serve to  stimulate  new  activities  that-  would  not  have
        been  undertaken  Nvithout   RMP   support   and   that   the       Thank  you vor,,- much for  the  kind  introduction  @iiid
        activities  are  sufficiently  attractive  that  the  medical    the chance to be here before a grotil)  of  people  in  whose
        market place is willing to  support  them  after  an  initial    work the AMA is so deeply   concerned.
        trial   period.   The   extent   to    which    RMP-generated       Now, when  I  s,@ak  for  the  AT\,IA  I  Should  make  it
                                                                         clear that I am spe,-,'%-iiia, for an or-aiiiz@itioii of pricticiii-
        projects are sustained  after  RMP  funds  are  withdrawn  is                                      t,                        t,
        an important measure of effectiveness.                           physicians, and as such we  are  concerned   primarily   with
          Training  programs   should   increase   the   ability   of    the problems of pri@ticiiic, physicians  -  in  the  maiitiil
        health professionals to deliver health services  by  bringing    aspects of the  deanery  of  Heart[)  care  services  -  tire
        them  up  to  date  on  recent  technological   developments.    people who acttiall\r touch the patient.
        They  should  also   increase   the   productivity   of   the       Our major pro,,)!@iiis ire those  (,eiieral  problems  you
        medical sector, Health professionals  should  be  trained  to    know so well:
        use  new  capital-intensive  devices.  Physicians  should  be       The manpower shortage ... And I'd  like  to  say  this  is
        trained  to  use  new  forms  of   medical   manpower.   Non-    mainly a  sliorta@e  of  frontline  troops.  If  any  of  you
        physicians should  learn  new  functions  so  that  they  can    have  followed  the  development  of  armies  over  the  last
        substitute for  physicians  and  thereby  pern-dt  physicians    century or so,        know that,  in  years  gone  by,  if  a
        to spend time on  activities  that  only  they  can  perform.    general had I 00,000 troops, he  could  usually  expect  that
        As with projects, one  might  ask  why  individuals'  courses    most of them would -et into the fiolit  on  the  day  of  the
        require  RMP  funds  and  why  they  are  not  supported   in    battle.
        the private market place.                                           As  you  no\@,  kTiow,  if  a  general  has  an  army   of
          Program  evaluation  has  at  least  one   function   other    100,000, he's Itick@. if I 0,000 of these  men  actually  get
        than  simply  leading  to  decisions   on   whether   program    into the fightin-.  Or maybe they're unlucky.
        expenditures  are  justified.   In   particular,   evaluation       What we find in the niedic@il  care  field  is  very  much
        should result in redesign of the program.  Thus,  if  certain    the same sort of thiiio. Everybody wants  to  be  a  consult-
        program  activities  appear  to  be  successful  and   others                         c'
        not,  the  successful  activities   should   be   emphasized.    ant. There's little reward or recognition for the  priiiiir@,
        Similarly,  one  would  hope  for  infomiation   to   improve    physician, and he sometimes gets  a  little  lonely  when  be
                                                                         thinks of all those niaht house calls tic  has  to  make  and
        the functioning of  even  the  most  successful  elements  of    all the people  who  are  planning  on  how  he  should  make
        the  program.  The  Office  of  Management  and   Budget   is    still more of them.
        interested in the quality of evaluation  at  all  levels.  We       So the manpower shortage  to  us  is  that  of  the  prac-
        are interested in evaluation of the  total  program,  of  in-    ticing  physician,  although  there  are  many  other  short-
        dividual RMP'S,  and  of  individual  projects  and  training    aaes as well,
        efforts.  We  expect  evaluation  to  lead  to  assisting  or    t,
        phasing out weak  programs  or  projects.  While  an  overall       Second, we have a  concern  about  quality  of  care  that
        cost-benefit analysis of RMP may  not  be  feasible  at  this    has been expressed before. today.
        time, we would like to see  a  few  clear  successes  perhaps       Third,  is  the  much  discussed  problem  of  cost  which
        along  with  quite  a  few  ambiguous  ones.   We   recognize    needs no elaboration before this audience.
        that there will always  be  some  mistakes  and  failures  in
        this  type  of  program,  although  one   would   hope   that       And,  of  course,  we  have  the  problem  of  remembering
        these would be as few in number as possible.                     that we are dealing  with  human  beings.  Problems  of  cost
                                                                         and  human  factors  are  certainly  widespread   today.   At
           The health care problems  of  this  country  will  not  be
        solved  simply  by  expanding  Federal  programs  to  support    least we know the  feelings  of  our  college  students,  who
        health services or  by  increasing  the  supply  of  existing    are well versed as  to  their  educational  institutions  and
        manpower  and  institutions.  RMP  should  be  at  the  fore-    the loss of human factors  in  some  of  the  larger  medical
        front  of  promoting  the  changes  required  at  the   local    teaching institutions.
        level to make the health care system and  its  related  tech-       The problem pertinent  to  this  meeting  is  how  to  get
        nology  more  efficient,  more  effective,   and   more   ac-    information to those doctors who, so to  speak,  are  in  the
        cessible to the American people.                                 front lines.

                                                                                                                                     25






                     We  find  that  what   the   practicing   physician   needs    and other institutions. In times gone  by,  clinical  research
                  most  is  help  in  solving  the  common  problems   of   com-    was  done  by  a  clinician,  who   took   an   afternoon   or
                  mon diseases in common people.                                    evening off,  or  even  went  on  a  sabbatical  and  did  re-
                     Sidney  Garfield  of  Kaiser  Permanente,  in  writing  for    search. He could then 'use this research in his practice.
                  Scienti.fic   Ameiican,   speaks   of   the   "slightly   sick       As  research  became  more  complex,  this  evolved   until
                  and  the   worried   well."   These   people   make   up   the    there  were  two  people,  the  clinician  and  the   research
                  volume of patients that these doctors see.                        man.  They  got  together  at  lunch  time  or  shared  common
                     We  need  help  in  knowing  how  to  see   them   in   the    meetings to exchange information.
                  office, and possibly even more  so,  we  need  help  in  keep-       It's often said now that we need  a  third  man,  a  trans-
                  ing these patients well so they don't  have  to  come  to  the    lator, who could tell the  research  man  what  the  clinician
                  office. And  what  is  even  more  important,.  we  need  help    was  doing,  and  who,  more  particularly  in  recent  years,
                  in  keeping  them  out  of  institutions,   particularly,   of    was able to explain to the clinician  just  what  it  is  that
                  course, hospitals.                                                the researcher did and what  it  means  to  the  clinician  in
                     Being in an institution is not only  bad  for  the  budget,    terms of his practice.
                  as Peter Fox has just stated, but  it's  bad  for  a  patient,       I think we need a fourth person  too,  and  he  is  in  the
                  and he should avoid it if at all possible,  Being  in  a  hos-    field  with  which  you  are  concerned   -   the   communica-
                  pital is bad for a  patient's  morale.  And  as  many  of  you    tion of this information to the clinician after it  is  trans-
                  know, it's  where  most  of  the  side  effects  of  treatment    lated so that he can understand it and can  use  it.  Just  as
                  occur.                                                            important  is  the   communication   back   to   the   medical
                     The physician needs help  in  the  prevention  of  disease.    school,  of  the  kind  of  information  that  the   clinician
                  I  don't  mean  by  this   just   immunization,   because   we    really needs, so that it can be translated, at  least  at  the
                  don't see  many  diseases  these  days  that  are  preventable    clinical level - not at the basic research level  -  into  the
                  by   immunization.   Maybe   it's   because   we   have    im-    kind of research that  is  going  to  help  him  do  a  better
                  munized people so well already.                                   job.
                     We  don't   need   help   in   delivering   more   physical       We  need  a  two-way  street.  Let  me  use   an   example.
                  exams. I won't  bore  you  with  the  argument  of  whether  a    Many  of  you  know  John  Hogness,  a  former  Dean  of   the
                  physical  examination  is  worth  the  money  spent  or   not,    Medical  School  at  Washington.  He   wrote   a   very   good
                  except to say it's a highly debatable subject  -  and  that  I    article and gave a superb  speech  about  the  time  he  spent
                  intend  to  go  on  getting  them.  As  a  good  internist,  I    a couple of weeks filling in for  a  general  practitioner  in
                  could do no other.                                                the rural areas of Eastern Washington.
                     But we do need help in the  real  problems  that  face  us,       He's not  quite  sure  how  i-nedical  care  in  that  com-
                  the things that cause people  to  get  sick  and  to  come  to    munity  fared  during  the  period  he  was  there,   or   how
                  the  doctor's  office   -   tobacco,   automobile   accidents,    much he helped it, but he is  very  sure  that  he  learned  a
                  pollution, the lack of exercise,  nutrition  -  in  the  inner    great  deal  that  was  of  value  to  him  in  training  more
                  city, too little nutrition,  and  in  groups  like  this,  too    physicians.
                  much  nutrition  -  urban   crowding,   sanitation,   alcohol,       It's a tN@-o-way street, with  which  we  need  your  help,
                  drugs,etc.                                                        because  we   need   practical   planning   -   planning   for
                     And then, of course,  what  causes  us  the  most  trouble,    people and not just for census tracts.
                  is the psychic stress of our  day  which  drives  the  patient       We  need  to  avoid  overspecialization  in  planning.   We
                  into the doctor's office.                                         need  to  avoid  the  problem  we  run  into  when  we   solve
                     This is  where  we  need  help.  And  as  we  look  at  and    one problem  and,  as  the  old  saying  goes,  we  cause  two
                  evaluate the ability  of  RMP  to  plan,  it's  not  just  how    others. It's all  well  and  good  to  solve-the  problems  of
                  many coronary care  units  are  set  up,  but  by  working  on    uremia with renal  dialysis  and  kidney  transplants  and  to
                  the  causes   of   disease,   how   many   people   could   be    make  these  procedures  generally  available,  but  in  doing
                  prevented from ever  having  to  use  a  coronary  care  unit.    so we diminish  the  budget  available  for  housing  and  for
                     The value to  the  provider  is  in  helping  him  to  take    pollution  and  for  the  other  problems   of   health   care
                  better care of  people.  And,  as  I  said  before,  he  needs    which may be more important to more people.
                  help   in   dealing   with   the   comi-non   diseases   which       As we  solve  the  problems  of  keeping  the  elderly  and
                  common people develop commonly.                                   the chronically ill  alive,  we  build  up  the  problems  in-
                     Now,  there  is  an  historical  problem   that   has   de-    volved in the  population  explosion.  As  these  people  stay
                  veloped with  getting  this  help  from  the  medical  schools    alive there is less for the rest of us. Or, if you believe  in


                  26





        the  theory  of  population   zero,   the   longer   you   keep   an    Senators   and   their   personal   and   committee   staffers,   top
        older  person  alive  by   now   modern   techniques,   the   longer    officials   in   HEW   and   through    the    department's    health
        it is before a new life may enter this world.                           agencies,    and    many    representatives    of    voluntary    and
            We   must   have   overall    planning,    not    single-problem    professional health association and consumer groups.
        planning.                                                                  We  also  go  to  a   select   mafling   list   of   medical   and
            The   doctor   himself   must   always   be    concerned    with    science   writers   on   major   newspapers   across   the   country.
        his  own  individual  patient.   He   cannot   be   concerned   with    Hardly  a  week  goes  by   that   MAN   is   not   quoted   in   the
        whether  keeping  his   individual   patient   alive   is   a   good    press,  or  on  radio  or  television  news.   So,   the   public   I
        thing  or  a  bad  thing.   He   is   committed   to   keeping   the    represent is far wider than the medical community.
        patient alive.                                                             In  Washington,  my  beat   is   primarily   the   political   and
            And,   therefore,   you   must   be   concerned   with   whether    economic  side  of   health.   I   regard   my   role   as   one   of
        his  success  and  your  success   will   actually   make   this   a    evaluator.  I  watch  what  is  going  on,  attend   hearings,   read
        poorer world for all of us to live in.                                  testimony,  talk  to  dozens   of   policy   makers   both   on   the
            Our  evaluation  may  not   be   as   sophisticated   as   those    Hill  and  in  the  Administration,  listen  to   the   reaction   of
        which   you   have   heard   and    which    will    be    discussed    other  groups,  then  when  the  time  is   ripe   attempt   to   set
        further  at  this  meeting,  but  we  will   certainly   ask:   Have    events into some perspective for my readers.
        you  in  your  planning  helped   physicians   to   deliver   better       As    to    Regional    Medical    Programs?    I    joined    the
        care  and  have  you  helped  the  people   of   this   country   to    magazine   during   the   days   of    the    DeBakey    Comniission,
        live a better life?                                                     and  began   to   cover   Capitol   Hill   during   the   House   and
            If  you  have  done  this,   then   our   evaluation   of   your    Senate    hearings    on    Regional    Medical    Complexes.    That
        planning will be that it is a total success.                            was  the  time   of   the   mighty   89th   Congress   when   passage
                                                                                of  a  new  federal  program  was   regarded   as   the   answer   to
                                                                                all   the   problems   which   plague    mankind.    You    know    -
                            JOHN M. BLAMPHIN                                    take  one  RMP,  add  water   and   stir.   Voila!   Instant   health
                Assistant Bureau Chief - Washington Office                      care  for  all.  I  believe  that  approach,  incidentally,  did  you
                             Medical World News                                 a great deal of harm. But more about that later.
            For   several   years   now,   I've   been   covering    medical       So   I   watched   what   went   into   the   Congressional   mill
        meetings and listening to speakers get        up  and   give   their    and  I  saw  what  came   out.   I've   been   watching   and   eval-
        papers. It  always  seemed  so  easy.  I  figured  all  you  had  to    uating,   and   reporting   your   progress   ever   since.   Evalua-
        do  was  to  step  up  to  the  microphone   and   say   with   some    tion  of  RMP  takes  a  simple  format  for  me.   I   merely   look
        degree   of   confidence   -   "First   slide   please."   And   the    to  see  what  progress  you're   making   toward   a   single   goal
        projectionist would do all the rest.                                    -  the  delivery  of  high  quality  medical  care   -   the   latest
            So  here's  my  big  chance  and  I  didn't  bring  any  slides.    medical  science  has   to   offer   -   to   patients   with   heart
        As  a  result  I  stand  up  here  this  morning  with  a   bit   of    disease,  cancer,  stroke  and  related  diseases.  I  also  look  to
        hesitation,  knowing  full  well  that  of  all  present   at   this    see  in  which   ways   the   means   developed   to   deliver   that
        conference,  I   know   the   least   about   the   intracacies   of    specialized  care  are  also   used   to   cope   with   other   more
        RMP and the science of evaluation.                                      general health problems.
            But   before   I   jump   into   the   topic    of    evaluating       Over  the  past  five  years,  I  have  performed   this   evalua-
        Regional  Medical   Programs   from   the   public   side   of   the    tion   by   reading   your   annual   reports,   by   hearing    your
        fence,  I  thought  I  should  tell  you   something   of   what   I    representatives     before     congressional     conunittees,      by
        do,  and  how  I  view  my  own  relationship  to   what   you   are    talking   to   RMP   officials    in    Washington,    by    visiting
        trying to do.                                                           regions  whenever  I  can   find   the   time,   and   by   listening
            As   you   know,   I    work    for    Medical    World    News,    and   reading   what   others   say   about   RMP   -    the    usual
        McGraw-Hfll's  weekly  news   magazine   for   physicians.   It   is    routine a reporter goes through covering his beat.
        my  job,  simply  put,  to  tell  doctors  what  is  going   on   in       In  the  course  of  this   evaluation,   I   have   formed   some
        Washington   that   is   important   to    their    practices.    My    opinions  about  RMP   and   health   care   in   general   which   I
        primary  audience   is   about   200,000   physicians   in   private    believe  are  shared  by  a   great   many   people   in   Washington
        Practice and on hospital staffs.                                        these  days.  To  me,  the  quality  of  care  and  the  way  it   is
            In  addition,  we  go  to  about   5,000   people   around   the    delivered  go  hand   in   hand.   One   is   useless   without   the
        country  who  subscribe  or  are   on   our   "Freebee"   list.   In    other.   It   does   no   good   to   tune   an   automobile   engine
        Washington,  the   list   includes   dozens   of   Congressmen   and    with   new   points   and   plugs,   and    add    a    fancy    fuel

                                                                                                                                                   27





                  injection system, if the car's transmission is shot,  and  the    heart attack has  dropped  better  than  60  percent.  I  cal
                  tires are bald. It's the same in the  health  biz.  Tu@ng  the    that delivering health care to people.
                  skills of  physicians  and  hospitals  to  a  high  degree  of       In  region  after  region  RMP  has  successfully   brough
                  quality and efficiency  is  no  good  if  the  system  through    the  normally  fragmented  elements  of   the   health   corn
                  which those skills  are  passed  on  to  patients  has  broken    munity together to talk about the state  of  health  care  ii
                  down.                                                             this region, to admit that  gaps  and  weaknesses  exist,  t(
                     It  is  my  opinion  and   the   opinion   of   others   in    identify  them  and  then  to  plan  ways  to   improve   thi
                  Washington  I  spoke  to  about   this   before   coming   out    situation.
                  here, that a federal program such  as  yours  which  is  using       For the private sector  of  medicine  this  is  a  tremen.
                  the  taxpayer's  money,   cannot   stop   at   providing   the    dous  accomplishment.  For  the first  time      ' in    man)
                  physician, the hospital,  and  other  providers  with  quality    sections  of  the  country  there  are  evolving  systems  oi
                  tools. It must also do what is necessary  to  see  that  these    health care which are  more  than  the  sum  total  of  theii
                  tools are applied  to  patients.  Many  of  us  have  the  un-    parts.  This  advance  may  well  have  laid  the  groundwork
                  comfortable  feeling  that  there  are   those   in   Regional    for the development  of  new  health  care  systems  such  a@
                  Medical Programs  who  feel  their  responsibility  has  ended    the  Health  Maintenance  Organizations  now   being   touted
                  at  the  conclusion  of  a  continuing  education  course,  or    by  HEW,  which  could   never   come   about   without   the
                  after  the  technicians  have  installed  the  coronary   care    change in atmosphere which RMP's have created.
                  equipment.                                                          But lest  you  think  I  have  been  completely  snowed  by
                    Nevertheless,  I   have   seen   evidence   that   you   are    RMP, I must also say that I believe  this  success  has  been
                  moving - albeit  slowly  -  toward  a  patient-centered  goal.    spotty, and has worked better  in  rural  areas  and  smaller
                                                                                    communities  than  it  has  in  highly  complex  metropolitan
                  About  a  year  and  a  half  ago,  for  example,  Dr.  Robert    areas. As a colleague  of  mine  said  to  me  at  lunch  the
                  Headly,  a  Bowman  Gray  cardiologist,  took  me  on  a  tour    other day, "The real  test  of  the  RMPs  is  not  in  being
                  of several  small  hospitals  in  the  State  of  Franklin  in
                  Western North  Carolina.  During  our  visit,  to  the  50-bed    able to  organize  care  where  it  is  unorganized,  but  to
                  C.  J.  Harris  Community  Hospital,  the  doctor  showing  us    organize care  where  it  is  disorganized.  And  tills,"  he
                  around asked Dr. Headly  if  he  would  look  at  one  of  his    added, "just hasn't happened."
                  patients who was  in  the  hospital's  coronary  care  unit  -      Since I have followed  the  RMP  for  five  years,  perhaps
                  staffed  incidentally  by  nurses  trained  with  RMP   funds.    more closely  than  other  reporters  in  Washington,  I  un-
                    Dr. Headly  readily  agreed  and  a  few  moments  later  in    derstand the  significance  of  what  you  have  achieved.  I
                  the  hallway  I  heard  this  exchange:  "If  you  can   you'd    know also how difficult  it  is  to  evaluate  this  type  of
                  better send her  on  in,"  said  Dr.  Headly.  "You've  gotten    groundwork in  terms  of  the  usudl  morbidity  and  mortal-
                  her out of a failure this  time.  But  if  it  happens  again,    ity indices.
                  she'll probably go fast. If we give her a valve, she's got  an     But  let's  speak  about  the  public  for  a   moment.   By
                  857o  chance."   The   local   doctor   pondered   a   moment,    public  I  mean  just  about  everyone  outside  of  RMP  and
                  then asked  "When  can  you  take  her?"  "In  a  day  or  so,    the  health  professions.  Included  might  be  the   present
                  I'm sure," replied  the  younger  man.  He  pulled  a  pad  of    administration    in    Washington,    the    Congress    and
                  paper  from  Ms  inside  pocket  and  began   making   a   few    voluntary  and  professional  health  organizations.  How  do
                  notes. "You talk  with  her  family  and  I'll  let  you  know    they evaluate RMP?
                  tomorrow,  maybe  tonight,  when  to   bring   her   down   to     The  present   Administration   evaluates   RMP   in   terms
                  Winston-Salem."                                                   of national goals. And so far as health  is  concerned,  this
                   I don't  know  the  outcome  of  that  case,  but  I  suspect    means  using  federal  money  first  and  foremost   to   in-
                  the  exchange  between  rural  physician  and  medical  center    fluence  changes  in  the  organization,  and  delivery   and
                  specialist saved a life. I do  know  that  it  probably  would    financing of medical  care.  It  also  means  spending  money
                  never have  happened  were  it  not  for  the  North  Carolina    in a more  flexible,  non-categorical  way.  One  needs  only
                  Regional Medical Program.                                         to  read  the  Ad"nistration's  Health  Services   Ii-nprove-
                   I  also  understand   that   after   the   @4P   helped   put    ment  Act  of  1970  to  get  the  Administration's   present
                  coronary care units in  several  of  the  small  hospitals  in    evaluation of RMP.
                  Western  North  Carolina  and  also   established   a   mobile     I  must  also  say  in  passing  that  the  Adninistration's
                  coronary  unit  staffed  with   rescue   squad   workers   and    committment  to  health   care   so'far   hasn't   manifested
                  aided by local physicians  -  that  the  mortality  rate  from    itself in much  more  than  rhetoric.  If  the  President  is

                  28




         serious about solving the so-called health care crisis, he  is     schools, voluntary health associations  -  all  of  whom  are
         going to have to substibute money for talk-.                       looking for a piece of the action themselves.
         Now,  to  the  Con-ress:  Here   the   evaluation   has   been     Now   why   is   there   so   much    cynicism    surrounding
         more  simplistic.  It  also  demonstrates  how  RMP  got   off     RMPS?  I  think  my  friend  Ed  Friedlander  would  boil  it
         to a bad start. It goes like  this:  "If  you  spend  all  the     down  to  a  problem  of  communication,  of  providing   the
         1-noney we appropriate,  you're  doing  a  good  job.  If  you     facts from which others can make evaluations.
         don't you  must  be  dragging  your  feet."  As  Rep.  William     Certainly,  not  everyone  can  spend   time   looking   over
         Springer of Illinois said during  the  hearings  on  extension     the  operating  projects  within  one  or  more  regions   to
         of RMP two  years  aco,  "The  initial  legislative  testimony     learn first hand what  is  going  on.  So,  I  would  suggest
         was  presented  before  this  conunittee  to  justify  a  pro-     that  you  consider  very  carefully  how  you  justify  your
         gram for a billion  dollars,  which  turned  out  three  years     existence to your publics. In  a  nutshell  they  -  and  I'm
         later to  have  spent  S85  million."  But  he  also  shows  a     talking about  those  in  Washington  -  want  to  know  what
         great deal  of  insight  into  the  nature  of  new  programs,     you are doing for people,  for  patients,  for  constituents.
         and especially of @NIP's when  he  says,  "I  think  it  ought     When they hear you talk  in  your  own  jargon  of  regional-
         to be brought out  here  that  what  we  get  in  the  way  of     ization,  of  cooperative  arrangements,  of   closed-circuit
         landslide testimony  here  is  a  selling  job  and  snow  job     TV  and  other  gadgetry,  they  are   going   to   go   away
         claiming that something can be done immediately."                  shaking  their  heads.  Maybe   you   can   translate   those
          It  has  been  my  observation  that   the   promises   which     matters into improved delivery  of  patient  care,  but  they
         were made for health  care  at  the  beginning  were  made  in     can't. As far as they  are  concerned,  you're  off  in  some
         terms  of  the  original  DeBakey  Commission   report.   They     other world.
         were not significantly modified  as  the  procram  itself  was
                                                    t,                      Let  me  give  you  an  example  of  what  I  mean.  About  a
         modified by the Congress.  As  a  result,  the  evaluation  of     year ago during the  budget  crunch  for  RMP,  the  Illinois
         the  promises  and  potential  was   made   before   the   RMP     program, like a lot of others, wrote letters to its  Senators
         even  began.  And  when  you  try   to   evaluate   what   you     pleading for reinstitution  of  RMP  funds  cut  out  by  the
         can't  even  define  -  and  who  in  1965  could   define   a     House,  and  describing  what  would  happen  to   the   pro-
         region   in   understandable   language   -   you   get   into     gram if the money is not  put  back.  The  letter  was  well-
         trouble.                                                           written,  telling  how  the  region   had   been   organized,
           Only  recently  have  members   of   the   legislative   and     about the  progress  toward  achieving  regionalization,  and
         appropriations   committees   begun    to    understand    the     how  after  months  of  planning,  grant  applications   were
         subtleties of RMP. But  they  too,  like  the  administration,     pending to  put  the  program  into  gear.  The  letter  said
         expect  RIMP  to   pay   more   attention   to   problems   of     that  if  the  money  wan't  forthcoming  that  those  grants
         delivering  health  care.  As  Sen.  Ralph   Yarborough   said     could not be awarded.
         earlier this year  in  introducing  his  bill  to  extend  the
         %MP   program:   "Explicitly,   the    extended    legislation      But  nowhere  did  it  say   what   the   money   would   be
         provides  that  Regional  Medical  Programs   concern   itself     used for in terms of  helping  the  people  of  Illinois  who
         with  improving  the  organization   and   delivery   of   all     just  happen  to  be  Senators'  Percy  and  Smith's  consti-
         health  services,  and  strengthening   our   primary   health     tuents.
         care system."                                                       You  must  remember  that   members   of   the   House   and
           In the meantime, RMP  still  has  to  prove  itself  on  the     Senate get dozens of letters a  day  pleading  one  cause  or
         Hill.  One  Capital  Hill  staff  member  told  me  the  other     another. Most of these are handled  by  aides  and  only  get
         day that  "Regional  Medical  Programs  have  failed  to  take     a cursory review by the boss. If you had been  sitting  in  a
         on broadened responsibility  in  health  care  and  have  hung     busy office on Capitol  Hill  reading  that  letter,  knowing
         tenaciously to heart  disease,  cancer  and  stroke  labels.,,     nothing about  the  concept  of  regionalizatidn  and  caring
            True,  you  have  pointed  out  that  without  the  disease     less, what would your evaluation have been?
         categories, medical center  specialists  may  be  less  likely       To me it  boils  down  to  this:  If  you  communicate  the
         to  participate.  There  may  be  some  truth  to  that.   But     proper information, and by proper I  also  mean  that  it  be
         many on the Hill read it as  a  cop  out  and  as  an  attempt     honest,  and  communicate  it  in  the  context  which   your
         to maintain the status quo.                                        audience  can  understand,  that   is   patient   care,   the
            Individuals  within  health  associations   in   Washington     evaluations  you  get  will  more  likely   approximate   the
         are also skeptical of  RMP  progress.  But  these,  of  course     true state of RMP. You will, of course,  still  have  critics
         represent  vest   interest   groups   -   hospitals,   medical     who will say  that  only  total  federal  control  of  health
                                                                                                                                        29





                services can eliminate the gaps in delivery and quality  we    program in health care in the eyes  of  the  administration,
                now face.                                                      of the Congress and of  the  public,  those  actions  in  my
                                                                               estimation  are  going  to  have   to   be   directed   more
                   In the  long run, you will  be  judged  by  the  changes    and  more  toward  improvement  in  the   quality   of   the
                that occur in the  quality  and  delivery  of  health  care    system  through   which   the   best   medicine   known   to
                which result from your activities. And in the  future,  for    science reaches the patient where  and  when  he  needs  it.
                Regional Medical Programs to survive  as  a  major  federal    At least that is going to be my yardstick.































             30







                                                    LUNCHEON REMARKS
                                                    HAROLDMARGULIES,M.D.
                                      Acting Director, Regional MedicafProolranis and Service
           When  Dr.  Glasgow  was   introducing   me,   he   talked    can't  see  how  they  could  have  been  on  the  moon.   My
           about my task of drawina to(,Cther theIthreads   of    the   TV set can't pull in New York.  How  could  it  pull  in  the
           morning session  as  a  "herculean  task."  I  remembered    nioon?" They talked to  a  man  in  North  Carolina,  and  he
           some of the labors  of  Hercules.  One  of  them  was  to    said, "You know, if you  got  on  an  airplane  and  went  to
           clean the  Aegean  stables.  I  guess  you  do  remember,    Ashville and  then  came  back  and  I  saw  you  again,  how
           don't you?                                                   would I know you had been in Ashville?"
           I was also thinking this morning as  I  listened  to  the     I'd like to also point  out  the  fact  that  1,200  of  the
           descriptions of evaluation and of  the  Regional  Medical    nation's  1,700  newspapers  carry  daily  horoscopes  -  and
           Proaraiiis, of something  that  I  had  almost  forgotten    a  few  years  ago  90  did.  And  last  year  there  were  2
           about - a flashback to my early youth where  I  once  saw    million ouija boards sold - which  is  the  greatest  bonanza
           a picture of a man standin- on the deck  of  a  ship.  He    in the history of the business. Now, those  are  just  casual
           was the greatest archer in the world, sort of the  modern    observations,  but  they  are,  at  the  same   time,   symp-
           Robin Hood. He was  standing  on  the  ship's  deck  with    tomatic of a drift  toward  the  mystic,  toward  the  intui-
           the deck going up  and  down.  There  was  an  empty  keg    tive,  toward  the   doubtful,   toward   the   seeking   for
           floatin                                                                      I
               in the water with a little  cork  in  the  bung,  and    solutions which , re non-rational,  at  the  same  time  that
           his job was to hit that cork with the arrow.                 we are trying to look very strongly  in  rational  directions.
           I  had  the  same  sensation  when  someone  was  talking      When  I  looked  in  the  New  York   Pmes   this   morning
           about evaluating the Regional Medical  Programs,  and  it    the present status of important legislation was  listed,  but
           gave me the opportunity as I sat  there  to  decide  what    as  always  the  Regional  Medical  Programs  were  not  men-
           the title of may talk should be, because I frequently give   tioned. I  think  this  also  helps  you  to  appreciate  the
           talks without titles and then somebody wants one.            environment in which we are functioning.
           I have selected one for this one. It is as  follows:  "By      Aside from  these  general  statements,  I  think  we  must
           the Time I Get to Where It's At,  It's  Always  Where  it    recognize that  in  looking  at  the  Regional  Medical  Pro-
           Was." Which seems highly reasonable.                         gram  or  any  other  health  activity  from  the  evaluative
           Before  I  comment  on  the   general   discussion   this    point of view, we have to enter  into  a  game  to  which  we
           morning, I must say it was superb  from  every  point  of    are  generally  unaccustomed.  The  health  profession   does
           view. The thanks should go not only to  the  participants    not characteristically evaluate  its  own  practices  or  its
           but to Pete and the people who have helped  him  put  the    own institutions. It may do so on an  individual  basis,  but
           program together. It's off to an awfully good start.         on a broad basis, little or not at all. If  you  doubt  that,
           I would also like to say a few things about  the  general    try  to  look  at  your  own  program   sometime,   or   look
           atmosphere, sort of overall environment in which  we  are    nationally  at  what  you  have  available  if  you  want  to
           thinking about evaluation, whether  in  Regional  Medical    measure the influence of  some  health  event  or  the  corn-
           Programs or in other areas. I  was  particularly  charmed    munity. And look very  hard  to  see  if  you  can  find  any
           by the sense of determination  to  deal  rationally  with    information that will  allow  you  to  say,  "Here  is  where
           systems which have often  been  dealt  vath  intuitively,    we  stand  and  here,  as  a  consequence  of  what  we   are
           and the expressed preference for the  rational  over  the    doing, is where we are  going,  and  here  in  retrospect  is
           intuitive,                                                   where we have been." It  s  an  astounding  fact  that  those
           At the same time, I had to realize that there is a  drift    kinds of data bases do not exist.
           in this nation, a preference for mysticism over  thought-      We do not,  generally  speaking,  relate  our  institutions
           fulness, which expresses itself in interesting ways.  The    to  the  processes  which  we  have  been  discussing  today.
           Knight newspapers did a -survey  not  long  ago  of  some    We do not relate  general  health  problems  to  the  efforts
           1,700 readers to  see  what  they  thought  about  people    in  which  we  invest.  And   we   allow   ourselves   little
           landing  on  the  moon.  Some  of  them  had  interesting    managing room  to  set  up  a  conceptual  basis  for  future
           comments which give you a sense of what at least part  of    planning.  To   ask   Regional   Medical   Programs,   as   a
           the country feels at the present time. One lady said,  "I    consequence, to enter into this kind of a process  is  to  be
                                                                                                                                    31





                   asking   for   perhaps   even   more   than   any    of    you    here     into  it explicitly. If it is true - and  I  believe  it  is  -  th@
                   realize.                                                                   RMP   has   changed   its   purposes   in   the   very   process   (
                       I  think  at  this  point  it  might   be   useful   for   me   to     serving  its  initial  requirements,  then  to  carry   out   evalu@
                   recapitulate  as  best   I   can   what   was   said   this   morning,     tive activities on what it has been doing runs the risk
                   @vfiicii,  I  believe  has  set   the   tone   for   this   afternoon.     being      arc@va'I   rather   than   programmatic.   We   must    ver
                   discussions.    There    were    some    general     themes,     which     carefully   distinguish   between   what    will    be    of    valu
                   were    sustained    throughout    the    entire     morning.     They     historically and what will be of value for the purpose
                   were  things  I  have  said  to  you  and  you   have   said   to   me     building a new kind of a program.
                   during  the  past   several   months.   They   included   the   deter-        In  fact,  if  I  were  to  make  a   generalization   about   RM
                   mination   that    this    health    program,    like    any    health     -  and  we  all  know  that   evaluation   and   generalization   ai
                   program             his  tli@-  backing   of   the   federal   govern-     dangerous   companions   -   I   would   say   that   certainly   on
                   anent  -  and   probably   those   with   other   kinds   of   backing     of  the  great  potentialities  of  RMP  is  to  create  an  envirot
                   -- must be ij-1vol\,,?d kN,itl-i i3iil)rovii)-  the  organization  and     meiit ai-nong  health  care  providers  that  will  allow  it  to  b
                   delivery of health care.                                                   as   well   sensitized   as    possible    to    the    indeter@nab]
                      Words   like   heart    disease,    cancer    and    stroke    were     changes  that  are  coming  -  a  kind   of   preparation   for   th
                   brought  up  eitliei-   on   a   pro   forina   or   on   a   somewhat     apocalyptic   -   which   may   be   a   rather   tall   order.   An
                   wistful  basis.  This  by  no   means   suggests   that   people   are     since  RMP  reaches   that   far,   it   has   to   be   thought   c
                   no   longer   concerned    with    the    specificities    of    these     perhaps not too loudly, but at least above a murmur.
                   diseases,  nor  does  it  suggest  that   our   legislation   or   our        I  also  heard  this  morning  that  we  must   be   careful   no
                   legislative   mandate   has   suddenly   changed.   What    it    does     to   ruffle   too   many   feathers.   I    suspect    my    emphasi
                   say   is   that   priorities   emerge   in   a   society,   and    the     would  be  to  ruffle  the  feathers  but  don't  smash   the   biri
                   priorities  which   have   emerged   in   this   society   are   being     in  the  process,  because  I  think  some  feathers  will   inde,ei
                   enunciated  from  several  points  of   view   -   from   the   public     have to be ruffled.
                   point  of  view,   from   the   evaluative   point   of   view,   from        I  heard  something  else  -  the  need   to   look   at   variou
                   the  fiscal  point   of   view,   from   the   provider's   point   of     aspects  of  the  evaluative  process,  the   justification   aspect
                   view.  We  were  told  that  if   we   are   to   look   at   Regional     for  instance,  which   from   my   point   of   view   is   a   ver
                   Medical   Programs   and   evaluate   their   usefulness,   it    must     urgent  one   because   I   have   to   defend   and   justify   the
                   be  with  tile   prior   determination   that   we   have   set   some     Regional   Medical   Programs   at   all    times    in    Rockville
                   value  structures,  that  @ve  have  a   clear   statement   of   what     Washington, or wherever I may be representing them.
                   it is @ve wish to be in the health care system.                               I  heard  about  evaluation.  It  has  something   to   do   witl
                     I   thought   the   major   speaker   of   the   morning   described     methods   of   controlling    what    happens,    of    confonnatioi
                   magnificently  the  problei-ns  involved.   He   told   us   that   we     to  standards  -   standards   which   are   established   according
                                                                                              to   the   requirements   of   the    program    and    which    ar(
                   must   have   some   gIoals  which  are   clearly   stated,   that   are   determined locally - and of a process which is              . cafle(
                   determined  locpll\  by   the   re-ioiis,   and   that   these   goals
                                                                                              learning  in  which  we   perceive   new   ways   of   doing   thos(
                   cannot   he   controlled   with   meaning   and   with   purpose    on     things  which  we  do,  and   their   relationship   to   where   w(
                   some  kind  of  a  central  basis.  He  described  to  us  a   pattern     wish  to  go.  I  was  especially  pleased  to  hear  references  t(
                   of   Regional    Nlcdical    Programs    which    have    their    own     the   need   to   avoid   replacing   the   objectives   which   at(
                   special   knowledge   and   their    own    special    issues    which     being  sought  with  the  measures  utilized   to   reach   a   goal
                   must   be   introduced   into   their   planning   flexibility,    and     It  is  so  easy  to  establish   measures   -   and   we   in   the
                   Vehicle    must    desinii    well    defined    goal-related     pro-
                   @,r,,ifni-iiatic  is  that  cat)  be  evaluated.  He  said,   as   did     bureaucracy  are  fully  capable  of  doing  that  -  so   easy   to
                   others,  that  sot-ne  of  this  i-nay  have  to   be   retrospective,     establish  measures  that   describe   how   to   get   there,   and
                   and  that  makes  me  t  little  nervous,   because   I   think   very     then   concentrate   on   meeting   those    measures    so    fully
                   little  of  what  we  can  evaluate  retrospectively   is   going   to     that why we established them disappears.
                   have   much   meaning   when   we   got   to   the   prospective   end        I  used  to  see  that  a  great  deal  when  I  was  working  ir
                   point,   and   a   reconsideration   of    what    Regional    Medical     Asia,  where  the  great  game  was  to  have   a   five-year   plan
                   Programs   are   all   about.   In   fact,   the   issue   of    which     The  five-year  plan  would  contribute  -   whether   it   was   in
                   direction RMP  ",ill  take  is  either  air  evaluative,  a  political     education  or  in  agriculture  or  in  health  -  what  was  to  be
                   issue,  or  a  social   issue   which   must   be   looked   at   very     done  from  point   zero   to   the   end   of   that   point.   One
                   Attentively,.                 this was  a  little  bit  vague  in  the     could  be  very  sure  that  at  the  end  of  the  first  year,  20
                   presentations  'L  1-iis  morning  or  there  wasn't   time   to   get     percent  of  the  goal  would  have  been  reached,   at   the   end

                   32



          of   the   second   year   40   percent,   and   so   fortli.    Any am  confident   that   whatever   Regional   Medical   Programs
          variation  from  that  was  easily  corrected   by   replacing   the must do, they will have  to  do  it  more  rapidly  than  seems
          fellow  who  was  doing  the  reporting.  This   is   one   way   of at all reasonable.
          getting  where   you   want   to   go,   but   it   does   seem   to There  are  two  other  aspects  of  the   evaluative   process
          ei-nphasize the measures more than the goals.                      that I  would  like  to  speak  about.  If  RMP,  as  you  have
          There   also   appeared   during   the   course    of    the    dis- heard this morning, is to be as diversified as  it  should  be,
          c,ussion   some   reference   to   the   need   to    examine    all and if it is to maintain the flexibility which is  one  of  its
          alternatives  instead   of   simply   taking   advanta,,e   of   the great assets, and if it is  to  mobilize  those  providers  who
          opportunities.  And  again  I  was  thinking  a   little   of   what are always going to have  to  be  involved  with  the  delivery
          might  be  called  a   kind   of   Mae   West   approach   to   this of health care services, it is going to do it in  a  variegated
          thing.  A  young  lady  asked   Mae   West   how   she   could   get fashion.  And  that's  fine.  But   this   presents   a   great
          out  of  the  particular  dilemma  she  was  in.  She   said,   "I'm difficulty  for  us  in  Regional  Medical  Programs   Service.
          sure  I  can  never  do  what  you  have  done,  which  is  to  find Because while this kind of  an  activity  is  going  on,  there
          a   man   who   loves   me   and   has   $10,000   and   who   would must  also  be  a  sense  of  coherence,  which  if  not  main-
          buy   me   the   beautiful   kind   of   a    mink    that    you're tained,  will  make  the  RMP  look  like  another  process  in
          @vearing."  And  Mae   said,   "Well,   honey,   you   could   think fragmentation and in activities  going  off  in  a  variety  of
          about  getting  10,000  men  with   a   dollar   each."   You   see, directions. As a consequence, I  think  it  is  essential  that
          she   understood   the   discovery   system   and   she   understood we  establish  more   effectively   within   RMPS   and   among
          the ways in which you do develop alternatives.                     the  Regional  Medical  Programs  an   understanding   in   the
          I   suspect   that   most   of   us   were   thinkin-   during   the process  of  programmatic  development  and  in   the   process
          course  of  the  morning  what  all   of   this   discussion   about of  pursuing   programmatic   goals,   a   communication   net-
          evaluation  meant  with   reference   to   the   process   that   we work  which  lets  everybody  know  what  is   going   on   and
          are   going   through   at   the   present    time    in    Regional which gives a  better  understanding  of  the  expectations  in
          Medical   Programs.   The   process   is    something    which    is RMPS,  with   reference   to   what   represent   HEW   overall
          called anniversary review,and   will   obviously   place   a    very goals.
          different  kind   of   burden   and   emphasis   on   the   Regional For  me  to  pretend  to   you   that   this   government   or
          Medical  Programs.  If  I  do  nothing  else  in  the   process,   Iany  government  can  support  activities   without   our   own
          would  like  to  say   that   I   am   convinced   that   what   you concept  of  what  those  goals  should  be,  and  without   at
          heard  this  morning  is  so  highly   consistent   with   what   we least  a  broad  kind   of   framework   in   which   we   will
          anticipate   in   the   process   of   decentralizing   the   RMPIS, function, is to be misleading.
          that  it  could  easily  be  played  back   again   to   you   every Now, it is not  likely  that  at  any  point  we  will  be  so
          morning  for  several  days  to   make   sure   that   the   message foolish as to direct the RMP's  to  do  a  specific  number  of
          is clear.                                                          things. We would fail in that effort.  But  I  think  you  need
           There  can  be  no  question  not  only   that   F.MP's   will   be to  join  with  us  in  the  interpretation  of   what   really
          given  the  prerogative,  but  it  will  be   demanded   that   they matters  in  this  country  in  the  health  care  system.  And
          establish   programmatic   directions,   and   within   those   pro- this you have  heard  over  and  over  again.  You  heard  that
          grammatic   concepts,   establish   projects   which    specifically people  are  concerned  about  the  costs  of   medical   care.
          fit  those  programs.  The  core  activities  will   all   have   to You  heard  that  people  are   concerned   about   access   to
          move  in  that  direction.  There's  no  question  that  this   will medical  care.  You  heard  that  they  are  concerned  -   and
          be the  way  in  which  we  will  have  to  go.  There  is  also  no I'm not sure in what way  this  is  true  -  with  the  quality
          question  that  there  will  be  a  need   to   evaluate   the   ef- of medical care.  In  the  public  mind  quality  has  a  lower
          fectiveness of   that   whole   process   and   that   the   way   in order of  priority.  I  think  access  and  cost  are  far  and
          which  you  evaluate  it  will  have   to   be   based   upon   your away the greater considerations.
          understanding  of   where   you   wish   to   go   and   what   your You  also  heard  from  the  people  who   are   looking   at
          senses of value are.                                               the evaluative system, where  we  will  have  to  go  and  what
            If there is anything to add to w
                                                   hat    was    said     this will have to be  done,  on  the  basis  of  what   we   have.
          morning,  it  is  that  there  was  probably  less  emphasis  on   aSimply flooding  more into  the  system  is  no  longer  going
          sense  of   expediency   than   I   would   have   liked   to   have to be the answer.
          heard.  It  came  out.  It  was  mentioned.  It  is  that  part   of  You  heard  a  very  strong  inference,  which  I  join  in,
          the  evaluative  process  that  had   to   do   with   how   rapidly from the Office  of   M.anagement  and   Budget,   that   there
             gs  are  to  be  accomplished.  But  at  the  present   time,   Iwill have to be  greater  selectivity  in  what  is  supported
                                                                                                                                           33





                      and  a  readiness  on  your  part   to   abandon   what   doesn't        There  are  times  to  use  judgment.  There  are  times   I
                      really  seem  to  be  working  very  well.   This   will   entail    exercise  your  own  knowledge  of   what   is   going   on   an
                      some risks,  but  careful  risks.  You  will  have  to  eliminate    what  needs  to  go  on.  And  what  you   will   have   to   do
                      what  appears  to  be   ambiguous,   and   give   heavy   support    establish   evaluative   techniques   which   anticipate    even
                      to  what  appears  to  be  a  strong  direction   in   which   to    and  then  be  ready  to  prove,  when   you   arrive   at   th,
                      i-nove  toward  the  kind  of  goals  which  we  have   embraced.    event,  that  you  have   done   what   is   necessary   in   th
                      Now, if we can manage  this  variety  of  activities  in  such  a    process   of   projection.   Anything   that   depends   entirel
                      way  that  we  can  interpret  them   coherently   in   our   own    upon  what  is  here  and  now  is  likely  to   fail.   Anythin
                      defense  of  Regional  Medical  Programs,  I  think  we  can   do    that is purely retrospective will surely fail.
                      well.                                                                   If  you  have  difficulty  in  deciding  where  you  need  ti
                         Now,  mind  you,  I'm  saying  this  at  a   time   when   our    go  or  what  matters,  I  think  a  careful  scrutiny   of   th
                      legislation has  not  yet  appeared.  We  are  really  living  on    daily newspaper is  very,  very  helpful.  If  you  need  to  gi
                      borrowed time  -  and  we're  used  to  that.  We  still  do  not    beyond  that,  it  helps  considerably  to  go  where   some   o
                      have  appropriations.  We  are  living  on   borrowed   money   -    the problei-ns  are,  to  talk  with  some  of  the  people  wh(
                      and  we're  used  to  that.   But   regardless   of   how   these    are  not  getting  the  kind  of  medical  care  they  wish,  t(
                      events emerge,  and  even  if  the  definition  of  our  legisla-    consider the fact that the quality  of  care  is  not  merely  ,
                      tion  is  fairly  narrow,  you  as  individuals  responsible  for    matter  of  considering  the  exchange  between   the   provider
                      RMP's  would  be  most   foolish   to   overlook   the   elements    and  the  lucky  person  who  enters   the   system   and   get!
                      of  evaluation  that  have  been  discussed  today,  and   there-    quality  care.  You   must   also   consider   that   the   fine
                      fore  the  elements  of  purpose   in   Regional   Medical   Pro-    measurement of quality  care  is  diluted  by  factors  such  a@
                      grams.                                                               those who do not get care,  or  where  the  quality  is  so  bad
                                                                                           that it is a very large minus.
                        If  we  came  out  with   legislation   that   says:   "Confine      These  broad   considerations   can   probably   be   resolved
                      yourself  to  categorical  programs   and   within   those   pro-    by  a  sense  of  societal  concern  which  has  been  expressed
                      grams  to  continuing  education,"   -   which   for   the   most    wherever  I  have  gone  in  the  Regional   Medical   Programs.
                      part consists  of  what  I  now  describe  as  episodic  informa-    But  there  is  a  difference  between  one  RMP  and  the  next
                      tion transfer - if  we  indeed  are  to  move,  are  mandated  to    in  the  determination  that  a  bold  direction   is   a   good
                      move  in  that  direction   and   we   do   indeed   respond   by    direction. In fact, at the present  time,  the  bold  ones  have
                      isolated  categorical  projects,  there  will  no  longer  be   a    been the wise ones,  and  in  a  kind  of  paradoxical  way  the
                      Regional Medical Program.                                            bold have been cautious.























                     34





                   HSNIHA - AN INSTRUMENT FOR IMPROVEMENT IN HEALTH SERVICES
                                                      VERNON E. WILSON, M.D.
                                  Administrator, Health Services and Mental Health Adi-ninistration
           Thank  you,  Harold.   There   are   enough   of   you   who      If  you   say   HSMHA   or   Health   Services   and   Mental
           have heard me speak  before,  that  no  one  will  expect  a      Health  Administration,  the  usual  reaction  is,  "What  is
           flowing,  eloquent  speech  that  is  "snake  charming"   in      that?" I understand that  reaction  because  it  was  my  own
           capability.                                                       when they first talked to me about HSMHA last May.
           I'm delighted to be evaluated by  this  kind  of  group.  It      Let me  give  just  a  precis  of  the  Health  Services  and
           seems to me if anyone will do  this  in  an  objective  way,      Mental  Health  Administration  for  those  of  you  who  may
           you will.                                                         not know what it is.
           As  many  of   you   know,   the   Regional   Medical   Pro-      Regional  Medical  Programs  is  one  part   of   HSMHA,   as
           grams have probably been closer to  my  heart  then  any  of      you   well   know.   The   Community   Health   Service    is
           the other new movements  of  the  federal  scene  in  recent      another substantial part. It is a program with  a  budget  of
           years. This is not a maudlin sentiment.  It  is  my  evalua-      some half billion  dollars  a  year.  Incidentally,  this  is
           tion of the promise of this program.                              where   Comprehensive   Health   Planning   fits   in.    The
           A substantial portion of  that  promise  arose  out  of  the      National  Center  for  Health  Services  Research   and   De-
           opportunity to allow the  grass-roots  mobilization  of  in-      velopment,   which   some   of   you   have   contacted,   is
           novation  and  the  grass-roots  decisionmaking  process  to      another  component  of  HSMHA.  The   National   Center   for
           take hold.                                                        Health Statistics is another.
           In  that  context  there  were  several  kinds  of  problems      In  the   newly   established   family   planning   endeavor,
           with  which  you've  been  struggling  over  the  past   few      Dr. Louis Heilman  is  setting  policy  for  the  Department.
           days. I'm not going  to  treat  anew  the  things  you  have      Dr. Frank Beckles, as Director of  the  National  Center  for
           been talking about. But among  them,  of  course,  has  been      Family Planning Services, has  most  of  the  adn-dnistrative
           the continuous struggle between  the  two  polar  tugs.  One      responsibilities in HSMHA.
           of these is to  give  clearcut  guidelines  so  that  people      The    Indian    Health    Service    is    another     HSMHA
           know specifically what to do in order  to  assure  a  "good"      program,  as  is  the  hospital  program  providing  care  to
           performance.  The  other  is  to  wait  patiently,  Rogerian      merchant  seamen  and  other  beneficiaries.   These   direct
           style, until from all of the massed intelligence,  the  dis-      care activities account  for  a  substantial  number  of  our
           comfort of silence brings forth the new idea.                     employees.  The  National   Institute   of   Mental   Health,
           This  has  been  an  extraordinarily  challenging  sort   of      HSMHA's largest single  component,  has  a  wide  variety  of
           process following  the  Rogerian  style,  because  Congress,      programs  in  research,  training,  and  service.  The  Hill-
           which votes money on the  strength  of  local  support,  has      Burton  program,  Maternal  and   Child   Health,   and   the
           had  some  difficulty  understanding   why   there   was   a      Center  for  Disease  Control  are  other   constituents   of
           strong movement.                                                  HSMHA.
           Some  of  you   need   to   keep   this   set   of   complex
           variables in n-dnd as you look at the way we are  trying  to       To present it in simplistic  terms,  in  the  organizational
           explain  to  the  Congress  how  extraordinarily   important      structure there  is  a  director  for  each  of  these  major
           we think it  is  that  we  let  the  grass  roots  make   the     HSMHA programs  who  has  a  direct  responsibility  for  our
           decisions.                                                        legal and  fiscal  relationship  to  the  Department  and  to
           If   one characteristic of  RMP  can  be  set  forth,  it  is     Congress.
           that RMP  has  not  had  a  distinct  public.  There  hasn't       In   addition,   included   in   our   programs   are   some
           been one particular  group,  external  to  the   organization     guidance  responsibilities   that   we   assume   for   other
           itself, which has  gone  to  Congress  and  said,  "We  must      agencies.  These  include,  for  instance,  the  Federal  Em-
           have this." Instead, there have  been  several  publics  who      ployee  Health  Service,  the  medical  portion  of  the  Ap-
           have  gone  to  Congress,  each  with  its  own   image   of      palachia programs,  the  foreign  programs  under  P.L.  480.
           Regional Medical Programs, and  therein  lies  part  of  the      And more recently we have  been  asked  to  have  a  look  at
           @,iroblem which I hope we are beginning to resolve.               the design of the Health Maintenance Organization.
                                                                                                                                         35





                    In each of  the  ten  regions,  which  recently  have  been    scope  of  our  agency's  role  should  be  as  a   responsible
                 slightly reoriented, there  is  a  Regional  Health  Director.    Federal  agency  in  health  services   delivery.   And   we're
                 Roughly  one-third  of  HSMHA's  resources   are   now   being    looking  at  HSMIIA  primarily  as   an   agent   for   working
                 expended  at  the  prerogative  and  under   the   administra-    with the systems of health delivery.
                 tive authority of the regional health director.                      The  Federal  role  in   health   care   has   been   moving
                    I answer  for  these  responsibilities  directly  to  Roger    recently from a  passive  to  a  more  active  involvement.  As
                 Egeberg who answers to the Secretary.                             you  know,   in   the   past   the   Federal   functions   have
                    It's an interesting  and  complex  organization.  I'll  not    emphasized   limited    direct    responsibility    and    con-
                 go further into this, other  than  to  say  that  the  author-    siderable use  of  various  kinds  of  stimulating  mechanisms.
                 ities for all of our programs are  vested  in  the  Secretary.    It  is  my  impression  that  even  when  it   is   stimulating
                 And  most  of  them,  with  other  than  policy  impact,   are    private initiatives, the Federal role in the future  will  tend
                 then delegated to the Office  of  the  Administrator  and,  in    increasingly  toward   setting   the   terms   and   conditions
                 turn,  to  the   program   directors   and   regional   health    under wlgch those initiatives will be carried out.
                 directors.                                                           Now,  we  in   HSMHA   have   made   some   assumptions   in
                   I hope   this outline  of  HSMHA's  organization  will  give    the  early  deliberations  during  my  85  days  of  this   in-
                 you some idea of  the  perspective  from  which  I  will  talk    carnation; and I  should  like  to  share  some  of  them  with
                 about RMP this morning.                                           you  for  your  thoughts.  These  are  not  dicta  but  are,  I
                   The   RMP   concept   has   always   attracted   stimulating    think, bases for departure in the analysis process.
                 and  innovative  people.  Ms  conference  is  simply   another      The  Federal  role  should   always   be   complementary   to
                 manifestation of this fortunate tendency.                         the private sector insofar as possible.
                   We are  at  a  critical  juncture,  a  decisionmaking  point      The  Federal  role  must,   however,   protect   the   common
                 in the health care field generally. There  are  a  substantial    good  where  inadequacies   or   inequities   appear   in   the
                 number of evaluations going on at  all  levels  and  with  all    system.
                 degrees of sophistication. Currently, there  isn't  an  effort      Maximum  effectiveness   must   be   assured   when   federal
                 in the health care field, public or private, that is  escaping    dollars are used either as an  expenditure  or  as  an  invest-
                 scrutiny;  and  apparently  no  assumption  is  going  to   be    ment.
                 taken for granted in the foreseeable future.                        And  Federal   leadership   must   assure   coordination   of
                   The  Executive  Branch  itself  is  engaged  in   a   funda-    efforts    and    common     communication     among     health
                 mental  reexamination  of   both   the   appropriateness   and    activities in both the public and private sectors.
                 the  effectiveness  of   its   health   care   programs.   The      None  of  these  is  new.  You  have  all  thought  of  them.
                 Congress   itself   is   entertaining   proposals   that   are    Perhaps the difference is that we intend to  act  on  this  set
                 enormous in their scope and  diversity.  And  all  across  the    of assumptions; and that might make a difference.
                 country groups of health  professionals,  such  as  this,  and      In   our   opinion,   the   government   should   help    the
                 individual  patients  themselves,  are  weighing  the  options    energy in the health care system  to  flow  where  it  will  do
                 available to them in  choosing  courses  of  action  that  are    the most good ' Viewed  in  terms  of  energy  flow,  we   have
                 now  beginning  to  determine  our  health  care   system   of    to look at  ourselves  in  proper  perspective.  For  instance,
                 the future.                                                       we  have  25,000   employees   who   contribute   25,000   man-
                  Some  of  these  evaluations,  like  the  three-day   session    years  to  the  health  care  system;  whereas  the  practicing
                 which you have had, are  objective  and  as  thorough  as  the    physicians  of  this  country  account  for   roughly   300,000
                 state of the art  will  permit.  Others  are  very  subjective    man-years.  And  the  7,000   hospitals   across   the   nation
                 and  based  only  on  anecdotes  or  fragments  of   evidence.    have  a  combined  energy  input  of   several   million   man-
                  It's  important  for  you,  I   think,   to   remember   that    years.
                 sophistication  carries  no  guarantee  of  acceptance  unless     While  there  is  no  such  thing  as   a   common   unit   of
                 we  make  sure  that  our  input  is  registered.  The   naive    health energy, it  is  evident  that  our  problem  is  how  to
                 assessments  may  be  the  ones  that  are  crucial   to   our    i-nake the contribution we  have  effective  in  a  very  large
                 future.                                                           system.
                  In  the  Health  Services  and  Mental   Health   Administra-     Ideally,   the   government   activity   should   concentrate,
                 tion,  we  too  are   deeply   engaged   in   self-evaluation.    as  we  have  already  said,  in  the  areas  where  the   free
                 Roughly  one  percent  of  our   total   expenditures,   which    i-narket is unable, for one reason or another, to  fulfill  the
                 are in the nature of $I.S billion a year  are  set  aside  for    public  need.  Such  areas  tend  to  occur  when  the  antici-
                 evaluative purposes, We  are  trying  to  find  out  what  the    pated  private  return  provides.insufficient   motivation   or

                 36




                        Child  Health,  the  Center  for  Fa@ly  Planning,  and   the        The planning  agency  had  its  greatest  promise  as  I
                        others.                                                          voice of the people  in  the  political  sense,  enunciating
                          I'm  aware  that  the  relationship  among  these  programs    the  providers  the  public   determination   of   needs   a
                        and   particularly   between   RMP,   Comprehensive    Health    priorities.
                        Planning,  and  the  R&D  Center  has  been  the  subject  of       It  has  a  geopolitical  responsibility  to  assure   to
                        endless   debate   since   these   programs   began.   Almost    constituents equity of care  at  the  highest  possible  le,
                        everybody in the  health  field  has  had  a  piece  of  this    of quality through the instrument of planning.
                        action.  We  have  had  advice  from   everybody,   but   the       The  R  &  D  Center,  the  newest  member  of  the  tria
                        subject still remains.                                           was'  envisioned  as  an  experimental  instrument   applyi@
                         I'm sure this is one of your  concerns.  It  is  one  of  my    scientific disciplines to the model  of  the  health  servic
                        highest priority items; so much so, in  fact,  that  we  have    delivery  system  in  the  community.   Hopefully   it   was
                        initiated  an  intensive   administrative   study   that   is    be a  generator,  tester,  and  evaluator  of  innovative  a
                        targeted  to  the  specific  mission  of  defining  separate,    proaches in the system, addressing  itself  to  such  things
                        distinct identities for these three major programs.              cost  containment,  equity   of   access,   and   efficiency
                         There  is  an  extra  special  group  of   consultants   who    resource utilization.
                        will be  functioning  in  various  ways.  The  work  will  be      These  philosophical  differences,   however   satisfactox
                        coordinated  by  Dr.  William  Willard   of   Kentucky.   Dr.    or unsatisfactory they may  be  in  the  intellectual  sense
                        Willard will be  spending  about  eight  days  a  month  with    haven't  provided  adequate  guidelines  for  practical   di
                        us over the next several months. His  efforts  are  going  to    tinctions in the health delivery system of  the  real  worl(
                        be  augmented  in  various  ways  by  Dr.  Monty  Duval,  Dr.      It is imperative that this situation be clarified  in  suc
                        Ed Pellegrino, Dr.  John  Hogeness,  Mr.  Nathan  Stark,  Dr.    a  way  that  we  maximize  cooperation  and   minimize   th
                        Julius Richmond, and  Dr.  Ward  Darley.  I  think  those  of    overlap  and  confusion  among   these   programs.   Unnece!
                        you who know some  of  the  stalwarts  in  the  field  recog-    sary duplication, with its resultant  waste  of  effort  ani
                        nize that we really have  pulled  out  the  biggest  guns  we    money is intolerable. In fact, it is destructive in the fac,
                        know  of  to  get  some  administrative  discussion  of   how    of a limited  budget  and  an  unlimited  need  for  improve
                        we can  do  this  constructively  and  preserve  the  tremen-    ment in health care.
                        dous promise of each of the programs.                             The effort at clarification to which  I  am  assigning  tol
                        In a generalized way,  the  shape  of  the  distinctions  can    priority is not to  be  construed  as  competitive.  We  ar(
                        be  deduced  from  the  terms   in   which   these   programs    not talking about one program  versus  another.  I  view  ii
                        were originally framed, at least as I understand them.           as essential if we are to justify and obtain continuing  an@
                        RMP  was  originally   conceived   as   a   bridge   bet%veen    productive  support  for  all  of  the  efforts   of   HSMHA
                        human  -need  and  scientific  advance,  if  you  put  it  in    wherever they may be.
                        simplistic terms. It represented in a sense a  practical  at-    If  you  are   concerned   about   administrative   arrange-
                        tenipt to  link  C.P.  Snow's  "two  worlds,"  which  may  be    ments  at  HSMHA  headquarters  that   have   an   important
                        somewhat out of date  now,  but,  nevertheless  is  what  was    bearing on the conduct of your  RMP  activities  across  the
                        in mind.                                                         country, I am sure that you will make  it  known  to  us  in
                        The  requirements  of  the  individual  patient  were  to  be    whatever unrestrained or restrained  iiianner  you  have  in
                        better   served   by   creating   arrangements   that   would    the past. We solicit that kind of interest and input.
                        enhance the flow  of  greatest  expertise  to  the  patient's    I have made a fairly fast  attempt  in  a  short  period  of
                        bedside  through  an  effective  linkage  of  the  providers-    time here to sketch out for you in  broad  strokes  some  of
                        and I should like  to  emphasize  the  effective  linkage  of    the dimensions of  the  broader  stage  on  which  your  in-
                        providers.                                                       dividual programs are enacting an important role.
                        It did, as we  have  already  said,  give  providers  an  op-
                        portunity to innovate from grass-roots ideas.                    RMP  is  an  integral  and  extremely  important   part   of
                        Comprehensive   Health   Planning   approached    the    same    HSMHA.  HSMHA,  in  turn,  is  the   agency   charged   with
                        ultimate objective from a different angle  of  attack.  Here,    exercising  appropriate  federal  stewardship  in  DHEW  for
                        by  fostering  planning  processes  at  the  State  and  com-    health care delivery. It is not simply a collection of  pro-
                        munity level, the intent or at  least  the  greatest  promise    grams; it is a composite. And  each  of  its  components  is
                        seemed to be to  encourage  a  political  consensus,  in  the    to contribute to a comnion mission.
                        broadest sense of a political consensus, as to  health  goals    The test of  our  perfon-nance,  yours  and  mine,  will  be
                        and the use of health resources.                                 whether or not  we  can  apply  ofir  combined  leverage  so
                        38



         ,vhere  the  scale  of  investment  required  is   beyond   the       When  it  becomes  apparent  that  a  given  segment  of   the
         capability of the private resources.                                  population-for  example,   expectant   mothers   or   niigrant
         I'd like to give  you  a  few  illustrations  of  fields  where       workers-is not receiving the kind of  health  care  it  has  a
         HSMHA believes  it  can  perform  a  valuable  service  to  the       right to expect,  someone  has  to  be  responsible  for  set-
         system as a whole.                                                    ting this forth in  clear  terms  and  making  it  a  part  of
         As   a   health   care   delivery   agency   of   the   federal       community thought.
                                                                               Someone  has   to   begin   a   stimulation   process   to   a
         Government,  we  can  meet  one  urgent  national  need  for  a                                  pond.   This   does   not   it-nply
         central source of  valid  and  creditable  information  on  the       point that the system will res
         delivery of health care.                                              direct action in  terms  of  meeting  the  need,  although  we
         Now,   I   don't   know   how   many   of   you   have   really       are involved  in  some  of  that,  but  I  think  more  impor-
         gone  at  the  business  of  looking  at  our  performance   in       tantly  involved  is  getting  the  selected   endeavor   into
         the health care field. But it turns out that any  set  of  data       realistic discussion.
         that  you  pursue  far  enough  seems  to  come  back  to   the       This function will  have  in  it  at  least  two  phases.  The
         National  Center  for   Health   Statistics,   often   somewhat       first is  a  continuing  systematic  and  sophisticated  over-
         mangled  in  the   process,   and   discouragingly   far   from       view of what  the  health  care  system  is  doing,  projected
         anything  that  tells  us  about  the  health  status  of   the       against  two  grids-what  it  could  do  and  what  the  needs
         nation.                                                               are. At this time  we  don't  have  really  adequate  surveil-
         It tells us a little about the  absence  of  disease,  but  not       lance of performance, capability, or need.
         a great deal about the health status of the nation.                   The   second   phase   involves   getting    something    done
         Somewhere  there   needs   to   be   a   competent   group   of       about it. And  certainly  from  our  relatively  small  fiscal
         experts to sift through  the  diverse  health  care  activities       base,  we   will   have   to   look   at   communication   and
         that  are  conducted  in  our  many   localities   under   many       persuasion  rather  than  direct  entry   into   meeting   the
         different  auspices,  and  to   analyze   and   summarize   the       need itself.
         national experience.                                                  The  tools  at  our  disposal  then  are  going  to  be   com-
         To  the  best  of  our  knowledge,  no   such   source   really       munication,  persuasion,  selective   encouragement   of   in-
         exists that  is  capable  of  providing  validated  information       novation.  And  that's  the  name  of  RMP  as  far   as   I'm
         on  multidimensional   and   multidisciplinary   questions   on       concerned.
         health care delivery.                                                 We need  to  use  that   instrument   well.   One   instrument
         In  my  opinion,   efficiency   dictates   a   single   central       of   stimulus   for   improvement   can   come   from    RMP's
         source, and logically  this  is  the  role  which  the  federal       functioning  as  a  center  of  expertise;  and  this  is  the
         Government should  do  as  it  has  already  done  in  agricul-       instrument  of  information  display  in  which  we  hope  you
         ture,  commerce,  and  to  a   substantial   extent   in   bio-       will join us.
         medical research through the NIH.                                     I think all of you are aware of the fact that      when      a
         It  is  our  intent  to   become   the   locus   of   such   an       company's stock is  performing  badly,  this  is  made  pretty
         activity. Interestingly enough,  we  have  the  mandate.  In  a       clearly visible in the daily listing on every financial  page.
         substantial  number  of  our   programs,   including   Regional       And  general  knowledge  is   a   powerful   spur   to   self-
         Medical    Programs,     Comprehensive     Health     Planning,       examination  and.  change  in  those  whose   stock   is   not
         National   Center   for   Health   Statistics,   and   National       doing so well.
         Center  for   Health   Services   Research   and   Development,       In  health  care  performance,   the   criteria   may   be   a
         we  even  have  the  models.  We  have  the  instruments,   and       little  harder  to  define  and  the  comparative  information
         we have the capacity.                                                 harder  to  acquire,  but  once  acquired  and  displayed,  it
          The role  represents one way         in   which   this   small       could and should have a similar effect.
         energy input can help direct  the  flow  of  a  larger  system.
          A  second  role   to   which   we   might   aspire   will   be       We  have  some  other   instruments   for   change,   programs
         characterized  as  a  kind   of   guardian   of   the   nation'       that  are  explicitly  designed  to  stimulate  innovation  in
                                                                       s
         standards in health affairs.  And  I'm  not  thinking  here  of       health  care  delivery  and  effective  synthesis  of   health
         regulation  as  much,  although  this  may  apply  in'  a   few       resources for the benefit of the  patient.  This,  of  course,
         cases, such as in  quarantine;  rather  I  am  thinking  of  an       again includes RMP.
         evaluator  of  performance-which   is   exactly   what   you're       It  also   includes   the   planning   and   project   support
         doing  here  today-and  an  activator  of   the   public   con-       activities  of  Community  Health  Services  and   the   other
         science.                                                              activities  we  have  talked  about  such  as   Maternal   and
                                                                                                                                            37





    the total impact of our efforts  is  greater  than  the  sum  of        Finally,  in  all  of   our   evaluative   activities   it   is
    the parts.                                                           imperative  we  keep  in  mind  the  ultimate  objective  of   our
       As  you   go   through   this   evaluation   conference   and     endeavors-that   what   happens   to   the   patient    or    pre-
    your  further  evaluative  efforts  in  RMP,  you  need  to   do     patient  is  really  what  we  are  supposed   to   be   concerned
    so  in   this   broader   context.   Your   evaluation   efforts     about. That's the hardest evaluation of all.
    should be an important  input  for  ours.  The  results  of  our        The one thing we  still  lack  is  the  measure  of  health  as
    evaluations, in  turn,  must  be  factored  into  the  equations     an ultimate yardstick.
    @vhich involve the total health care system.
       For  this  reason,  it  is  important  that   we   make   our        In  the  same  area,  we're  dealing  with  the   health   care
    findings  and  your  findings  widely  and   freely   available.     system which is still a  crisis-oriented  system.  It  pays  least
    Just  as  communication  is  one  of   the   strongest   instru-     attention  to  first   things-health   maintenance   and   disease
    rnents  for  change  within  the  system  as  a  whole,  it   is     avoidance-the  greatest  attention  to  illness   after   it   has
    almost our only  instrument  for  change.  it  is  going  to  be     occurred.
    effective  in  proportion  to  our   use   of   the   evaluation
    process.  If  we  don't  make  known  what  we  know,  we   will        We  need  to  be  sure,  if  we  are   thinking   truly   about
    have no cause for  complaint  if  we  are  not  a  part  of  the     serving  the  public  both  present  and  future,  that   we   are
    future.                                                              not similarly distracted in the planning process.


























                                                                                                                                          39






                                                       WORKSHOP SESSIONS
                                                       WORKSHOP ON DATA


                                                                  Participants
       @rthur R. Jacobs, M.D., M.P.H. - Moderator                         David E. Reed, M.D.
       )irector of Statistical and Evaluation Unit                        Assistant Director for Evaluation
       Zochester Regional Medical Program                                 Western Pennsylvania Regional Medical Program
       @ah J. Peterson - Associate Moderator
       3iostatistician
       @abius, New York
                                                                               Katherine G. Bauer
       -@les A. Nietzner, Ph.D.                                                Research Associate
       Professor of Medical Care Organization                                  Harvard Center for Community Health
       University of Michigan                                                  and Medical Care
       1. WiWam Gavett, Ph.D.
       Associate Professor, College of
       Management and Department of
       Preventive Medicine and Community                                       John E. Wennberg, M.D.      gland Regional
       Health                                                                  Coordinator, Northern New En
       University of Rochester School of Medicine                              Medical Program

                     CHARLES A. METZNER                                        invention  enticed  Herman  Hollerith  to  father   punched
                                                                               card  procedures  for  mechanical  data  processing,  which
       A  short  presentation  on  this  large  subject  can  only             now  make  possible,  particularly  since  electronic  pro-
       sketch  the  topics  and  arguments.  The  attempt  to   be             cedures have been substituted, the derivation  of  so  many
       short results  in  more  direct  and  unconditional  state-             tables that it is hard to find  our  way  around  in  them.
       ments than are strictly warranted,  but  this  may  be  the             One additional  historical  point  may  be  interesting  to
       basis of the discussion to follow, although my aim  is  not             you. It was in 1942, a  relatively  recent  date,  that  "A
       to  be  deliberately  argumentative  to  stimulate  contro-             New  Sample  of  the  Population"  was   developed,   which
       versy. I shall try to elucidate problems  and  lead  toward             embodied the  first  practicable  methods  for  probability
       some useful conclusions.  Explanations  are  not  complete,             samples of human populations. It  may  be  worth  recalling
       either, but questions, if necessary, can elicit more.  What             that these area sampling methods  were  a  product  of  the
       I am trying to do is  to  stimulate  thoughtful  considera-             WPA,  and  later  incorporated  into  the  Bureau  of   the
       tion.                                                                   Census.  Sampling  enabled       many   more   data    to    be
                                                                               generated at  much  lower  cost  when  estimates  are  suf-
        Censuses  are  not  new.  In  fact,  there   is   Biblical
       mention of a census and  the  ideological  response  to  it             ficient.
       then still has  repercussions  among  fundamentalists.  One
       is reminded also that total counts are sufficient -  Gideon                                        Sourc s
       became famous by applying a behavioral  test  to  select  a              There  are  some  guiies'to  data  that  are  useful.  The
       subset with characteristics he  wanted.  Now  many  charac-             Statistical Abstract of' the United States presents an  an-
       teristics are incorporated into census  data.  The  attempt             nual overview of data, with references to sources. It is  a
       even at health data is  not  utterly  recent,  however.  In             good index to availabflity. It is mentioned (on page  five)
       1870,  the  United  States  Census  became  very  ambitious             in what should  become  a  basic  reference,  the  National
       and, among  many  other  data,  tried  to  obtain  informa-             Center for Health  Statistics'  short  pamphlet,  "A  State
       tion  on  fliness.  The  procedures  were  somewhat  crude,             Center for Health  Statistics:  An  aid  in  planning  com-
       but the amount of data of all kinds was  so  voluminous  as             prehensive health statistics". (Revised October  1969.)  It
       to threaten the decennial census by taking over  ten  years             is available from  the  Center  or  the  U.  S.  Government
       to process. This is  the  point  at  which  the  mother  of             Printing Office. Among  other  items,  the  chart  on  page
                                                                                                                                               41




                       11  on  input-processing-output  relations  describes  roles    of  the  ordinary  beliefs   concerning   blood   transfusion
                       that may conflict and useful advice is  given  for  handling    Demoniacal  possession  is  not  included  in   many   disea!
                       these. In particular  a  number  of  user-designer  problems    classifications,  but  you  should   check   with   your   ow
                       are considered. On page 13 is a discussion  of  the  use  of    religious leader as to what he thinks your  beliefs  ought  t
                       computers and the  necessity  for  thought  that  makes  all    include.  It  is  my  understanding  from  nutritionists  th,
                       else  commentary.  A  rich  passage,  deserving   expansion,    the usual concept of  what  constitutes  a  "bland"  diet  h,
                       occurs on page 14  with  respect  to  cooperative  relations    little more  to  recommend  it  than  the  idea  of  "hot"  c
                       between users and suppliers of data.                            "cold" foods.
                       Some  state  agencies  have   been   developed,   and   many       The  point  should  now  be  made  that  we   have   vari(
                       health  and  planning   departments   generate   data   that    gated  subgroups  in  our  population.  The  discussion   nia
                       should  be  looked  into.  As  mentioned  later,  the   more    have  seemed  farfetched  and  rather  distant   from   data.
                       local the data the more  specific  the  estimates  that  may    believe I can make multiple use  of  the  ideas,  however,  i
                       be derived.                                                     trying to follow the implications of  a  pluralistic  societ:
                       As  an  introduction  to   problems,   another   publication    for a) the-, generality of  findings,  b)  the  generality  o
                       of  NCHS  is  valuable:  The  1970  Census  and  Vital   and    concepts used, and  c)  the  necessity,  flowing  from  these
                       Health Statistics.  A  Studv  Group  Report  of  the  Public    for greater  freedom  of  research,  particularly  in  govern
                       Health  Conference  on   Records   and   Statistics.   Docu-    nient.
                       ments   and   Committee   Reports.   PHS   Publication   No.
                       1000 - Series  4  -  No.  10.  Government  Printing  Office,                           Implications
                       April  1969.  This  is  a  planning  volume  for  the   1970    777e Geiierality of Findii?gs
                       Census, still useful on issues.
                                                                                         From the fact of  high  variation  within  our  society,  i
                                             Problems                                  follows that national data have little specificity  and  higf
                       National  data  involve   many   kinds   of   problems.   In    variation.  Certainly,  the  mean  outlay  per   person   foi
                       common   with    other    data,    becoming    knowledgeable    health  care  times  the  nui-nber  of  persons  equals  tli(
                       involves not only the names  of  variates  but  definitions,    economic load on the private  sector  for  health  care.  Bul
                       particularly  embodied  in  a  questionnaire,  the  instruc-    the  mean  is  not  very  representative.   Not   very   man)
                       tions, and codes - in short  -  all  processes  which  shape    people get the  i-nean  income.  The  standard  deviation  i@
                       the final product.                                              so  vast  as  to  encompass  most  of  the  information   and
                       There  are  some  special  issues   concerning   terms   and    render the mean  almost  meaningless.  If  you  look  at  any
                       definitions that arise in a nation like ours. Some  of  this    distributions in the health field, you can see this. This  is
                       may be easy to see nationally, but you  should  not  be  too    one  reason  why  insurance  is  so  important  a   mechanism
                       certain that this applies only to someone else.  Ours  is  a    for achieving a mean value.
                       pluralistic, individualistic society, with plural health care     There  are  several  ways  in  which   this   is   directly
                       systems. A single basis of  definition  does  not  encompass    important.  The  variations  in  national  data  can  and  do
                       all. Ordinary classifications, such  as  the  "International    spread between Census regions, states,  and  regions  in  the
                       List", assume  an  M.D.  etiological  base,  largely  micro-    @IP  sense.  National  data  do  not  necessarily   represent
                       biological. Because we  have  not  recognized  the  ways  in    your  area,  to  the  extent  that  your  area  is  distinct.
                       which other people live  and  think,  we  are  being  tested    National data must be sorted to yield  data  for  your  area,
                       again  concerning  some   accommodation   to   multiplicity.    which  may  be  done,  and  cognizance  should  be  taken  of
                       How far are we willing (or able) to go?                         the fact that the Census is willing  to  do  this,  and  that
                       Would   we   accept   a   voodoo    health    center?    The    one large study is  cheaper  than  several  si-nall  studies.
                       question is put in this form  to  test  associations.  Since    However, for  sample  data,  the  results  may  well  be  un-
                       the audience is  more  or  less  white,  and  more  or  less    reliable  for  small  areas.  And  equally  important,  while
                       Christian (although perhaps  not  up  to  the  standards  of    sai-npling distributions for the estimates  may  be  derived,
                       Dr. Martin Luther King),  we  are  inclined  to  be  shocked    in general the common statistical tests (t,  chi-square)  are
                       but accept the racist  implication  that  this  would  be  a    not valid for these statistics.
                       black  enterprise.  We  should  examine  our  readiness   to     A  frequent  procedure   to   "adapt"   or   "derive   esti-
                       accept the implication. California does pay  faith  healers,    mates" for a particular area from  data  for  another  is  to
                       Christian Scientists do not cet the diseases of  the  Inter-    analyze the data  by  other  variables  (age,  sex,  previous
                       national List, and Jehovah's Witnesses  do  not  accept  all    condition of servitude) and use  regression  or  standardiza-


                       42





    tion  techniques  to  estimate  values  for  our  specific  dis-     does  not  settle  conceptual  problems,  at   least   correctly.
    tribution  of  the  analytic   variables.   Unfortunately,   the     To  settle  an  issue  of  the  best,  which  is  to   say   most
    residual  variation   also   applies,   because   the   analytic     predictive   or   homogeneous,   definition   of   what   is   an
    variables  can  only  transfer  as  much  of  the  variation  as     epidemic  or  what  constitutes  group  practice,  we   have   to
    they  absorb,  and  what  remains  is  error,  in  both   cases.     try  them  all  and  find  what  difficulties   ensue   or   what
    For health  data,  our  analyses  so  far  do  not  account  for     utilities  be   in   each.   Frequently,   we   are   forced   by
    much  variation,   and   the   estimates   are   correspondingly     circumstance  into   premature   definition   which   is   copied
    poor. The  fact  that  it  is  done  with  mathematical  statis-     and  standardized.  Sometimes  we   just   pick   tip   a   handy
    tical  formulae  on  a  computer  will  not  improve  it.   More     classification, as in  the  case  of  health  studies  using  the
    complete  statistical  analyses  or  a  mathematical   presenta-     International  List,  reflecting  etiology.  This  classification
    tion would  verify  the  locical  argument  above.  Any  statis-     is no doubt  valuable  for  the  practice  of  medicine,  but  it
    tical   text   dealin-   with   multivariate   techniques   will     does  not  resolve  (or  predict)  the  use  of  services   which
    explain this.                                                        forms  the  basis  for  manpower  and  cost  studies.  At  least,
                                                                         some  concept  such  as  the  seriousness  of  the  illness  must
    The Generality of Concepts                                           be  added  when  the  fiscal  or  personal  impact  is  what   we
                                                                         really   desire.   Much   more   must   be   done   to    develop
      Again, there are several     limitations    encompassed     in     concepts  stiited  to   purposes.   And   this   leads   to   the
    the  generality  of   concepts.   The   first   involves   coni-     concluding, issue.
    iiiunicability  and   response.   Respondents   understand   and
    report  only  those   terms   they   know.   And   much   knowl-     Freedom of Reseat-cli
    edge  of  what  is  or  was  wrong  with  us  comes  from  t   e       The argume
    health  care  system.  This  is  a  feedback  process  of   some                     nt thus far culminates,  I  believe,  in  a  plea
    importance.  We  learn   as   we   use,   both   in   terms   of     for  greater  support  for  many  kinds  of  data  and  for   re-
    diagnosis, which must  be  given  us  to  be  at  all  reliable,     search  more  nearly  directed  toward   well   specified   prob-
    and  even  in   terms   of   recognizing   what   symptoms   are     lems. Much of this  may  best  be  done  in  the  locality  of  a
    important   by   how   health   professionals   respond.    What     problem  where  the  distinctive  character  of   the   situation
    this implies  is  a  strong  bias  against  reporting  by  those     may  be   seen,   although   without   effort   and   receptivity
    not  habitual  or  economically   enabled   users.   The   prob-     there is nothing to  warrant  a  belief  that  being  next  to  a
    lems   of   non-users   are   not   reported   by    a    system     problem  ensures  noticing  it.  Most  people  with  glasses   do
    assuming   use,   and   the   resulting   confounding   conceals     not  report  any  disability,  and  it  is   hard   to   convince
    problems of  the  system.  When  a  symptom  list  is  used,  it     people that they are deaf.
    will  help  those  who  recognize  the  symptoms,  but  it  will       Mainly,  I  believe,  it  is   necessary   for   our   national
    not  elicit  a  misery  or  a  devil  bothering   a   respondent     policy  to  incorporate  the   fact   that   to   encompass   the
    unless it includes these. It is much easier  to  adapt  to  this     variety  and  subtlety  of  our   national   life   entails   in-
    locally,  since  many  terms   are   regional,   although   they     dependent   thought   and   effort   and   the   development   of
    may be  ethnic  or  status-related  also.  To  check,  find  out     queer  and  unpopular   ideas,   and   mistakes.   Our   affluent
    from  your  friends  from  different  parts   of   the   country     society  does  have  people  suffering  from  hunger.   We   must
    when  evening  begins  for  them,  and   you   will   get   some     acknowledge  that   we   do   not   with   any   certainty   know
    idea  of  the  problem.  At  any  rate,  the   reliability   and     how   to   interrupt   the   transmission   of    poverty    from
    validity of data are  high  only  for  those  using  the  system     generation  to  generation.   Uncovering   the   hunger   implies
    generating  the  concepts,  and   may   cause   serious   under-     allowing  studies,   and   particularly   analyses.   Discovering
    reporting   of   illness   and   the    unorthodox    treatments     procedures  for  bringing  ghetto   dwellers   fully   into   the
    engaged   in   by   those   uninvolved   with   orthodoxy.    Of     society  or  organization   into   the   health   system   neces-
    course, only  a  national  study  can  demonstrate  all  of  the     sitates   evaluated   experimentation.    But    we    all    too
    variability, but then only if  they  are  prepared  for  it.  If     frequently   constrain   those   with   the   information    from
    one  wants  to  find   out   about   problems,   one   must   be     using it  for  analytic  monographs,  and  insist  that  a  prob-
    receptive.                                                           lem  be  fitted  with  a  single  agreed-upon  solution.   Diver-
       A  second  issue  is  the  problem   of   "general   purpose"     sity  in  the  society  must  be  matched  by  diversity  in  ap-
    data  to  be  used  by  many.  Of  course,  agreement  on   what     proach, conceptually and operationally.
    information to  get  is  a  political  and  economic  necessity,        Our  national   agencies   are   producing   many   good   and
    but  however  valuable  compromise  may  be  in   politics,   it     useful  data.  If  they  themselves,  who  know   a   number   of

                                                                                                                                        43






                   the  weaknesses  better  than  those  with  second-hand   ac-       The purpose of this article is to consider the  relati(
                   quaintance,  were  allowed  to  use   them   to   draw   con-    ships between patient classification, data  collection,  a
                   clusions, we might do better. They are  willing  to  meet  us    facilities and  manpower  utilization  for  ambulatory  ea
                   more  than  half  way,  though.  There   are   many   special    Before doing so three caveats are offered:
                   analyses that may  be  obtained,  if  we  ask,  and  although       1. The  design  and  implementation  of  a  data  systi
                   they will not be free, they  are  less  costly  than  special    for  ambulatory  care  should   proceed   concurrently   w
                   purpose  studies.  The  Census  Bureau  will,  for   example,    the  development  of  hypotheses  about  the  planning   a
                   design samples for us using  their  rich  data  base.  Within    organization  of  such  care.  The  data  will  provide  I
                   the  limits  of  confidentiality,  information   on   special    decision maker with the  necessary  statistical  informati
                   groups may be obtained.                                          for evaluation of alternative  ai-nbulatory  care  proposz
                      National data can be  exceedingly  useful,  but  they  are    Unless hypotheses  about  changes  are  offered  prior  to
                   no  panacea.  They  are  not  universally  applicable,   they    simultaneously with the  design  of  the  data  system,  I
                   are not fully analyzed,  and  they  do  not  serve  id]  pur-    latter effort may be  extremely  costly  for  what  inforn
                   poses. We must consider  the  limitations  in  the  light  of    tion is required and used.
                   our  objectives,  and  we  may  thereby  help  to   eliminate       2. There is no  single  decision  maker  in  the  typi,
                   some limitations.                                                cottzmtinity   ambulatory   care   system.   The   commune
                                                                                    system is typically fractioned and consists of  a  variety
                              Data for Ambulatory Care Planning                     independent  organizations  (listed  previously).  Changes
                                    J. WILLIAM GAVETT                               individual and  independent  organizations  can  make  set
                                                                                    from a community systems  point  of  view  only  if  there
                     American   communities   are   concerned   with   the   in-    coordinated    and    cooperative    community    planning
                   adequacies  of  existing  primary  ambulatory  services,  but    unless severe  legal  constraints  are  b-nposed  in  such
                   do not  have  quantitative  data  necessary  to  plan  alter-    manner as to  force  the  consideration  of  conimunity-wi
                   native  systems  for  the  delivery   of   ambulatory   care.    objectives.   The   purpose   of   voluntary    "communit,
                                                                                    planning  is  to  provide  the  independent  decision  mak
                   Studies  of  primary  ambulatory  care  are  relatively   new
                   compared to studies of  hospital  care.  New  techniques  are    (administrators, physicians,  etc.)  with  information  t@
                   needed to evaluate existing, as well as proposed  facilities,    permits a rational evaluation of  their  decisions  releva
                   for the delivery of prii-nary  ambulatory  health  care.  The    to community objectives.
                   existing  facilities  include:  private  practice  (solo  and      3. Data  and  information  collection  must  be  related
                   various  forms  of  group  practice),   occupational   health    both  the  consumer  of  health  care  (the  patient)  and
                   services,  school   health   services,   hospital   emergency    the processes (methods)  of  ambulatory  health  care.  T.
                   departments,   hospital   out-patient   departments,   neigh-    efficacy  of  a  data  and  information  system  in   aidi
                   borhood  health  centers,  health  department   clinics,   as    planning  will  depend  on  the   manner   in   which   bo
                   well  as  various  state  and  federal   primary   ambulatory    patients  and  processes  are  classified,  described,  ai
                                                                                    measured. It is on this issue that  the  remainder  of  t]
                   services. Proposed models  include  facilities  differing  in    discussion will focus.
                   manpower, financing,  and  utilization  patterns  located  in    -   -,
                   different  areas  under  private,  voluntary  or   government      It is suggested that a classification  schei-ne  that  w
                                                                                    relate  health  care  demand  to  the  manpower,  equipmer
                   auspices.                                                        and facility requirement is needed. Traditional  classific
                      Variables to be studied  rnioit  include  legal,  contrac-    tion  methods  include  patient  characteristics   (matern
                   tual,  and  business   arrangements;   availability,   acces-    and child health,  pediatrics,  veterans  services),  path
                   sibility; degree  of  specialization  vs.  generalization  of    physiologic processes  (tuberculosis,  cancer,  hemophilia
                   services;   consumer   payment   mechanisms;    reimbursement    services  rendered  (radiology,  medical,  surgical),  ar,
                   for    services    mechanisms;    manpower     configuration;    (neighborhood  health  center),  as  well  as  organizatie
                   equipment; ancillary services;  capital,  financing  arrange-    rendering the care (private group  practice,  hospital  ot
                   ments;  and  characteristics  of  services  rendered.  Within    patient department,  occupational  health  services).  Eai
                   the  context  of  defining  the  basic   characteristics   of    of these classifications have some use in health care  pla
                   primary   ambulatory   care   organizations,   consumer   at-    ning. For primary ambulatory care  planning,  a  classific
                   titudes, outcome of care,  and  design  characteristics  such    tion  system  is  needed  to   categorize,   compare,   ai
                   as:   working    spaces,    procedure    and    communication    project patient utilization for different types  of  prima
                   systems, etc. are less important.                                ambulatory   care   delivery    units.    A    measurement


                   44





         demand  in  a  given  ambulatory  unit  is  the  first   step   in   the    power    and    equipment)    are    involved.    The    other     extreme
         conceptualizing   of    alternate    ways    of    satisfying    demand.    is   the   critical,   medically   complex   case   in   which   extensive
         The   demand   or   load   placed    upon    a    primary    care    or-    resources  are  used  often  within   a   short   period   of   time.   It
         ganization  at  any  given  time  can   be   expressed   in   terms   of    is  suggested   that   cases   might   be   classified   into   categories
         the   number   of   cases    (patients),    the    episodes    (specific    such   as   A,   B,   and   C,   where   A   is   the   urgent,   complex,
         medical    problems    requiring    management),    and    the     visit    resource-intensive   case;   and   C   is    the    simple    case,    in-
         (interface    betiveen    the    patient    and    the    health    care    volving    minimal    resources.    The    B    cases    would     include
         sj,stem).   A   classification   system   based   upon    a    set    of    those   involving   long   term    episodic    illnesses    where    diag-
         criteria  related  to  the  characteristics  of   the   case   and   the    nostic    skills,    continuity    of    care,     complex     therapeutic
         visit  as  they  relate   to   the   services   rendered   is   proposed    measures,     and     extended     support     and     observation     are
         tinder   the   assumption   that   such    a    classification    scheme    required.
         will   facilitate   the   conception   of   alternate   care   organiza-        Figure  I  represents  a   definition   of   each   class   in   terms
         tional   designs.   This   classification   method   focuses   on    the    of     specific     attributes.     These     include     manpower     and
         complexity of the case and visit.                                           facility   requirements,   frequency   of   visits   for   the    episodic
         The    primary     care     setting     encompasses     a     continuous    illness,   diagnostic   problems   and    disposition    of    the    case
         scale   of    case    complexities.    At    one    extreme    is    the    visit.   A   fourth    class    might    be    developed    for    psycho-
         medically   simple   case    in    which    modest    resources    (man.    somatic cases and minor psychiatric cases.


                                                        FI G U R E I.-Case Classification Table

                   Manpower                 Facilities            Frequency of           Diagnosis              Disposition                 Comments
                                                                      visit   for                                  of visit
                                                                          episode
                MD required             ICU                        Not     appli-      Extensive Dx           Hospital,              Acute
                                                                       cable                                     home care,
         Class  Likely team             ED                                             Skills required           or                  Life-threatening
         A          effort                                                                 or not             Long term care
         Case.                          Hospital facility                                  required              facility            Totally -nterrupts
                                           required                                                                                      normal living
                MD required             MD office or more          Revisits   re-        Diff'lcult           Home                   Non-acute
                                           extensive                   quired
                Possibly referral          Dx or gx                                      Obscure              Possible hospital      Ongoing         comprehen-
         c7ass      or                     facilities                                                                                    sive and conti-
         B          consultation                                                       Chronic                Long term care             nuity   care
         Case.                          Possible hospital                                                        facility                important
                                                                                                                                       Interrupts        normal
                                                                                                                                         living

                MD or nurse             Average MD office          One   or   two      Relatively             Home                   Acute   or   not
                                                                       short               simple,
                Problem does not        Dispensary                     visits              obvious,                                  Accessibility and
                    demand a                                                               self-                                         availability of
                    specialist                                                             limited                                       service      important
         Oass                                                                              (URI or                                       to   patient
         c                                                                                 minor injury)
                                                                                                                                     Interrupts normal
                                                                                                                                         living to
                                                                                                                                         minor degree

                                                                                                                                     Support    and    reassur-
                                                                                                                                         ance  may  be  a   ma-
                                                                                                                                         jor attribute


                                                                                                                                                              45





                          The  ABC  classification,  and  further  refinements  of   it    ambulatory  care  data  systems  for  patient   care,   institi
                       to   include   subclassifications,   provides   a   basis    for    tional management, and for community planning.
                       considering  the  questions   of   ambulatory   care   organiza-       10.  What  is  the  role  of  the  hospital   in   ambulator
                       tion. For example:                                                  care?   Analysis   of   the   Emergency   Department   and   Ou
                          1.  For  a  given  community,   what   proportion   of   case    Patient  Department  by  case   classification   characteristi(
                       visits  by  A,  B,  or  C  type  are  made  to  which  organiza-    may  provide  quantitative  data  for  reorganization   of   tt
                       tions;  are  resources  allocated  among  organizations  in   an    hospital's ambulatory services.
                       intelligent   manner,   e.g.   perhaps   demand   for   type   A       I   1.   Does   the   measurement   of   low   income   urba
                       services should  be  consolidated  at  one  or  more  hospitals?    ambulatory  care  demand  by  the  case   classification   tecl
                          2.  For  a   given   ambulatory   care   organization,   what    nique  provide  insights  as  to  how  to  organize   ambulatoi
                       proportion of case visits are  in  each  of  the  A,  B,  and  C    services for the urban poor?
                       classes; are  the  unit's  resources  intelligently  related  to
                       the given proportions?
                          3. How  can  a  rural  conununity,  that  cannot  attract  or       Information Systems to Meet Common Data Needs
                       hold  an  MD,  benefit  by  an  A-B-C  classification   of   its                         of Health Agencies
                       ambulatory patient load? e.g.:                                                       KATHARINE G. BAUER
                          a)  Could  class   A   cases   be   serviced   by   volunteer
                       community-supported    emergency    units,    highly     trained       It  has  been   observed   that   information   is   to   tt
                       to   provide   on-the-scene   first   aid   and   transportation    decision-making   process   what   oil   is   to   the   intern
                       service  to  the  nearest  intensive   care   unit   (presumably    combustion  machine.  It  does  not  itself  make   the   proce
                       located centrally)?                                                 work,  but  without  it  there  is  considerable  wasted  effor
                          b)  Could  class  C  cases  be  treated  at  a   private   or    misdirected motion, and eventual breakdown.               Tho.,
                       community-supported    convenience     clinic,     manned     by    who  are  at  the  wheel  in  making  health  policy   decisiot
                       paramedic  personnel,  and  organizationally   linked   to   the    usually  find  themselves  in  the  position  of   the   motori
                       nearest conununity hospital or group practice?                      with a dry  engine  in  the  middle-of  a  Texas  oilfield.  Tt
                          c)  Could  class  B  patients  be  provided  with  long  term    million  barrel  output  of  raw   material   surrounding   him
                       episodic  care  by  nearest  physician  (patient's  choice)  but    useless  to  meet  his  urgent  need  for  a  mere   two   quar
                       with  routine  and  non-complex  class  C  visits  serviced   by    which  have  been  suitably  processed  to  meet   his   engine
                       the convenience clinic?                                             requirements.  We  would  all  agree  that  health  data  gush(
                          4. Where  in  the  management  of  A,  B,  and  C  cases  are    more  freely  than  oil  -  and  that  for  the  most  part   A
                       community-sponsored    facilities    advantageous     in     the    haven't  yet  found  very  satisfactory  ways  of  tapping   an
                       larger   community?   For   example,    what    community    or-    refining  it  for  the  particular  uses  of  those   who   ma@
                       ganizations   should   sponsor   multiphasic   screening,   con-    health decisions - whether these involve expenditures
                       venience clinics,  special  preventive  medicine  clinics,  etc.    thousands of dollars, or of millions of dollars.
                          5. Can  a  clinic  for  the  treatment  of  C  cases  be  ef-       My  assignment  today  is  to  discuss  the  organization  (
                       fectively  used  also  for  the  purposes  of   triaging   non-C    a  health  information  system  as  a  means  of  meeting   suc
                       cases  to  other  community  health   care   organizations   for    important  needs,   particularly   those   of   RMP   evaluator
                       those  individuals   who   do   not   have   access   to   other    and  their  opposite.numbers  in  other  agencies.   Can   such
                       primary care organizations?                                         system  be  designed  to  supply,  link  and  refine  the   man
                          6.  What  proportion  of   ambulatory   case   visits   clas-    streams of  health  data  that  are  routinely  being  generate
                       sified as  C  cases  involve  mainly  support  and  reassurance?    from diverse  independent  sources  -  such  as  the  facilitie
                          7.  How   are   the   concepts   of   family   medicine   and    manpower,   and   vital'   statistics    compiled    by    Stal
                       comprehensive  care  relevant  to  the  class  A,   B,   and   C    agencies,   the   various   utilization   and   patient   origi
                       cases?                                                              records  from  hospitals  and  other  service  providers?   All
                          8.  How  is  the   question   of   the   use   of   paramedic    can these  be  i-nore  usefully  related  to  the  basic  denl(
                       personnel  specifically  related  to  A-B-C  case   care?   Does    graphic  and  health  statistics  from  the  U.S.   Census   an
                       the  C  case  and  C  visit  load  on  the   community   consume    the National Center  for  Health  Statistics?  I  was  asked  t
                       significant  physician  resources  such  that  extended  use  of    lead off this  discussion  by  virtue  of  my  association  wit
                       paramedic personnel isjustified?                                    a two-year study of  this  question  at  the  Joint  Center  f(
                          9.  The  case  classification  technique  may  reveal   which    Urban  Studies  of  M.I.T.  and  Harvard  -  a   study   larger
                       variables  are  important  and  should  be   incorporated   into    inspired  by  Dr.  Osler  Peterson,  Director  of  Research  fc

                       46





      the Tri-State @IP, who  made  major  inputs  to  it  at  every       This  means  that  throughout  almost   all   health   pro-
      stage-2                                                           grams,  not  just  RMP,  researchers   will   be   trying   to
         What  do  we  mean  by   the   term   "health   information    construct  various  types  of  performance  indicators  -   to
      system?" I suspect this  is  I  one  of  those  in-terms  that    permit  coi-nparisons   of   past   and   present   experience
      people have come  to  use  quite  widely  without  benefit  of    within  a  program.  To  measure  the  impact  of  their  pro-
      definition. For  our  purposes  today,  I'm  simply  going  to    gram  on  specific  target  populations  over  time,  and   to
      be  generic  and  talk  about   a   systematic   approach   to    compare  their  program  results  with  those  of  other  pro-
      producing,  storing  and  gaining  access  to  many  kinds  of    grams  which  use  other  techniques  or   methods.   However,
      health  data  produced  from  many   sources,   for   multiple    as we all know, the  right  kinds  and  quality  of  data  are
      uses, by multiple users. Also,  in  an  effort  to  put  first    rarely  available  to  permit  this  crucially  important  re-
      things first, this paper focuses primarily  on  the  organiza-    search to be carried out. One  can  make  a  safe  guess  that
      tional  process  for  systematizing  this  access  to  data  -    not   only   throughout   our   concurrent   workshops    now,
      only  incidently  on  computers.  As  an  analogue  -  if   we    but in similar  health  evaluation  research  meetings  every-
      didn't   yet   have   libraries   but   many   writers    were    where,  the  identical  complaints  are  being  voiced:   "The
      producing  books   which   many   potential   readers   needed    1960  census  data  were  obviously  useless   for   computing
      to read, a first step would be  to  organize  some  system  to    1969 rates - we simply can't tell the trend so far . .  ."  or
      identify the books of  interest  and  to  decide  on  policies    "Unfortunately  the  reporting   system   changed,   so   it's
      for their acquisition, storage  and  circulation.  Only  as  a    impossible  to  compare  past  and  present  performance"   or
      second   step   would   one    commission    a    computerized    "we can't compare  our  results  with  those  of  prograi-n  x
      index.   The   health   information   broker   system   recom-    because  they  used  entirely  different  age  breaks  -   and
      mended  by  our  Boston  study  naturally  differs   in   many    besides  we  have  no  way  to  get  comparable  unit  costs@"
      respects  from  conventional  library   organization   -   yet    One  concludes   that   all   concerned   have   an   enormous
      there are underlying similarities of function.                    stake in iniprovinc, the kind and quality of  the  data  base.
         -The   broker   system   predicates   that   it   would   be      To  provide  the  denominators  of  the  rates  they   need
      mutually  beneficial  to   a   region's   major   public   and    for their various pruposes, researchers in  all  major  health
      private  health  programs   and   agencies,   such   as   RMP,    programs  seem   almost   universally   to   require   certain
      Blue   Cross,   State   Health   Departments,    comprehensive    common  types  of  data  -  the  demographic,  health  status,
      and  area  facility  planning  agencies,  to  join  forces  to    vital  records,  facility  and  manpower  and  the  kinds   of
      obtain and share  the  kinds  of  data  they  need  in  common    utilization  data  reviewed  here  earlier.   Some   of   this
      for their separate research and planning  activities.  At  the    simply isn't now available  -  such  as  disability  rates  of
      same  time,  the  study  warned  against   constructing   tin-    populations  in  cities  or  small  geographic  areas.   Other
      workable   multi-million    dollar    data    banks.    Before    widely  needed   data,   however,   such   as   about   health
      describing this model,  and  telling  you  some  of  the  con-    facilities,  are  being  routinely  generated  for  their  own
      siderations  that  influenced  its  design,  let  us   briefly    operating  or  management  purposes   by   some   one   agency
      review some  of  the  reasons  it  seems  particularly  timely    which,  in  turn,,  may  need  management  -  generated   data
      right now to promote  this  or  some  other  type  of  cooper-    from other agencies for its own evaluation research.
      ative organization for improving health statistics.                  Finally,  staffs  in              a-g-e-nc-ie-s- -q--u-i-te--- often
                                                                        duplicate their  research  efforts,  both  in  their  separate
      Wliy a Health Information Systeol ?                               quests  for  identical source  materials,  and      in   time-
         As budgets  in  every  sector  of  the  health  system  get    consuming     activities  such  as  constructing    S.M.S.A.
      tighter in the face  of  medical  price  inflation,  it  seems    profiles, or population projections. This costs     everyone
      certain that in every type  of  program,  public  or  private,    money.
      the big questions of  accountability  raised  to  you  yester-       Given such common needs and problems             it   would
      day  in  the  plenary  session  will  be  increasingly  posed:    seem  that  major  health  organizations  have  everything  to
      what  benefits  are  patients  actually  receiving   for   the    gain by joining forces at least for the limited goals of:
      money  spent?  How  can  the  program  policies  be   modified
      and  adapted  to  improve   these   cost-benefits?   Obviously          improving  the  quality  and  comparability  of   exist-
      the day is almost over when  those  who  pay  the  bills  will          ing data commonly shared,
      be satisfied by simple tallies  of  patient  days  and  OJI.D.          identifying    commonly    needed    data    now     un-
                                                                                          9
      visits  juxtaposed  with  total   dollars   expended   and   a          available, and finding means to secure them,
      request for a 15% budget increase next year.                            eliminating duplications  of  research  effort,

                                                                                                                                    47






                       arriving  at  agreements  for  specific  types  of   data         development,  sharing  and  use   of   information   perti.
                       sharing.                                                          nent  to  their   common   needs.   Such   an   information
                                                                                         system should  be  planned  at  the  outset  as  the  first
                   Although funds      were  not  available  last  year  for   a         step   in   a   more    complex    communication    network
                proposed  demonstration  of   our   Boston   model,   it   seems         should future expansion seem indicated."
                possible  that  within  the  next  few   weeks   Congress   will         A   broker   function   between   independent   health   in
                authorize  federal   support   for   experimental   health   in-     formation  subsystei-ns  rather   than   a   centralized   datz
                formation  systems  of  this  kind  as  part   of   the   Health     bank was recommended because it would:
                Services Improvement Act of 1970.*                                       o  adapt better  to  the  predictably  ever-changiiig  dat,
                   Funding  is  only  one  aspect  of  the  problei-n  of   data            needs of its users,
                sharing   among    independent    organizations.    Given    the         o  provide  better  quality  information  over   the   lonE
                realities of  the  operating  environment,  can  a  satisfactory            run,
                means   be   found   to   promote   inter-agency    cooperation?         o  avoid direct  confrontation  of  the  issues  of  agency
                When  one  looks  at  the  activities  of  the  Bureau  of   the            confidentiality  and  of  individual  patient   privacy,
                Census  and  of   the   N.C.H.S.   and   other   important   in-         o  function  better  within  the  present  limitations   ol
                formation  centers  at  the   federal   level   one   can   feel            compute  .r software-yet peri-nit  adaptation  to  future
                hopeful, But further  down  the  line  at  the  regional,  state            technological advances expected there.
                and  sub-state  levels  where  mixes  of  various   public   and         Finally,  a  consortium  of  users   was   recommended   a@
                private  data  sources  are  sought,  issues  of   agency   con-     the  policy-making  body   for   the   broker   system,   witt,
                fidentiality  and  of  inter-agency  power  struggles  inject  a     administration  temporarily  vested  in  a  university.   Thi,;
                host  of  complexities.  Whether  organizational   fort-as   can     structure was put forth  in  order  to  avoid  threatening  the
                be  devised  during  the  next  few  years  to  circumvent   the     existin-  power  relationships  among   agencies   sufficiently
                problems  while  fulfilling  the  need  remains  to   be   seen.     to foreclose their participation.
                Our    Boston    study's    recommendations    represent     one         Before going on  to  elaborate  on  some  of  these  points
                possible  approach  -  Dr.  Wennberg   will   tell   you   about     I'd  like  briefly  to  mention  some  activities  proposed  in
                another,  and  I  know  that  several  other  people  here  have     the Boston model.
                been  wrestling  with  these  problems  in  their  own   regions
                - from New Mexico to Ohio.                                           Soiiie Possible Functions and Activities
                The Broker System Model                                              of a Sli,7red Itiforination Systeni
                                                                                         1. Nlakin- data  more  available  for  secondary  analysis,
                   The Boston study concluded that (and I quote):                    by:
                      "The  needs  of  health  plannin-  and  research  in  this
                                                        t)                               *  inventorying  and  cataloging  data  sources  and  files
                   area at the present time will best     be  served  not  by  a            furnishing  detailed  descriptions  of  data  files   tc
                   new  prime  data  processing  coi-nputer  system,  but  by  a            guide the user - such  as  dates  and  methods  of  data
                   mechanisi-n   designed   to   interface    between    several            collection  and  up-date;   sample   size;   format   ir
                   newly   developing   hospital,    public    health,    mental            Nvliich preserved (file folders, i-nagnetic tape,  etc.)
                   health,   and   social   welfare   information   systems   at            person  responsible  for  maintaining  files;  condition
                   regional,   state,   metropolitan    area    and    municipal            of access, etc.,
                   levels.   Such   a   mechanism   should   promote    compati-            guiding and  helping  the  user  select  and  use  corn.
                   bility  between   the   subsystems   and   thereby   maximize            puter programs best suited to his needs.
                   the  possibilities  for   mutually   beneficial   information         2. Improving the utility  of  available  data  by  actively
                   spin-offs,  now  and  in  the   future.   A   consortium   of     eiicoura-ina data generating agencies to arrive at:
                   health  planners,  major   health   agencies   and   research             @  0
                   organizations   should   establish   a   health   information            compatibility of  key  items  on  report  forms  -  such
                   systei,n to serve this broker  function,  to  facilitate  the            as age, residencc, condition, service, etc.,
                                                                                            cojiipatibile  definitions  of  terms   used   in   such
                                                                                            reporting.
                   *"'flic Secretary is authorized, directly or by contract,  to         3. Identifying    common    unmet    data    needs,     and
                undertake research, development, demonstration  and  evaluation,     helping meet them by:
                relating to the  design  and  implementation  of  a  cooperative            promoting  addition  of  new  categories  of  inforitia-
                system for producing  comparable  and  uniform  health  informa-            tioii in existing data sources  -  such  as.  finer  age
                tion and statistics at the Federal, State and local levels."                breaks in a State census,


                48




            developing  directly,  or  contracting  to  develop,   new    and  promote  common  efforts  to  contract  for  or  in  other
            sources   of   information   -    such    as    population    ways  gain  access  to  commonly  needed  new  data,  such   as
            health surveys.                                               from  small  area  Population  health  surveys.  Finally,   the
          4.  Helping  users  Find  the  coi-nputer   resources   most    broker   system   staff   would   provide    direct    research
      appropriate to their needs by:                                      services,  such   as   file-merging   operations,   and   would
          * organizing   local    conferences    and    workshops    -    furnish  regular  monitoring  reports  on  health  and   social
            such as are now conducted by the census,                      indicators  requested  by  users.  However,  it   was   assumed
          * inventorying    and    brokering    use    of    agencies'    that  the  system  would   not   require   its   own   computer
            partially idle computer hardware,                             facilities at  least  in  the  foreseeable  future,  but  would
          * evaluating   software   packages,   and   purchasing   for    contract for the use of the necessary resources.
            joint  use,
          * demonstrating,   through   case   examples,    the    uses    Why a System of Stib-Systems?
            and    bene@its    of    new    advances    in    computer      A    coordinating     mechanism     between     independently
            science.                                                      organized  information  sub-systems  rather  than   a   central
          5. Developina policies regarding privacy:
                       0                                                  data   bank   was   dictated   by   users'   requirements   for
            formulating    policies    governin-    agreements     for
            sharing data,                      D                          flexibility, quality, and privacy - as well  as  by  the  state
            prornotina  codes  of   ethics;   specific   legal   safe-    of computer art. I will touch briefly on these points.
                                                                            FIexibility.-Health   researchers   need    to    tap    data
            guards.                                                       flowing  from  many  sources.  Although  much   of   it   comes
          6, Furnishing  routine   monitoring   and   special   status    from  the  operational   and   management   reporting   systems
      reports such  as:                                                   of  institutions  and  programs,  it  is  important  to  remem-
            trends in the locality's death rates,  health  facilities,    ber that  despite  the  overlaps  between  the  specific  types
            manpower, utilization, etc.                                   of  information   required   for   good   research   and   good
            comparisons with other regions, states, etc.
          7. Promoting  the  integration   of   separate   streams   of   program   management,   there   are   usually    marked    dif-
      data by:                                                            ferences in  the  characteristics  of  the  data  required  for
          9 negotiating   agreements   between   agencies   for   data    these  different  purposes.  For   example,   instant   on-line
              0                                                           inputs and retrieval  are  hardly  necessary  to  provide  data
            sharing,                                                      for  studying  the  effectiveness  of  appointment  systems  in
          o advising  on  legal  matters   and   computer   locks   to    following, no-show  cancer  patients,  yet  can  be  invaluable
            safeguard privacy,                                                    C,
          * conducting   file   merging   operations   and   providing    for   actual   appointment   scheduling.   Above    all,    the
            tables  and  maps  -  such  as  county,  city  or   census    particular characteristics  of  the  data  a  researcher  needs
            tract  profiles  showing  health  status,  mortality,  the    usually  changes  with  every   new   problem   he   addresses.
            population's  use  of  hospitals  and   health   resources    For each  study  he  may  need  not  only  different  types  of
            -  and  utilization   profiles   according   to   service,    data,  but  different  geographic   breaks,   frequencies   -of
            patient  characteristics,   conditions,   and   proportion    data  updating,  degrees  of  individual  patient   identifica-
            of community served.                                          tion, etc.
                                                                            At  an  even  higher  level  of  generality,  maximum  flexi-
      It  is  assumed  that  any   such   information   system   would    bflity  is  imperative  in  a  system  designed  to  serve  the
      build  in  its   own   evaluation   process   and   would   con-    information  needs  of  policy  makers.   There   can   be   no
      tinuously re-cycle  on  the  basis  of  experience,  new  health    fixed  solutions  to  the  problem  of  providing  health   in-
      research  and   planning   needs,   and   new   computer   tech-    formation  since  both  needs   and   solutions   are   dynamic
      nology.                                                             and ever-changing. Many methods of ea       re  and  facilities
          You  will  note  the  heavy   emphasis   placed   on   staff    for  treatment  we  regard   as   essential   today   will   be
      activities.  One  important  thrust  of  their  work  would   be    obsolete   or   unnecessary   ten   years   from    now.    New
      to  inventory  and   catalogue   data   sources   and   computer    methods   of   payment   will   be    adopted.    New    health
      hardware  resources  in  the  region,  and  to  evaluate   corn-    professions  will  emerge.  Information   to   serve   research
      Puter  software  packages.  Another  set  of   functions   would    and policy  makers  must  therefore,  above  all,  be  designed
      relate   to   improving   the   quality   of   the   data,    by    to  anticipate  and  to  accommodate  to  change.   A   network
      negotiating    format     compatibilities,     and     promoting    of sub-systems permits this.
      adoption   of   common   definitions.   Again,    staff    would       Quality.-In  view  of  the  massive   data   base   required
      'help  negotiate  inter-agency  agreements  for  data   sharing,    and  the  large  number  of  ffles  that  might  need   to   be
                                                                                                                                       49





                 tapped for all the  various  types  of  health  delivery  system    this  model  or  any  other  cooperative  health   infori-nation
                 evaluation that might be  desired  now  and  in  the  future  it    system.  The  answer  will  deteri-nine   whether   the   system
                 would  be  sheer  fantasy  to  expect  that  any  one   central-    ever  actually  gets  funded   and   into   operation;   whether
                 ized  system   could   incorporate   them   and   manage   their    those  who   generate   the   needed   data   will,   in   fact,
                 updating  and  quality  control.  Nor   would   that   be   pro-    contribute to it, and  finally,  whether  it  will  truly  serve
                 gress. It is far more  desirable  that  each  organization  have    the  purposes  of  research  and  policy  guidance   for   which
                 a genuine  and  active  concern  within  itself  to  continually    it was designed.
                 improve   its   own   information   management,   while   taking        The  National  Center  for  Health  Statistics,   which   as
                 due  cognizance  of  the  needs  and  requirements  of   others.    you  may  know  has  recently  published  a  description  of   a
                    Privacy.-The  privacy  issue   was   another   major   factor    model for state centers  for  health  statistics,  states  as  a
                 in   recommending   a   broker   system   where   every   agency    cardinal  premise  the  absolute  need  for   information   that
                 would  maintain  custody  of  its  own  files.   Clearly,   data    is  completely  unbiased  and  authoritative.  I   quote:   "The
                 sharing  is  an  area  fraught  with  fears  and  ambiguities  -    inevitable  disagreements   on   how   to   deal   with   health
                 where  the  power  of   information   can   be   used   on   in-    problems  must   not   be   confounded   by   controversy   over
                 dividuals  and  institutions  alike  for  good   or   ill.   And    the basic facts of the situation ... This  also  means  that  no
                 where the conclusions as  to  what  is  good  and  what  is  ill    pertinent facts be suppressed.  .  In  effect,  the  statistical
                 depend  very  much  on   who   is   making   them,   and   under    function  must  be   discharged   with   high   competence   and
                 what  circumstances.  Or,   more   succinctly,   whose   ox   is    cannot  be  captive  to  a  particular  point  of  view."   Thus
                 being  gored.  Confidentiality  of  information   about   insti-    the  N.C.H.S.  model  calls  for  the  information   system   to
                 tutions and organizations  relates  clearly  to  issues  of  the    be   administratively   independent   of   any   one    planning
                 confidentiality  of  their  actions   and   effectiveness.   The    agency, though with strong working relations with all.
                 Boston study, as  the  better  part  of  valor,  resolved  these        But  how  does  one   identify   an   administering   agency
                 issues  by  recommending  data  be   limited   to   that   which    which  will  command  the  trust  and  respect  of  all,  in  an
                 could  be  used   in   aggregated   form,   and   by   promoting    environment   where   knowledge   is   indeed   power   -    and
                 specific    inter-agency    agreements    on    data     sharing    where, in  almost  all  programs,  worry  about  loss  of  power
                 designed  with  appropriate  legal   consultation.   After   the    is the name of the game?  If  there  is  an  answer  at  all  to
                 system   had   proved   itself    and    appropriate    controls    this auspices question, I suspect it will  be  a  different  one
                 designed,  moves   mi-ht   be   made   towards   more   specific    in  each  region  or  state.  Some  possibilities  to  be   con-
                 sharing of fine--rained data.                                       sidered are:
                   Computer     Limitations.-A     huge,     centralized     data        o a generalized state statistical center,
                 system  incorporating   many   files   presents   problems   not        o some  other  state  agency  (possibly   the   university),
                 as  yet  adequately  solved  by  computer  science.  With   long        o a regional commission or center,
                 lead  times  for  design,  by  the  time  such  a  system   goes        o a quasi-public information authority.
                 into operation it is apt already  to  be  behind  the  state  of        In addition  to  auspices,  many   important   questions   of
                 the  rapidly  changing  computer  art  -  to  become  a   vastly    staffing and function, of  cost,  and  of  the  cost-benefit  of
                 expensive  antique.  Such  disasters  have   occured   regularly    such   information   systems   remain   to   be   explored.   If
                 in  the  urban  information  systems  so   hopefully   installed    Congress  does  now  authorize  the  funding  of   experii-nents
                 in  the  late  60s.  The  M.I.T.  computer  scientists  on   the    in  cooperative  health  information  activities   perhaps   you
                 Boston   study    recommended    instead,    careful    develop-    can  all  soon  begin   learning   by   doing.   Certainly   the
                 ment   towards   a   network   structure   among   participating    failure  to  develop  satisfactory  efforts  along  these  lines
                 programs,  where  hopefully  in  the   future   a   variety   of    can  only   mean   the   continued   burden   of   handicap   to
                 computers  of  different  types  and   sizes,   with   different    those  who  try  to  measure  the  successes  and  failures   of
                 hardware  and  software  configurations   might   be   able   to    our operating  prograi-ns  and  thus  to  give  the  public  the
                 talk  to  each  other   under   the   control   of   appropriate    most value for its health dollar.
                 permissions.  They  expect  that  the  next  decade   may   well
                 witness   revolutionary    software    and    hardware    break-        lPeter J. Henriot, "Political Questions about  Social  Indi-
                 throughs to make this possible.                                     cators," IVestern Political Quarterly, Vol. XXIII, No.  2,  June
                                                                                     1970.
                 ftlio Plays the Role of the Broker?                                     2NIoynihan, Beshers, and Cydel. Problems and Perspectives in
                                                                                     the Design  of  a  Community  Health  Information  System  (U.S.
                   Undoubtedly  this  is  at  once   the   most   sensitive   and    Public Health Service Contract PH  110-234),  Joint  Center  for
                 the  i-nost  crucial  -question  to  be  faced  in  implementing    Urban Studies of M.I.T. and Harvard, Feb. 1969.


                 50



       The Northern New England Ptepional Medical Program              opportunity  to  accumulate  experience  with  the  technical
                                      n                                and   management   problems   of   developiiia   large   data
                   Health Planning Data Base                                                                       t,
                  JOHN E. WENN BERG, M.D.                              systems. it also  tllows  one  to  evaluate  the  utility  of
                                                                       components  of  the  system.  This  should  be  of  value  to
                                                                       the  future  development  of  prospective,  population  based
       Those  of  us   involved   in   Regional   Medical   Programs   health information  systems  which,  I  think,  clearly  will
       are  continually  reminded  that  health  plannin-,   without
                                                     C,                be given central roles as part of the iiianacyenient  strlic-
       an  adequate  data  base,  is  more  of  a  visceral  than  a                                               C,
       cerebral process. We are often  asked  to  support  solutions   ttire of a iiitioiiil health insurance system.
       which clearly are  proposed  without  proper  ideiitificatior      The  immediate  purposes  of  the  Northern  New   England
                                                                       Recioiial Nledical Proorai-n data base  are  to  provide  in-
       of  the  problems  involved  and  are  usually  made  without      C,                 LI
                                                                       formation for lic@ilth problem identification  and  pro-raiii
       reference to other priorities. To a very  large  extent,  our                                                        0
                                                                       plannin-,  evaluation  and  management.  It  supports   plan-
       informational  base  for  planning  decisions  in  health  is         Z,
                                                                       nino efforts  at  the  areawide  and  state  health  plannina
       limited  to  impressionistic,  non-verifiable  opinion  corn-      0                                                        C,
       nionly  arranged  or  provided  by   parties   advocating   a   levels. A primary Customer of the  system  is  therefore  the
       particular  solution.  Under  these  circumstances   planning   Vermont     Comprehensive     Health     Planning     Agency.
                                                                       Contractual   arrangements   have   been   made   with   that
       decisions run a high risk of being - at best-irrelevant.                          t,
       A    prospective    population-based    health    information   acency  to  supply  them  with  necessary  information.   The
       system appears  as  a  particularly  attractive  solution  to   data base also supports  planning  and  operating  activities
       our  data  problems.   As   you   have   heard   today,   the   of the  Re@ional  Medical  Program,  including  primary  care
       necessary technology is not obscure. In  fact,  it  has  been   activities  and  disease  control  and  continuing  education
       available  for  some  time.  Why,  then,   does   it   remain   programs.  Finally,  certain  features  of  the  system  have
       generally  unimplemented?  The  reason  for  this  cannot  be   . been Of use  to  operating  health  agencies  and  in  some
       simply cost;  it  can  be  convincingly  argued  that  health   instances to planning  agencies  outside  of  our  area.  For
       information  systems  could  more  than  pay  for  themselves   example,  aspects  have  been  utilized  by  Vermont  Planned
       by providing the informational base  for  wise  decisions.  A   Parenthood,  The  Province  of  New  Brunswick   in   Canada,
                                                                       The  Maine   Re-ional   Medical   Program   and   the   Maine
       more  probable  obstacle  to  establishing  prospective   in-                   0
       formation systems derives from the  direct  lack  of  utility   Facilities Planning Council.
       of the system  to  the  provider  of  the  data.  To  provide      Basically the  data  system  provides  a  characterization
       accurate data is a bother'  and  the  effort  to  produce  it   of the health system in terms of:
       must be either rewarded or required by law.    Under      our      1. the  coi-nniunities  being   served   in   demographic,
       existing  pluralistic  planning   and   management   systems,         socioeconomic environmental terms;
                                                                          2. the manpower, facility  and  dollar  resources  of  the
       good planning  is  neither  strongly  rewarded  nor  required         health delivery system;
       by law.  Under  these  circumstances  the  establishment  and      3. utilization supply  and  distribution  aspects  of  the
       maintenance  of  prospective  health   infon-nation   systems         health care system;
       are expensive - probably intolerably  expensive  -  in  terms      4. outcome,  as  measured  by  morbidity,  mortality   and
       of   currently   available    management    and    pursuasive         patient satisfaction.
       energies,                                                          The   major   products   are   planning   documents    and
        If  the  current   Regional   Medical   Program   and   Com-   status  reports   covering   the   above   mentioned   areas.
       prehensive  Health  Planning  legislation  does  not  contain   Examples  are  available  froi-n  the   Program   office   on
       the  mandate  necessary   to   promote   prospective   health   request.
       information  systems,  is  there  an   alternative   approach
       which can begin  to  achieve  the  data  base  necessary  for      Establishing the data base has  required  a  major  effort
       planning and  management  systems?  I  think  the  answer  is   which  cannot  be  systematically  reported  at  this   time.
       a  qualified  Yes:  under  certain  circumstances,   Regional   However, I would like to el     aborate  on  five   important
       Medical Programs can  establish  an  ad  hoc  but  systematic   features of our approach:  (1)  choice  of  the  New  England
       data  base  which  minimizes   administrative   inconvenience   town  as  die  geographic  base;   (2)   strategy   governing
       to  participating  institutions  and  is  at  the  same  time   collection of  data;  (3)  resume  of  the  contents  of  our
       highly  useful   to   its   own   planning   and   evaluation   data file; (4) approach  to  data  processing;  (5)  approach
       Purposes,  to  Comprehensive  Health  Planning   and   other'   to data analysis.
       health agencies. In addition  to  the  immediate  utility  of      The  geographic  region  covered  by  the  data  base  in-
       the  data,  establishment  of  ad  hoc  systems  affords  the   cludes  the  entire  service  area  of   the   Northern   New
                                                                                                                                   5 1






                    England     Regional     Medical     Program.      However,      in    records  -  we  have   used   part-time   Regional   Medical   Pr
                    designing  our  approach,   we   wished   to   use   the   smallest    grant  personnel  under  close  core   staff   supervision.   Und
                    feasible   geographic   unit   that   was   available.   The    New    these     circumstances     cooperation     has     been     near
                    England  town  turned  out  to  be   nearly   ideal   -   for   the    universal.
                    following  reasons:  (1)  it  appears  in  the  census;  (2)   most        While  much  of  our   data   base   spans   more   than   oi
                    unit  records  in   the   region   (for   example   hospitals   and    year and    is updated periodically, the costs involved
                    vital  records)  contain  the  individual's  town  of  residence  -    fielding special utilization surveys led to a decision
                    thus  utilization  rates  can  be  calculated  on  a  town   basis;    restrict (at least initially) the complete utilization file
                    (3) there are a total  of  356  distinct  towns  in  the  region  -    the   calendar   year   1969.   Informational   items    correct(
                    251 towns and  gores  in  the  State  of  Vermont  -  5  1  in  the    through  special  protocols  have  been   kept   to   a   minimur
                    three  counties  of   upper   New   York   -   and   54   in   con-    These    include    patient    record    number,     age,     se:
                    tiguous  portions   of   New   Hampshire.   Populations   in   each    diagnoses, procedures performed, length         of stay,  date  (
                    town  vary  b?tween  35,000   and   IO,   with   a   median   value    admission,   type   of   insurance,   referring   and    attendir
                    of  about  1500.   Thus,   using   the   town   as   a   population    physicians.
                    base  allows  for   a   large   number   of   discrete   geographic        Resume  of  Contei7t  of  Data  Files:  Currently,  our   dal
                    units  in  the  system.  This  in  turn  provides  great   analytic    files contain the following information:         '
                    flexibility.                                                               1.  Utilization   review:   hospitals,   nursing   homes   an
                      Strategy  governing  collection   of   data:   all   effort   has          home health agencies.
                    been  made  to  avoid   duplication   of   existing   data.   When-        A complete review - based  on  unit  records  -  of  all  are
                    ever  possible,  we   have   used   existing   sources   of   data,    hospitals  for  the  year  1969.   68,000   records   were   take
                    either  published  or  existing  in  unit  record  files  collected    from  PAS  and  29,000  collected   by   staff   review   of   th
                    by cooperating agencies:                                               hospital  records.  In  addition,  referral  hospitals   in   Har
                          Existing   data   includes   those   collected,    processed,    over, Albany and Montreal have been reviewed.
                      and   published   by   local,   state,   federal   and   national        A  complete  1969  review   of   all   area   nursing   homeE
                      agencies:   for   example,   reports    of    the    Bureau    of    (8S homes, and 4,000 records).
                      Census,   National   Center   for   Health   Statistics,    Amer-        A  complete  review  of  area   home   health   agencies   (4
                      ican    Hospital    Association,    Blue    Cross/Blue    Shield,    agencies, and 8,000 records).
                      State    Health    Department    and    The    State     Planning        2. Vital records:
                      Agency.                                                                  Through    excellent    cooperation    with    the    Vermont
                         Existing   unit   record   files   include   those   collected    N.H.  and  N.Y.  health  departments,  decade  files   of   birt]
                      by  operating  agencies  and  i-nade  available   to   the   Pro-    and  death  records  have   been   established.   Mortality   dat
                      grain  by   special   cooperative   arrangements:   by   way   of
                      example,  a   three   year   file   of   200,000   patient   dis-    is  a  particularly  useful  source  for  defining  major  healti
                      charge   abstracts   obtained   from   the   hospitals   partici-    problem areas for measuring outcome.
                      pating   in   Professional    Activity    Survey    (PAS),    the        3. Manpower file:
                      decade   files    of    the    Ven-nont-New    Hampshire    Vital        Hospitals  staff  listings  have  been   obtained   from   al
                      Records  and  the  individual  tax   returns   from   the   State    institutions   of   the    region.    Health    Department    anc
                      of Vermont Tax Department for 1967.                                  AM.A.  registries  are  being  utilized  to  classify  physician!
                                                                                           in the region by locality of practice,      specialty    training
                      Special  collection   protocols   have   been   established   for    age,  board  certification  etc.,  both  on  a  current  and   ar
                    missing"  data.   This   includes   surveys,   conducted   by   the    historical basis.
                  staff,   of   hospitals,    nursing    homes    and    home    health        4. Facilities:
                  agency   records.   It   also    includes    a    household    survey        In   cooperation   with   the   Tri-State   Regional   Medica
                  capability.*                                                             Program,    special    inventories    of    hospital    facihtie@
                      The    avoidance    of    administrative     inconvenience     to
                  institutions  in   providing   data   is   fundamental   to   success    throughout   the   region   have   been   completed   with    the
                  of   an   ad   hoc   data   system.   When   data   collection    has    following  areas   being   stressed;   coronary   and   intensive
                  required   staff   time   -   such   as   reviewing   unit   hospital    care,   emergency   care,   stroke   care,    radiotherapy    and
                                                                                           chronic pulmonary care.
                                                                                               In   addition,   published   data    encompassing    facility
                      *While an integral part of  the  "data  base",  this  paper  does    staffing, size and location  as  well  as  cost  data  have  been
                  not discuss the NNE/RMP social survey capability.                        compiled  from  a  variety  of   secondary   sources   for   hos-

                  52



       pitals,  home  health  agencies   and   nursin-   homes.   Sources    age  to   problems   in   format   design   and   basic   character
                                                   t,
       include   Blue   Cross,   American   Hospital   Association    and    configurations.  As   an   example,   sources   of   data   include
       state agencies.                                                       magnetic   tape   obtained    from    PAS    (Minneapolis    Honey-
         s. Socio-econoinic and environmental:                               well),   Vermont   State   Government   (General   Electric)    and
         Arrangements   have   been   made   to   secure   1970    census    New    York    Health    Department    (Burroughs).    To     solve
       tapes   containing   available   processed   tables.   This   will    these   problems   generalized   recoding   and   formatting   pro-
       establish  age-specific  population  rates  on   a   town   basis.    grams have been developed.
       of  particular   importance   is   intercemal   estimates   on   a      Approaches   to   Data   Analysis.-The   usefulness    of    ttie
       town  and  other  small   area   basis.   Work   has   been   com-    data base  relates  to  1)  the  completeness  of  each  file  (for
       pleted   in   conjunction   with   the   State   Health   Planning    example,  one  year  of   hospital   experience   for   the   total
       Agency   personnel   to    construct    inter-censal    population    population)  and,  2)  the   inclusiveness   of   the   system   in
       age-sex structure for towns and counties.                             terms  of  the  large  numbers  of  separate  data  files  contain-
         Indicators   of   economic   status   are   beinc,   constructed    ing   relevant   health   data.   This   enables   (for    example)
       through  the  use  of  income  data.  In  conjunction   with   the    correlations     between     demographic     and      environmental
       State   Tax   Department    individual    income    tax    returns    factors  in  health  status.  Much  of  the   analysis   undertaken
       have  been  analyzed  by  town,   occupation   and   industry   to    by  the  RMP  has  been  computer  based   and   allows   for   the
       provide  an  economic  profile  of  the   State   and   its   sub-    study   of    complex    relationships    between    "input"    and
       divisions.                                                            "output"    variables.    Examples    of    correlation    analyses
         6. Published data:                                                  that  are  possible  include  relationships  between   per   capita
         For   example,   complete    set    of    reports    from    the    income,   admission   rates,   death   rates,   infant    mortality
       National Center for Health Statistics.                                rates,   expenditures   for   medical   care,   procedure    rates,
         Approach     to     Data     Processing.     Routine     reports    etc.
       prepared   by   acencies   and   organizations   in   the   health       While  a   number   of   general   statistical   programs   have
       field  rarely  provide  direct  answers  to   specific   questions    been  adapted,  we  have   also   developed   a   series   of   new
       as  they   arise   in   planning,   manacenient   and   evaluation        .  novative   types   of    health    system    analysis.    Of
                                            C,                               an  in
       activities  within  a  local  or   regional   context.   Processed    particular  note   is   a   program   des.igned    to    characterize
       data,   organized   and   tabulated    according    to    external    total   utilization   and   allocation   of    medical    resources
       dictates,  is  often  irrelevant  to   immediate   concerns.   The    relative  to  the  patient  service  areas  of  particular   insti-
       limited  utility  of  reports  furnished  hospitals  by  the  Com-    tutions.  This  includes  resource  allocation   rates   in   terms
       mission   of   Professional   and   Hospital   Activities    (PAS)    of  admissions,  patient  days,  beds,  dollars  or  skilled   man-
       and  of  publications  of  state  and   federal   health   depart-    power.   Because   virtually   all   utilization   experience   for
       ments  reflect  the   series   of   compromises   that   must   be    each  town  in  the  region  is   known,   these   rates   describe
       made   in   developing   multi-purpose   reports.   From   several    the  total  experience   of   the   population,   Thus,   for   the
       standpoints,   the   most   effective   method   of    information    first  time,  an  accurate  estimate,  based  on  a  small  popula-
       storage is raw data  on  individual  cases.  This  is  particular-    tion, is possible:  this  includes  total  cost  for  institutional
       ly  true  when  efficient  storage  and  retrieval   methods   are    care,  procedure  rates,  bed  utilization   and   beds   available
       available.                                                            rates, etc.
         ,Accordingly,  the  RMP  has  devoted   a   significant   effort        During  the  next   year,   the   NNE/RMP   will   complete   a
       to   the   development   of   individual   case   files.   Because    number   of   reports   for    areawide    and    state    planning
       accessible  data  derives  from  diverse  sources,  a  number   of    purposes.  I  hope  that  the  next  time  I  report  to   you   on
       compatibility    problems    have    been    encountered.    These    the  data  base  we  will  have  much  more  to   say   about   the
       range from  differences  in  coding  of  such  items  as  sex  and    effect the data has had on the planning process.






                                                                                                                                              53




                                WORKSHOP ON MEASURING CHANGES IN BEHAVIOR

                                                                  Participants
      john S. Lloyd, Ph.D. - Moderator                                       Mitchell Schoro@v
      Assistant Coordinator, Evaluation                                      Assistant Coordinator for Educational
      California Regional Medical Program -                                     Planning and Evaluation
            Area V                                                           intermountain Regional Medical Program
      William R. Crawford, Ed.D.                                             Barbara J. Andrew, Ph.D.
      Associate, Evaluation Studies Section                                  Assistant Professor Medical Education Research
      Center for Study o@ Medical Education                                  Division of Research in Medical Education
      University of Illinois                                                 University of Southern California

                   Nleasuring Changes in Knowledge
                       WILLIAM R. CRAWFORD                                   Multiple Ozoice Items
                                                                                 A. Advantaues
                                                                                              0
            Sometimes   the    measurement    of    knowledge    seems    to            1. Some task clearly defined for each exwninee
            be  a  straightforward   procedure.   Perhaps   that   is   true
                                                                                        '-). Larce sample of items permissible
            when  one  is  interested  in  measurement  of   simple   recall                Z,
                                                                                        3. Scorin- keys are standardized
            of  basic   information   which   has   been   memorized.   How-                   0
            ever,  simple  recall  of  basic  information  is  usually   not            4. Easy to score
            sufficient  for  measuring  the   achievement   of   educational           B. Disadvantages
            objectives  in  areas  as  conceptually  complex   as   medicine            1. Requires recognition of correct response, not
                                                                                         production of it
                                                      ost   cases   we   are
            and the allied health professions. In rn                                    2. Permits guessmg
            interested  in  assessing  chanues  which  are  related  to  the
                                        0                                               3. Difficult to construct
            ability to  apply  principles,  solve  problems,  and  interpret            4. Task is completely structured
            data,  to  name  only  a  few.  Clearly,  these  complex  intel-
            lectual  functions   cannot   be   assessed   with   instruments            Nleasuring Changes in Clinical Performance
            designed  to  provide  an  estimate  of  the  number  and   kind
            of memorized facts which can be recalled.                                         BARBARA J. ANDREW, Ph.D.
            How,  then,   can   we   approach   the   greater   problem   of       The  health  professional's  ability  to solve clinical
            measuring  the  ability  to  engage  in  more   complex   intel-    problems has long been regarded as one of the most
            lectual  functions?  The  obvious  first  step  is   to   define    important dimensions of quality health care delivery.
            what  those  functions  are,  why  they   are   important,   and    Yet because of its complexity and the challenges which
                                                                  be    per-    it presents for quantitative measurement, clinical per-
            how they relate to specific tasks which must
            formed on the job. Defining these functions is        a    major    formance has not been as widely used as a criteria for
            operation,  and  an  essential  step  before          specific      evaluation as its importance would suggest.
            measurement   instruments   can   be   developed.   The   second            Clinical performance is essentially a problem solving
                                     initions  and   translate   them   into    process which involves:
            step is to take these def
            instruments  which  can  validly  and   reliably   measure   the            1. knowing what data are relevant;
            functions,  and  which  will  produce  meaningful   data.   Con-            2. gathering the data;
            current  with   the   development   of   the   instruments   one            3 .analyzing the data and evaluating their relative
            must  develop  a  procedure  for  scoring   and   a   plan   for            importance and significance;
            reporting and interpreting the scores.                                      4. synthesizing the data into conclusions;
                                                                                        5 .knowing about available health care strategies;
              Following  is  a  brief  outline  of  the  topics   covered   in          6. selecting  and  applying  the  most  appropriate
            this  session  of  the  workshop,   each   of   which   was   con-          strategies;
            sidered in more depth in the working session.                               7. evaluating the effectiveness of the strategies;
                                                                                                                                                55





                 8.  making  whatever  changes  in  health   care   strategies         and  used  by  different  raters  (e.g.,   non-verbal   as@
                    which are needed.                                                  well  as  verbal  interaction  during  history  taking)]  ;
                 Specitic  clinical  problem   solving   activities   can   be      7. finally,  the  validity  and  reliability  of  the  instru-
             classified   as   primarily   diagnostic   or   therapeutic    in         ment  are  estimated.   (In   instances   where   the   in-
             nature.  That  is,  while  diagnosis  and  therapy   are   inter-         strument  has  been  designed  for   use   by   raters   to
             dependent  components   of   clinical   problem   solving,   some         observe  clinical  performance,  sufficient   training   to
             health   professionals   have    primary    responsibility    for         improve   inter-rater   reliability   should   be    under-
             diagnosis,  while  others  are  concerned  with   suggesting   or         taken).
             administering  therapeutic   procedures.   Still   other   health      The  selection  of  appropriate   validity   and   reliability
             professionals,  such  as  the  physician,  are  responsible   for    estimates  depends  upon  the  nature  of  the   measuring   in-
             diagnosis as well as therapy.                                        strument  itself  and  upon  the  purposes  for  which   testing
                 The   measurement   of   clinical   performance   can   focus    data are gathered (3).
             either  upon  the  entire  problem   solving   process   employed      In  estimating  the   reliability   of   observation   devices
             by  a  specific  health   professional   or   solely   upon   the    one  needs  to  determine  the  correlation  among   the   eval.
             frequency   with   which    certain    behaviors    within    the    uations  of  several  raters  of  the  same  clinical   perform-
             process  are  observed.  In   measuring   changes   in   clinical    ance.   This   procedure   necessitates   the   refinement   and
             performance  to  determine  the   effectiveness   of   particular    careful definition of  the  skills  to  be  measured  and  cate-
             experimental   treatments,   the   decision   to   observe    the    gories for  recording  performance,  as  well  as  the  training
             entire  problem   solving   process   or   only   some   specific    of observers so  that  acceptable  inter-rater  reliability  can
             behaviors  within  the  process  will  be  a  function   of   the    be achieved.
             purposes  of  the   study   and   the   hypotheses   which   have      When  the  measuring  device   consists   of   a   paper   and
             been stated.                                                         pencil test of clinical  performance  or  the  simulation  of  a
                 The   validity   of    clinical    performance    measurement    clinical  situation,  comparability   of   forms   and   compar-
             will, of  course,  rest  upon  the  quality  of  the  instruments    isoiis over  time  offer  the  best  estimates  of  reliability.
             which are devised to record the problem s           olving    be-    Estimates  of   the   test-retest   reliability   of   simulated
             havior.  The  following  procedures   should   be   followed   in    clinical  performance  test   are   complicated,   however,   by
             the development of such instruments:                                 the  fact  that  these  simulation  tests  permit  the  examinee
                 1  .the clinical skills to be measured are identified;           to  receive  feedback  from  his  selections  and,   hence,   to
                 2. criteria for evaluating these skills are developed;           some  extent  constitute  a  learning  situation.  Even  if  the
                 3. the criteria are stated  in  terms  of  specific  clinical    time  interval  between  test  administrations   is   lengthened
                    behaviors;                                                    to  enhance   forgetting,   one   cannot   control   intervening
                 4. a method  of  scoring  is  developed  which  is  logically    variables   which   might   improve   the   subjects'    problem
                    appropriate to the skills being measured;                     solving skills.
                    a. the  assignment  of  differential  scores  to   various      Since  in  i-neasuring  changes  in   clinical   perfori-nance
                       levels  of  performance  should  be   clearly   defined    one  is  primarily  interested  in  determining  the  degree  to
                       and  require  as  little  subjective  judgment  of  the    which  the  health  professional  possesses   certain   clinical
                       rater as possible;                                         problem solving skills,  the  use  of  criterion-related  valid-
                    b. scoring   intervals    need    to    be    sufficiently    itY is somewhat  less  pertinent  than  is  construct  validity.
                       sensitive  to  permit  the   discrimination   of   dif-       The  establishment  of  construct  validity  can  be   under-
                       ferent levels of clinical performance;                     taken   by   hypothesizing   outcoi-nes   of   performance   for
                 5. prior to establishing  the  validity  and  reliability  of    various  groups  on  the  problem   solving   test,   and   sub-
                    the  instrument,  extensive  pre-testing   is   undertaken    sequently  adi-ninistering  the  test   to   determine   whether
                    to  deten-nine  its  usability  and  capacity  to  measure    the   hypothesized   outcomes   occur.   In   instances    where
                    all relevant aspects of the specific clinical skills;         other  tests  of  the  same  clinical  performance  exist,   the
                 6. if  the  instrument  is  to  be  used  by  a   rater   who    correlations  between  the  test  being  developed   and   these
                    observes an  actual  or  simulated  clinical  setting,  it    other measures should be estimated.
                    should   not   attempt   to   measure   more   than    can       Regardless  of  the  kinds  of   validity   and   reliability
                    reasonably  be  observed  and   recorded   by   a   single    which  are  considered  appropriate  for  a   specific   measure
                    individual.    [If    two     or     more     simultaneous    of  clinical  performance,  the  subjects   on   whom   validity
                    dimensions  of  clinical  performance  are   to   be   ob-    and   reliability   studies   are   conducted   should   closely
                    served,   additional   instruments   can   be    developed    resemble  the  population   for   whom   the   test   has   been

                56



                                    ir    composition     and     relevant     physical     examination,     laboratory     procedures,      and
        designed, in  terms of the                                             therapy.   particularly   relevant   to   measurement   of   this
        characteristics.                             asurei-nent        of     kind  is  that  the  evaluation  of  clinical  performance  be  a
        Two  general  approaches  to  the  me                                            specific   disease   entities   and   the    diagnostic
        clinical  performance  may   be   taken:   1)   the   direct   ob-     function of                  hicli are indicated f L'
        servation    and    measurement    of    actual    or    Simulated     and therapeutic procedures W                       or       each.
        clinical  situations;  2)  the  indirect  measurement  of   actual     Thus,  the  validity  of  the   conclusions   drawn   from   this
        or simulated clinical situations.                                      kind   of   measurement   depends   not   only   upon   the   ap-
        The  advantages   of   evaluating   actual   clinical   situations     propriatciicss  and  sensitivity  of   the   observation   forms,
        result  primarily  from  the  difficulties  in   simulating   some     but  upon  the  clinical  competence  of  the  observer  who   in
        of  the   complexities   and   spontaneous   aspects   of   actual     an  actual  situation  must  not  only   record   physician   be-
        problem  solving  settings.  For  example,   the   clinical   per-     havior,  but  must  develop  his  own  diagnosis  in   order   to
        forrnance  of  some  medical  technologists   requires   the   use     evaluate  the   appropriateness   of   that   behavior   to   the
        of   actual   specimens,   thus   rendering   observations   under     particular clinical problem.
        simulated    conditions    considerably    distorted    and     of     While    the    direct    measurement    of     a     physician's
        limited value. This sai-ne difficulty  is  posed  by  the  use  of     complete iiiani(,,eiiient of a clinical case results  in  a  more
        simulated  patients   from   whom   the   physician   could   take     comprehensive    evaluation     of     physician     performance,
        a  history  and   perform,   in   some   instances,   a   physical     soriie  studies  have  focused  upon   specific   components   of
        examination,   but   on   whom   it   would   be   impossible   to     patient  iiiatiioeiiient  (2,  14).   Foster   and   Lass   (14')
        perform   laboratory   procedures    not    only    because    the                 0
        obtained  data  would  be  inconsistent,  but   because   of   the     will  soon  be  reporting,   procedures   for   the   measurement
        understandable    unwillingness    of    subjects    to    underco     and  evaluation  of   patient   interviewing.   The   measurement
                                                                         0     of   patient   interviewing   skills   can   emphasize    content
        such  experiences.  Thus,  the  use  of  simulated  clinical  set-     (how  much  and  what  kinds   of   information   are   elicited)
        tin(ys  restricts  to  some  extent  the  range  of  skills  which
         LI                                                                    and/or  process  (the  techniques  used   to   elicit   infon-na-
        can be measured.                                                       tion).  In  order  to  measure  the  process  of  patient  inter-
         However,    since    the    measurement    of    clinical    per-     viewing   one   needs   to:   1)   identify   those    dimensions
        formance  is  generally  for  the   purpose   of   assessing   the     which will account  for  all  possible  aspects  of  interaction;
        effects  of  an  independent  variable   upon   clinical   problem
                                                                               2)   determine   whether   these   dimensions   are   essentially
        solving  behavior,  or  to   make   comparisons   among   individ-     verbal  or  non-verbal;  3)  develop  observation   forms   which
        uals   regarding   their   clinical   competence,   the   use   of     provide  sufficient  scope  and   flexibility   to   permit   the
        actual  clinical  settings  may  pose  difficulties  in  obtaining     recording   of   relevant   aspects    of    communication    and
        uniform   testing   conditions   and    in    securing    adequate     interaction.
        numbers  of  subjects.  Thus,  if  one  wanted  to   measure   the
        effects  of  an  instructional   film   on   the   management   of      Barrows   and   Abrahamson   (4)   have   reported    the    use
        hypertensive  patients  in  a  hospital  clinic  one  would   need     of trained  actors   to   simulate   patients   with   neurological
        a  sufficiently  large  patient   population   randomly   assi-ned     disorders  in  order  to  measure  history  taking  and  physical
        to  clinic  physicians  in  order  to  permit  valid   conclusions     examination  skills.  Although  the   use   of   the   programmed
        to be drawn.                                                           patient  imposes  limitations  upon  the   kinds   of   disorders
          The   decision   to   employ   either   direct    or    indirect     which  can   be   simulated,   the   pre-determined   nature   of
        measurement    of    clinical    performance    in    actual    or     the  medical  setting  permits  more   accurate   evaluation   of
        simulated  situations  will  usually  be  based  upon   a   number     the  extent  to  which  pertinent  data   have   been   uncovered
        of considerations such as: 1) the kind     Iof clinical skills  to     by'the examinee.
        be  measured;  2)   the   availability   of   subjects   and   ob-       In    a    somewhat    different    approach    to    measuring
        servers;  3)  the  number  and  extensiveness  of   the   clinical     competence  in  data   gathering   and   analysis@   Cline   (7),
        skills   to   be   measured;   and   4)   the   amount   of   time     Langsley  (19),  and  Levit  (22)  have  reported  the   use   of
        required for observation.                                              motion  pictures   to   assess   observation   and   interpretive
           Peterson's study  of     North   Carolina    general    practi.     skills. The  films  which  consist  of  a  history  and  physical
        tioners   (28)   represents   perhaps   the   most   comprehensive     examination  show  a  wide  range   of   signs   and   symptoi-ns
        attempt  to   measure   physicians'   clinical   problem   solving     which   are   both   pertinent   and   non-pertinent    to    the
        skills  by  direct  observation  of  an  actual   situation.   The     formulation  of  a  correct  diagnosis.  The  data  is  presented
        observation   forms   developed   by   Peterson   and   his   col-     with  equal  emphasis   and   in   such   a   manner   that   the
        leagues  measure  the  physician's  skills  in   history   taking,     exan-dnee  must  analyze   all   data,   make   judgments   about
                                                                                                                                               57






                    their relative  significance,  and  draw  conclusions  con-                  5. Beaumont,  Graham,  et  al.  "Medical  Auditing   in   a   Comp
                    cernitig the nature of the patient's illness.                                liensive  Clinic  Program."   Journal   of   Medical   Educati4
                       The  medical  audit,  which  in  essence  is  an   a   posteriori         42:359-367, April, 1967.
                    evaluation  of  the  clinical  management   of   an   actual   case,         6. Butler, John J.,  and  Quinlan,  J.  William.  "Internal  Audit
                                                                                                 the   Department   of   Medicine   of   a   Community   Hospita
                    has     been     the     subject      of      numerous      articles         Journal of the  A?izerican  Medical  Association,  16  7:567-5'
                    (5,6,20,21,26,27,29,30).-Such     a     process     requires     the         May 31, 1958.
                    careful   establishment   of   criteria   by   which   the   medical         7. Cline,  Marvin.  "A  Film  Test  of  Clinical  Skills  in  Medi
                    record  is  evaluated  and  the  training  of  medical   specialists         Students."   Journal    of    Medical    Education,    36:908-9
                    who  will  serve  as  auditors  of  the  medical  record.  There  is         August, 1961.
                                                                                                 8. Colton, Theodore, and Peterson, Osler L     ' "An      Assay
                    the   danger,   however,   that   one   may   be    measuring    the         Medical  Students."  Abilities  by  Oral  Examination."   Jo@ri
                    accuracy   and   completeness   of   the   medical   records   them-         of Afedical Education, 42:1005-1014, November, 1967.
                    selves,  rather  than  the  clinical  performance   of   physicians.         9. Cowles,  John  T.  "A   Critical   Comments   Approach   to   t
                       Yet   another   indirect   evaluation   of    clinical    problem         Rating  of  Medical  Students'  Clinical  Performance."   Jouri
                    solving  is   the   so-called   "patient   management   problem"   -         ofMedicalEducation, 40:188-198, February, 1965.
                                                                                            10.  De   Tornyay,   Rlieba.   "Measuring   Problem-Solving   Skills
                    a  written  simulation   of   a   clinical   case   which   measures         Means  of  the  Simulated   Clinical   Nursing   Problem   Tesi
                    data   gathering   and   interpretive   as   well    as    decision-         JouriialofNursingEducation, 7:3-8, August, 1968.
                    inal@ing   skills   (10,16,25,31,33).   Although   its    use    has    11.  Donabedian,  Avedis.  "Promoting   Quality   Through   Evaluati
                    been  reported   primarily   with   physicians   and   nurses,   its         the Process of  Patient  Care."  Medical  Care,  6:181-202,  M@
                    applicability    to    other    health     professionals     appears         June, 1968.
                                                                                            1.2. Evans, Lloyd R., Ingersoll,  Ralph  W.,  and  Smith,  Edwin  Ja
                    feasible.  The  problem-solving   exercise   is   initiated   by   a         "The  Reliability,  Validity,  and  Taxonomic  Structure  of  t
                    brief  description  of  the  patient  and  consists  of  "a   series         Oral    Examination."    Journal    of     Medical     Educatio
                    of   sequential,   interdependent   decisions    representing    the         41:651-657, July, 1966.
                    various  stages  in  the  management  of  the   patient"   (25:   1)    13.  Foster, Judilynn T., et ad. "Analysis  of  an  Oral  Examinati(
                    in  which  the  results  of  each  decision   are   given   in   the         Used in Specialty  Board  Certification."  Journal  of  ilfedic
                                                                                                 Education, 44:951-954, October, 1969.
                    form   in   which   the   health    professional    would    receive    14.  -,  and  Lass,  Sandra   L.   "'ne   Identification   of   Intc
                    them  in  an  actual  clinical  setting.   Moreover,   the   problem         action   Patterns   in   Student-Patient         Communication@
                    not  only  allows  the  examinee   to   make   a   wide   range   of         Abstract of a Paper to  be  presented  at  the  1970  Confereni
                    decisions  froi-n  very  harmful  to  very   helpful,   but   forces         on   Medical   Education,   Association   of   American   Medic
                                                                                                 Colleges.
                    him  to  deal  with   the   consequences   of   his   decisions   by    15.  Hinz, Carl F., Jr., "Direct Observation as a Means  of  Teachil
                    presenting     additional     choices     through     which      the         andEvaluatingClinica]Skills."JournalofMedicalEducatio,
                    examinee   can   either   correct   or   further   coi-npouiid   his         41:150-160, February, 1966.
                    n-dstakes.   Allowances   are   also   made,    where    applicable,    16.  Hubbard,  John  P.  "Programnied  Testing  in  the  Examinatioi
                    for  the  use  of   more   than   one   acceptable   diagnostic   or         of  the  National  Board  of  Medical   Examiners."   Education
                                                                                                 Testing Service,  Proceedings  of  the  1963  Invitatiot7al  Co
                    therapeutic procedure.                                                       ferelice on Testii7gProblems.
                    .  The   following   selected   bibliography   has   been   included    17.           ., et.al. "An Objective  Evaluation  of  Clinical  Con
                    so  that  individuals  wi@ng  to   do   so   may   further   explore         petcncc:  New  Techniques  Used  by  the   National   Board   (
                    the literature on clinical performance measurement.                          Medical  Examiners."   New   England   Journal   of   Medicit2,
                                                                                                 272:1321-1328, June 24, 1965.
                                      Selected Bibliography                                 18.  Kilpatrick,  G.S.  "Observer  Error   in   Medicine,"   Journal
                                                                                                 ,Ifedical Education, 38:38-43, January, 1963.
                    1. Abrahamson, Stephen. "Evaluation in        Continuing     Medical    19.  Langsley, Donald G.  "Fflmed  Interviews  for  Testing  Clinic@
                       Education." Journal of the Ainericaii     Afedical   Association,         Skills."  Journal  of  Medical  Education,  45:52-58,   Januarl
                       206:625-628, October 14, 1968.                                            1970.
                    2. Adler,  Leta  McKinney,  and  Enclow,  Allen  J.  "An  Instrument    20.  Lenibcke, Paul A. "Evolution of  the  Medical  Audit."  Jotirne
                       to Nleasure Skill in  Diagnostic  Interviewing:  A  Teaching  and         of the  Aitiericait  MedicalAssociation,  199:111-118,  Februar
                       Evaluation    Tool."    Journal    of     Aledical     Education,         20,  1967.
                       41:281-288, March, 1966.                                             21.            .   "Medical   Auditing   -by   Scientific   Methods.
                    3. American  Psychological  Association,  et   al.   Standards   for         Journal  of  the  Ainerican  Medical  Association;  162:646-65'
                       Educational and  Psj,chological  Tests  and  Alatiuals.  Washing-         October 13, 1956.
                       ton, D.C.: American Psychological Association, 1966.                 22.  Levit, Edithe, J. "The Use of Motion  Pictures  in  Testing  th
                    4. Barrows,  Harold  S.,  and   Abrahamson,   Stephen.   "The   Pro-         Clinical   Competence   of   Physicians."   Annals   New    Yor
                       graiiimed  Patient:  A  Technique  for  Appraising  Student  Per-         A cadeiny of Sciences, 142:449-454, March 31, 1967.
                       fonnance in Clinical  Neurology."  Journal  of  iVedical  Edtiea-    23.  NlcGuire, Christine H. "Medical  Education,  Part  1:  A  Scien
                       tioii, 39:802-805, August, 1964.                                          tific  Approach  to  Problems   of   Professional   Assessment.


                    58




           Canadian Medical Association Journal, 100: 593-598, April  5,        Practice,  1953-1954.  Evanston,  Illinois:   Association   of
           1969.                                                                American i\lcdicai colleges, 1956.
      24,  _.  "The  Oral  Exai-nination  as  a   Measure   of   Profes-   29.  Phancuf, Maria C.  "Analysis  of  a  Nursing  Audit."  Nursing
           siotial   Competence."   Journal   of   Medical    Education,        Outlook, 16:57-60, January, 1968.
           41:267-274, March, 1966.                                        30.          .  "The  Nursing  Audit  for  Evaluation  of   Patient
      25.  -.   "Simulation   Technique   in    the    Measurement    of        Care." Nursing Outlook, 14:5 1-54, June, 1966.
           Problem-Solving Skills.. Journal  of  Educational  Aleasure.-   31.  Rinioldi, H.J.A. "The Test of Diagnostic Skills." Journal  oj'
           metit, 4: 1 -1 0, Spring, 196 7.                                     Medical Education, 36:73-79, January, 196 1.
      26,  Morehead, Nlfldred A. "fhe @Nledical Audit as an  Operational   32.  Vigliano,  Aldo,  and  Gaitonde,   Mangesh.   "Evaluation   of
           Tool."  American  Journal  of  hiblic  Health,  57:1643-1656,        Student Performance in a Clinical Psychiatry Clerkship."
           September, 1967.                                                     Journal of.Ifedical  Education.  40:205-213,  February,  1965.
      27.  Payne, Beverly  C.  "Continued  Evaluation  of  a  System  of
           Medical Care Appraisal." Journal of the  American  lvledica,l   33.  Williamson, John W.  "Assessing  Clinical  Judgment."  Journal
           Association, 201:126-130, August 14, 1967.                           of medical Education, 40:180-187, February, 1965.
      28.  Peterson, Osler, Andrews, Leon P., Spain,  L.P.,  and  Green-   34.  Wilson,  G.M.  et.al.  "Examination  of  Clinical  Examiners."
           berg, R.S. "An Analytical Study of North    Carolina  General        Lancet, 1:37-40, January 4, 1969.



























                                                                                                                                         59





                       WORKSHOP ON THE EVAI,UATION OF CHANGING [IEALTH STATUS

                                                                          Participants
            Robert R. Carpenter, M.D, - i%loderator                                  Howard R. Keltiiaii, Ph.D.
            Director, Western Pennsylvania Regional                                  Department of Preventive Medicine
            Medical Program                                                          and Public Health
                                                                                     New York Medical College

            Sam Shapiro
            Director of Research and Statistics
            Health Insurance Plan of Greater New York
            Nlaureen M. Henderson, NI.D.                                             Charles E. Lewis, iNI.D.
            professor, Preventive lvledicine                                         Prof@sor and Head
            Department of Preventive Medi@e and                                      Health Administration Division
            Rehabilitation                                                           School of Public Health
                                                                                     Uni@-ersit%l of California
            University of Maryland
            Transcript of Workshop-Remarks by Nloderator                             seldom    that    any    individual's    results    are    sufficiently
                                                                                     better   than   those   of   his   colleagues   so   that   he    would
                    ROBERT R. CARPENTER, M.D.                                        desire   such   comparison.   Perhaps   the   results   as   a    whole
            DR.     CARPENTER:     Thanks     to      Mitch      Schorow,      I     would   not   be   good   enough   to   impress   the    public    very
                                                                                     favorably.  (2)  An  effort  to  thus  analyze  is   difficult,   time-
            found  an  interesting   book,   published   in   Boston   in   1917
            by   E.   A.   Codrnan.   It's   called   "A   STUDY   IN   HOSPITAL     consuming,    troublesome    and    would     lead,     by     pointing
            EFFICIENCY,      A       DEMONSTRATION       BY       THE       CASE     out   lines   for    improvement,    to    such    onerous    committee
            REPORT     METHOD     OF     THE     FIRST     FIVE     YEARS     IN     work by members of the staff."
            A PRIVATE HOSPITAL."                                                      "Neither  trustees  of  the   hospital   nor   the   public   are   as
            It  says  by  way  of  foreward   that   this   hospital   has   for     yet willing to pay for this effort."
                                                                                      "Althouc,h   the   staff   would    admit    that    such    follow-up
            sale  a  product  of  a  standard  which  is  to  be  described   on                0
            pages 12 through  63,  It  aims  to  be  a  S  I  00  hospital  with     analysis  was  a  good  thing  for  all,   yet   each   practical   man
            a $100 surgeon.                                                          -  and  the   practical   men   always   hold   the   power   -   would
            The   volume    is    dedicated    to    Richard    Cabot    because     wait for somebody else to do the work."
            Dr.   Codman    respected    his    motives    and    admired    his       And   he   goes   on   to   point   out   that   the   superintendent
            courage   and   energy   though   he   heartily    disapproved    of     would  be  the  last  one   to   undertake   this   task   because   he
            some  of  his  opinions  and  methods.  "He   seems   to   want   to     surel@, would lose his job.
            reform  the   profession   from   the   bottom   whereas   I   think       I  enjoyed,  that  1917   description   of   what   we   are   trying
            the blame belongs at the top," says Dr. Codman.                          to  do  in  Western   Pennsylvania   in   1970   and   I   don't   know
            The   case   report   is   subtitled   "A   Practical   Illustration     that  we  have  come  terribly  far   in   our   ability   to   measure
            of  the  Fact  that  It's   Possible   to   Use   the   End   Result     health  status  and  particularly   any   change   in   health   status
            System in a Hospital."                                                   attributable   to   any   of   our   efforts   to   improve   what   we
            And  the  first  page  I  think   suggests   how   little   progress     are doina.'
            we  have  made   since   1917:   "The   trustees   of   our   chari-       Yesterday   we   heard    the    public    and    the    Bureau    of
            table  hospital  do  not  consider  it  their  duty  to   see   that     Budcet  --  good  morning,  Dr.   Fox   -   ask   for   health   status
            good  results  are  obtained   in   the   treatment   of   patients.     outcome    measurements.    I    think    RMPS    asked     for     end
            They  see  to  it  that  their  financial   accounts   are   audited     results   but   not   really   health   status   end   results.    They
            and  they   take   no   inventory   of   the   product   for   which     were   asking   for   lower   cost   and   better    distribution    of
            their money is expended."                                                care    which    is    significantly     different     than     outcome
             "It  is  against  the   individual   interests   of   the   medical     analysis going to end results.
            and  surgical  staffs  of   hospitals   to   follow   up,   compare,        I   was   interested   in   this   workshop   above   all   of   the
            analyze  and  standardize  all  their  results  because  (1)  it  is     others.  Since  I  am   interested   in   the   Regional   Program   as
                                                                                                                                                           61





                   a  way  to  improve  health,   I   want   to   know   how   to       The  acceptance  of  the  desirability  to  detern-dne  the
                   measure health and its improvement.                               effect of health programs on  the  well  being  of  a  popula,
                     I  think  we  have  an  unusually  talented  group  with  us    tion is quite general not only  among  researchers,  but  alsc
                   this morning to  help  us  do  this.  The  speakers  who  will    among    planners,    administrators,    and    among    those
                   enter  into  discussions  with  you  off  and  on  during  the    responsible for allocation of resources.
                   morning  have  spent  a  goo d  many  years   measuring   health     This  acceptance  moves  from  a  state  of  passivity   to
                   status; I look forward to learning a  great  deal  from  them.    worried  preoccupation  when   change   is   contemplated   or
                     They  are   Dr.   Henderson,   Mr.   Shapiro,   Dr.   Kelman    alternatives  weighed  in   circumstances   ranging   from   a
                   and Dr. Lewis.                                                    highly  specific  component  of  health  care  to  the   broad
                     I  want  to  show  you  just  four  numbers  as  an  example    design of  organization  and  financing  of  health  services.
                   of the problem  and  promise  of  end  result  evaluation.  We       It usually slips back to  an  uneasy  but  quiescent  state
                   looked  at  the  hospitals   that   serve   a   community   of    when  the  complexities  of  end  result  measurement,  costs,
                   200,000  and  at  the   mortality   from   stroke   in   those    and time requirements become apparent.
                   hospitals. We  were  surprised  to  find  that  patients  with       This is not in the nature  of  a  sharp  criticism  of  the
                   heavy  paresis  when  they  were  cared  for  by   generalists    past. The difficulties of assessing the impact  of  particular
                   died  more  frequently  whether  they  were  male  or   female    actions on health status were and still  are  great.  Further,
                   than  did  patients  with   the   same   reported   neurologic    the  introduction  of  changes  affecting  the   availability,
                   signs if they were cared for by internists.                       delivery  and  economics  of  health  care  often  could   not
                     I  hoped  from  this  la-t  we  could  attract  the  medical    and will not  in  the  future  wait  for  hard  evidence  from
                   staffs' interest to  more  careful  care  of  stroke,  attract    studies of impact.
                   interest in helping to understand these results.                     Similarly   for   the   introduction   of   some   programs
                    We  identified  some  cases  that  the  medical  staffs  were    aimed   at   modification    of    primary    and    secondary
                   particularly  interested  in  reviewing:   The   patient   who    prevention of specific diseases.
                   died with a diagnosis of  cerebral  vascular  disease  without       However,  many  of  the  problems  and   issues   we   face
                   any neurologic sicns, for instance.                               are stubborn, and courses of action are not  at  all  certain.
                    1 hope  as  the  morning  goes  on  that  we  can  learn  the    Because  selection   of   an   available   alternative   often
                   value  of  such  measures  of  outcome  as  others  who   have    involves   commitment   of    scarce    manpower,    equipment
                   made them  more  frequently  have  seen  this  value.  I  hope    and  financial  resources  for  which  there  is  sharp   com-
                   that we can  find  out  when  it's  worthwhile  to  make  such    petition, implementation faces serious obstacles.
                   measures  and  how  to  make  them.  I  hope  we   can   learn       As  we  all  know,  these  are  the   considerations   that
                   how  to  interpret  them  once  we  have  them.  I  hope  that    force  i-nany  of  us  to  think  in  terms  of  demonstration
                   we can learn, particularly  from  Dr.  Kelman,  the  key  data    projects or  R  &  D  projects  in  which  operational  @ffec-
                   bits  that  help  us  to  measure  and  talk  about   outcome.    tiveness related to costs  and  manpower  is  a  central  con-
                   And,  finally,  I  hope  we  can   learn   how   outcome   and    cern.
                   process analysis interrelate.                                        Now, this  is  fine,  but  often  the  question  that  will
                    Mr.  Sam  Shapiro  will  begin  the   discussion.   He   will    remain even after a project has been well execut      ed    is
                   describe some of his studies  and  discuss  why  and  when  it    whether any health benefits have resulted.
                   is worth measuring health status.                                    Bypassing  the  issue  can  compromise  the  potential  for
                                                                                     i-noving  from  demonstration  to   general   acceptance.   In
                           The N'alue of Health Status Nleasures                     fact, where the effort required for  the'extension  is  great,
                                                                                     absence of evidence  of  impact  on  health  status  may  well
                                      SAM SHAPIRO                                    prevent such extension.
                                                                                        Conversely,  availability  of  evidence  of   a   program's
                    I'm  not  sure  that  I'm  going  to  be  dealing  with   the    health  benefit  can  stimulate  widespread  consideration  of
                   questions  precisely  the  way   you   have   outlined   them.    early implementation.
                   What I  thought  might  be  useful  is  for  me  to  give  you       I  want  to  emphasize  that  many  prograi-ns  cannot   be
                   some  general  considerations  that   underlie   the   concern    nor need be tested for  health  benefits  although  there  are
                   with  measurement  of  health  status  changes  and  then  use    programs  under  active  consideration  today  that  will   be
                   a  few  samples  principally  from  my   own   experience   to    plagued  by  doubts  and  challenges  until   the   issue   of
                   illustrate what's  really  at  stake  when  you  get  involved    health benefits is dealt with  effectively.  Just  to  mention
                   with health status measures.                                      a  few:  early  disease  detection  through  automated  multi-

                   62





                   breast    and    mammography.    The    measure    is    change    in        All   screening   exai-ninations   have   been   completed.   An
                   mortality from breast cancer.                                                at every   stage   of   the   investigation   when   findings   wer
                       This  is  the  situation.  It's   generally   acknowledged   that        reviewed   it   was   clear   that   mammography   and   clinic,
                   screening  will  lead  to  earlier  diagnosis   of   breast   cancer,        examinations   contributed   independently    to    the    dete(
                   but  there  has  been  no  evidence  that  this  results   in   lower        tion   of   breast   cancer.    If    mammography    had    beei
                   mortality.   Costs   for   including    breast    examination    with        excluded,  31  percent  of   the   cancers   would   have   beei
                   mammography,  in   particular,   are   high.   And,   in   fact,   in        missed  during  screening.  If  the  clinical  examination   ha(
                   automated    multiphasic    health     testing     programs     where        been omitted, 44 percent would have been missed.
                   this  procedure   is   used,   mammography   is   the   most   costly        Further,   screening   did   lead   to   detection   of    large
                   single test.                                                                 proportions of  breast  cancer  with  no  evidence  of  axillar3
                      In  short,  a  national  effort  to  screen   women   for   breast        nodal  involvement  -  70  percent  -   than   among   the   con
                   cancer   would   require   massive   expenditures    and    diversion        trol group - 45 percent.
                   of    equipment    and    manpower    from    other    health    care        Preliminary  results  on  mortality   are   now   beginning   t(
                   activities.                                                                  be  collected  and  will  shortly  appear  in  an   article   ir
                      Clearly,   to   acquire   a   high   priority,    breast    cancer        JAMA.  The  findings   are   highly   encouraging.   There   arc
                   screening  should  justify   its   value   in   the   most   rigorous        52  deaths  due  to  breast  cancer  in  the  control  group  a@
                   manner    possible.    And    as    many    of    you     know,     a        compared with  31  breast  cancer  deaths  in  the  total  study
                   randomized  clinical  trial  directed  to   this   issue   has   been        group in the period available for follow-up.
                   underway  since   1963   in   HIP   under   a   contract   with   the        The  case  fatality  rates   for   cases   with   histologically
                   National Cancer Institute.                                                   confirmed  breast  cancers  reinforce   the   impression   that,
                      The   main   objective   is   to    establish    whether    breast        in  the  short  run  at  least,  screening  leads   to   lowered
                   cancer     screening     using     mammography      and      clinical        mortality.
                   examinations   results   in   a   reduction    in    breast    cancer        These  observations  are   preliminary,   and   more   time   is
                   mortality.  Other  objectives   relate   to   the   epidemiology   of        needed  to  establish  whether  the  effect  of  the   screening
                   breast  cancer  and   the   search   for   high-risk   factors   that        program is short-term or long-term.
                   might be useful in future screening programs.                                However,    the    findings    do    provide     grounds     for
                     I  don't  want   to   go   into   the   details   of   methodology.        cautious  optimism  and  it  would   appear   prudent   to   ac-
                   These  have  appeared  elsewhere.  But   a   few   key   points   are        celerate  efforts  to  develop  and  test  methods  capable   of
                   important for me to touch on in this discussion.                             dealing  with  the  broad  demand  for   periodic   breast   ex-
                     Thirty-one   thousand   women    aged    40    to    64    enrolled        aminations that might emerge within a few years.
                   in   HIP   have   been   assigned   randomly   to   a   study   group        What   I'm   describing   is    a    progression    from    very
                   and  a   similar   random   sample   to   a   control   group.   Only        intensive study involving  huge  resources,  a  long  period  of
                   study   group    women    have    been    invited    for    screening        time,  and  dedication  of  large  numbers   of   personnel   to
                   examinations.   About   65   percent   appeared   for   the   initial        achieve  a  result  which   if   sustained   can   significantly
                   screenings.                                                                  affect the  approach  that  medical  care  might  be  taking  to
                     Three    additional    screening     examinations     at     annual        the whole issue of screening for breast cancer.
                   intervals   were   scheduled,   and   large   proportions   of    the        Those  of  you  who  have  been  close  to   this   field   over
                                                                                                the   years   know   how   much   disappointi-nent   there   has
                   women   with   an   initial    examination    have    returned    for
                   these.                                                                       been  in  dealing  with  the  problem  of  breast  cancer,   and
                                                                                                how  widespread  is  the  pessii-nism  about  the  effectiveness
                     Control    group    women     continue     to     receive     their        of breast cancer screening.
                   regular medical care.                                                        There is a great deal at stake  in  this  study,  and  as  I  see
                    Screening   examinations   have   been   performed    at    23    of        it,  these  preliminary  results  are  placing   high   on   the
                   the  HIP  medical   group   centers.   The   clinician   and   radio-        agenda  a  new  set  of  concerns,   mainly   related   to   the
                   logist   record   their    examination    findings    and    recoiii-        question,  "what  kind   of   screening   prol@prrn   would   be
                   mendations    independently.     Later     their     findings     are        required to reach effectively  @ll.(Te
                   reviewed   jointly   by   a   physician   for    final    recommenda-        the present findings persist?"
                   tions.   Intensive   follow-up    to    identify    breast    cancers        I  want  to  turn  now  to  a  much   broader   type   ot'effort
                   diagnosed  and  mortality   is   carried   out   with   equal   rigor        in the field  of  preventive  care,  that  of  automated  multi-
                   for   women   screened,   women   who    refused    screening,    and        phasic   health   testing.   There   are   many   justifications
                   control group women.                                                         advanced  for  introducing   AMHT,   and   I   don't   want   to

                   64




                                                                             covering a broad a(,e r,,itige-12 years and  older-ftoni  an
       make  out  a  case  for  or  against  such  programs,  However,                     0
       everyone  will  agree  that  AMHT  is  a  costly  addition   to       -,absolute standpoint and relative to  a  noiipoverty  group
       the spectrum of  health  services  and  most  will  agree  that       that will also have AMI-IT.                hese       health
       it  is  il-nportant  to  seek  out  opportunities   to   assess       Action  to modify  adverse aspects of t
       AMHT's  effect  on  health  status   and   on   behavior   that       components  aiiiono  the  poor  is  to  be  instituted,  and
       might be  expected  to  have  a  desirable  effect  oil  health       evaluation is in terms of change asIcompared     with     what
       and well being.                                                       occurs in the iioiipoverty group.
       Two  projects  are  now  directed  to  this  issue.   The   one       An   underlying   question   is    whether    through    the
       of  longer  duration  is  being  conducted  by  the  Permanente       AMFIT  program,  and  activities  generated   by   it,   the
       Medical  Group  in  California.  One  phase  of  that   program       anticipated  gaps  between  the  two  groups  can  be   nar-
       is  very  well  known,  probably   much   better   known   than       rowed.
       the second  phase  which  deals  with  the  end  result  issue.       A  broad  spectrum  of  measures  are  being  developed   to
       This  is  to   demonstrate   how   automation   and   computers       measure liealtli  impact.  These  include  chan-es  in  itn-
       can  be  applied  to  improve  -  and   I'm   now   quoting   -       pairnient of  function,  immunization  status  of  children,
       "speed,  efficiency,  and  quality   control   in   multiphasic       and complications of disease.
       screening  techniques  so  that  not  only  more   tests,   but       The last project I want  to  describe  is  in  the  proposal
       more   accurate   and   quantitative   measurements   can    be       stage and is now being reviewed  for  possible  funding.  It
       performed,  and  at  a  lower   cost."   All   very   important       concerns sudden death from coronary heart disease.
       operational objectives.                                               There   is   cencral   agreement   that   until    effective
       The  other  phase  of  the  program  includes  a  set  of   end       primary  prevention  methods  can   .be    identified     and
       result  criteria  in  the  evaluation.  Two  randomly  selected       implemented,  significant  progress  in  reducing  the   in-
       samples   of   the   plan's   member   have   been   designated       cidence of this cause of death will  depend  on  changes  in
       study  and  control  groups.  Efforts  are  made  to  have  the       community  practice  which  bring   advances   in   coronary
       study  group  appear   for   the   examination.   The   control       care to patients who under present  circui-nstances  do  not
       group  is  not   approached,   but   those   who   request   an       survive to reach a hospital.
       examination are accommodated.                                         It is estii@ated that about 60  percent  of  deaths  due  to
        Morbidity,  disability,  and  medical   utilization   patterns       acute myocardial  infarction  occur  outside  the  hospital,
       are  to  be  determined  over  a  long  period   of   follow-up       and a great effort is being made  to  cope  with  the  prob-
       through periodic questionnaires and medical records.                  lem of rapid response to  requests  for  medical  care  when
        This  is  an   ambitious   undertaking.   But   it   has   the       a heart attack is suspected. Also, increasing, attention  is
       potential  of  providing  decisive  information  on  the  value       being  given  to  finding  out  how   patients   and   their
       of periodic  health  examinations  generally  and  of  selected       families  behave  when  faced   with   prodromal   symptoms.
       components of it particularly.                                        The  proposed  project  is  designed  to  incorporate  these
        Anyone   who    questions    the    time    requirement    for       approaches in a  comprehensive  action  program.  It  repre-
       reaching an answer really  has  to  look  very  hard  at  other       sents a combined  effort  of  HIP  and  two  of  its  Queens
       issues  that  have  come  up  in  the  past  which  have   been       medical groups with  a  population  of  about  50,000,  aged
       plagued by  doubts  and  questions  long  after  the  point  in       35  to  74,  and  HIP's  LaGuardia  Hospital  which   serves
       time when it would  have  been  possible  to  initiate  an  end       both medical groups.
       result investigation.                                                 The goal is  to  effect  more  rapid  requests  for  medical
         One  of  the  outstanding  examples  is  the  Pap  smear.  It       care after the onset of a heart attack  or  suspected  heart
       is no longer possible to carry out  a  control  study  in  this       attack and to institute a system capable at all times  of  a
       country  on   Pap   smear   as   an   effective   measure   for       rapid  and  appropriate  response   which   fully   utilizes
       reducing  mortality  from  cervical  cancer.  There  are   very       current medical knowledge.
       few  people  on  the  firing  line   who   really   raise   any       The  end  result  sought  is  a  reduction  in  the  present
       questions  about  Pap  smear.  But   if   you   look   at   the       high rate of sudden  dealth  from  coronary  heart  disease.
       scientific  literature,  there  are  some  very  serious  ques-       Basic changes  to  be  made  in  the  health  services  sys-
       tions being raised about the Pap smear.                               tem consist of the following main elements:
         The  second  end   result   study   in   multiphasic   health
       testing recently started at  HIP.  This  project  is  utilizing          Patient   education.   Varied   educational    approaches
       repetitive  health  testing  to  define  the   health   status,          will be made  to  the  entire  adult  population  of  the
       practices  and  attitudes  of  a  defined  poverty   population          two   participating   medical    groups    and.,    their
                                                                                                                                        65





                      physicians with  the  aim  of  reducing  delays  geii-       Each of the studies I have described  contains  an  er,
                      erated  by  patients  or  their  families  in  seeking    result criterion. In the breast cancer study  we  probab'.
                      medical care for  possible  acute  coronary  episodes.    have the hardest type of evidence. It's a  single  measure
                      A special target will  be  individuals  at  relatively    mortality. A randomized clinical trial  approach  has  bec
                      high risk for sudden  death  (those  with  prior  CHD,    used, and it takes a very unusual set of  circumstances  i
                      hypertension,  hypercholesterolemia,  etc.,  as  iden-    make this type of approach a practical one.
                      tified through  the  HIP  centralized  medical  record       The other investigations shade off in hardness,  but  a
                      system).                                                  effort is made  to  maximize  the  opportunity  within  tf
                   o  Centralized   telephone   screening    at    LaGuardia    medical care setting where the programs are to  be  carric
                      Hospital by physicians  of  calls  from  all  possible    out, to reach sufficiently hard conclusions  about  the  e
                      coronary suspects in the population  will  take  place    fectiveness  of  the   programs,   from   the   standpoint
                      24 hours  a  day,  7  days  a  week.  The  aim  is  to    health status measures, to serve  as  a  basis  for  futui
                      reduce   communication   delays   in   bringing    the    action.
                      patient's  symptoms  to  trained  medical   attention.       One  question  that  often  comes  up  is  whether  all
                   o  Operation  of  a  special   pre-coronary   care   area    most  demonstration  programs   should   attempt   to   ii
                      (PCA)  at  LaGuardia  Hospital  for   observation   of    corporate an end result  criterion?  I  don't  believe  s(
                      patients  in  defined   categories,   one   of   which    Costs are high.  Technical  requirements  are  g-reat.  Ar
                      consists of persons who  do  not  meet  usual  current    frequently the kinds of issues that  are  being  faced  ai
                      criteria for hospitalization, but who  may  be  in  an    not susceptible to the inclusion of an end  result  measur
                      early stage of  an  acute  MI  not  yet  recognizable.       But in the field of medical care, with all  the  change
                      The other  consists  of  patients  who  might  be  ex-    that people hope will take  place  over  the  next  decade
                      periencing  an  ischemic  episode  not   destined   to    maybe  they're  being  optimistic  about  the  next  decad
                      lead to  MI  but  capable  of  inducing  a  fatal  ar-    but let's say the next generation - there  are  very  larl
                      rhythmia.                                                 issues with very large stakes  associated  with  them.  NI
                For purposes of this meeting I think what is  of  particular    point is that it is essential to seek  out  those  few  si
                importance  is  that  two  types  of  evaluation  have  been    uations where such issues can be  investigated  effective'
                planned for. The first is directed at those aspects  of  the    utilizing end  result  measures  and  thereby  provide  tt
                project that bear on generalizing  experience  for  possible    basis  for  making  judgments   that   have   regional   i
                use  by  other  organized  providers  of  medical  care   in    national implications.
                Queens   and   the   New   York   area.   Information   will       DR.  CARPENTER:   Thank   you   very   much.   I   gath
                become available  regarding  the  operational  effectiveness    that hard work makes a man  cautious.  It  seems  that  yc
                of  the  educational   program,   communication   procedures    are  an  enthusiast  for  health  status  measurement  und
                for rapid response to  patient's  call,  training  of  para-    proper conditions.
                medical  personnel,  and   the   operation   of   the   pre-       I heard you saying that the detailed effort required  -
                coronary care area.                                             carry out a study  significant  enough  to  be  generaliz(
                   This   information   would   be   related   to   manpower    from one hospital to another  is  very  great  indeed.  Y(
                requirements and costs.                                         alluded  to  an  indirect  method  of  end  result  analy!
                                                                                which  sounded  as  though  it  might  be  more  often   a
                   By itself, this would  represent  an  important   advance    plicable  to  the  problems  faced   by   Regional   Medic
                in knowledge concerning  the  modification  of  health  care    Program  evaluators.  In  just  a  moment  we  will   have
                iystems to  reach  a  patient  early  when  a  heart  attack    chance to  discuss  these  and  some  other  issueswitliyo
                occurs. However, we  would  still  be  left  with  the  uii-       Before we go on, it's worth  noting  that  the  man  wi
                answered question as to whether  the  effort  involved  does    is evaluating the evaluation  conference  is  with  us-GI(
                have payoff in reducing mortality.                              Hastings  from  Nassau-Suffolk.  Welcome,  Glen;  we'll  I
                   A second type  of  evaluation  has  been  included  which    very careful what we say from here on.
                is aimed at answering this  question.  The  approach  is  to       Before we discuss Mr. Shapiro's  paper,  Dr.  Heiiders(
                compare  the  rate  of   sudden   coronary   heart   disease    will speak to closely related issues. Maureen  will  discu
                deaths  in  the  demonstration  groups  with  the  rate   in    some of the problems of end-results  anslysis,  particular
                other HIP  medical  groups,  and  also  provision  has  been    as she  experienced  these  within  the  framework  of  t]
                made to  compare  the  mortality  situation  in  the  demon-    Maryland  Regional  Medical  Program  where   she   is   A
                stration groups before and after start of the program.          sociate  Director  of  the   Epidemiology   and   Statisti

                66




        Center.   She   will   also   discuss    the    relationship    between       Let  me   briefly   describe   the   collection   of   the   informa-
        evaluation and medical care research.                                         tion  I  ai-n  going  to  show.  We   took   a   random   sample   of
                                                                                      admissions   to   every   short-term   general   hospital   in    the
                                                                                      region during a I -month   period   just    before    the    regional
                MAUREEN M. HENDERSON, M.D.                                            medical   program   belan.    The.    data,    therefore,    describe
                                                                                      patients    and    procedures    in    every     "acute"     hospital
        I  propose   to   review   a   verv   different   level   of   research       whether  or  not  it  prepares  its  own  statistics  or  has  easily
        from   Sam   Shapiro.   One   major   value   of   this   workshop   is       accessible  records.   In   most   other   morbidity   surveys,   in-
        the  way  it   illustrates   the   need   for   many   different   dis-       formation  is   collected   only   froi-n   hospitals   with   viable
        ciplines   and    approaches    in    the    evaluation    of    health       (for research) record systems.
        services.  Mine  is  a  very  limited   approach   within   the   total       The   Maryland   Re-ion   includes    all    of    Maryland    except
        context  of  health   services   research   and   evaluation.   As   an       Mont(yolnery    County    and    includes    York    County,    Penn-
        epidemiologist,   I    am    most    interested    and    only    corn-       sylvania.
        petent   to   deal    with    biological    measurements.    The    end       The   specific   medical   records    reviewed    in    our    sample
        results  I  have  been  looking  at  in  relation   to   the   Maryland       were   identified   by   our   own   staff    and    abstracted    by
        Program    have     therefore     been     measurements     of     mor-       trained    medical    abstractors     tinder     constant     quality
        bidity.                                                                       control   and   surveillance.    Standardized    abstracting    forms
         I  believe  it  is  important  to  talk   about   ways   of   i-naking       and procedures were used.
        these      particular      measurements      because       non-epidemi-       The     measurement     data     collected     were      specifically
        ologists   are   not   always   aware   of   the   series    of    con-       selected to:
        foundino   issues   and   problems   related    to    their    observa-
                                                                                      1. get estimates of need;
        tions.                                                                        2.  look  at  the  secular  effects  of   the   total   program   and
          I   trust   those   present   who   are   sophisticated   in   epide-           of individual pro-rams
        miologic  techniques     will     bear     with     the     fundamental                               LI
        levels I am going to discuss.                                                 3.  insure   proper   coi-nparisons    in    assessing    needs    or
                                                                                          effects.
          Let    us    first    consider    biologic    outcome    measurements
        in  relation  to  overall  evaluation   of   regional   services.   The       The  last   purpose   is   one   Sam   Shapiro   spoke   about   very
        two   types   of   measurements    consistently    used    are    those       briefly  and  one  on  which  I  would   like   to   enlarge.   When-
        of  death  and  morbidity.  We   have   made   very   little   use   of       ever  you  examine  an  effect  or  an  end   result   in   different
        death records.                                                                time   periods   or   between   different   groups   of   poeple   or
          In  looking  at  the  picture  of   our   total   region,   we   have       different  geographic  areas,  you  must  be  sure   that   you   are
        been  using  case  fatality  rates.   The   latter   are   of   limited       comparing  like  with   like.   The   original   numbers   that   Bob
        use   now   for   two   reasons:   (1)   there    have    been    great       Carpenter   presented   draw   attention   to   this    point,    and
        changes    in     denon-Linators-the     census     population     from       because  he   mentioned   that   he   was   going   to   show   those
        1960-1970    and  (2)  death  rates   have   been   at   a   standstill       particular  stroke  data,  I  brought  some   of   our   own   stroke
        for the past  few years.                                                      data to illustrate and ai-nphfy this point.
           In  terms  of  disease  or  morbidity  we   are   looking   at   the        This   slide   describes   short-term    general    hospital    dis-
        prevalence   and   severity-that   is,   the   distribution   of    the       charges   in   the   region   of   Maryland   before   the   Regional
        frequency  and  severity   of   disease   as   we   see   it   in   the       Medical   Program   began.   It   shows    annual    case    fatality
        region.   We   are   also   looking   at   aggregations   of   disease-       rates  from  four   hospitals.   The   rates   are   estimated   from
        that  is,  multiplicity   of   disease   problems   in   patients   who       our   sample.   They   vary   enormously   from   16    percent    to
        are found at points on each disease spectrum.                                 60  percent   between   the   four   hospitals.   Just   looking   at
           One   good   example   is    the    presence    of    cardiovascular       the   total   numbers,   you   might   infer   that   the   hospitals
        diseases   and   diabetes   mellitus   in    stroke    patients    with       seeing  the  most  stroke   patients   give   the   best   care   and
        rnild or serious neurological deficits.                                       have  the  lowest  case  fatality  rate.  But   in   Maryland   there
           The   greatest   amount   of   the    data    we    are    currently       is  a  great  difference  in  the   patients   admitted   into   dif-
        studying   comes   from   hospital   in-patient   records.   I    think       ferent   hospitals.    The    easiest    and    quickest    way    to
        it   is   appropriate   to   speak   mostly    about    hospital    in-       describe patient differences is to look at  tire  -r  -acia-l-  dis--
        patient   records   today    because    I    am    sure    that    most       tribution.   The   next    slide    shows    how    proportions    of
        ,regional  programs  use  these  as  their   major   source   of   mor-       black  and  white  patients  differ  from   one   hospital   to   the
        ".@bidity data.                                                               next.

                                                                                                                                                          67








                                                                                                 of               analvses
                                  The next slide shows one of a whole series                                             Pdtiellts had  at  least  one heart  Iiseas.,
                             t@@      id@-Iltifv      truly      comparable      groups      of       patients.       With you   are   looking   at   tire   otitcoll,    'of    sti
                             ,:c)niparable     groups     of     patients     we     can     begin     to     look      at one place or from one liosiiital to til   11
                             the     outcot-ne     of      care      in      different      groups      of      hospitals. ignore   the   fact   that   a    lot    of    p@itielits
                                 Ili      this      analysis       we       divided       all       "immediately       ad- disorders    such    as    heart    diseise    @k7ili,:Il II
                             iiiitted"     stroke     patients     accordance     to     the     severitv     of     their lfllood    of     survival     and     recovery.     Once
                                                                                                                                                                                                                              le@s
                             condition     on     admission.     Classes     of     severity      are      in      ranking adjust  for the  presence of' other dismiss.,
                             order      and      are      exclusive.      The      worst      class      included      all say whether outcomes  of different tr@it@il
                                                                                                                                                                                                                    [is I
                             patients      who      were      not       conscious;       the       second       identified are more Or less successful. All these ex@!-@,@
                             those      who      were      conscious       but       had       swallowing       difficulty why     the     first     qL)CStiOll      epidemiologists                                   c i I 11  r 1.@
                             The       third       identified       those       with       speech       problems       who look at anv evaluation                                                                                c
                                                                                                                                                         is: Are the pitietit                                       ,jagc        a'L
                             were      conscious      and      could       swallow.       Tire       fourth       catcoory The  second  question  is:  flave  tile  1)1)\
                             includes     those     with     none     of     the     three      more      severe      con- equal  pains  to  make  the  cli:I,L,llosjs,.,  Ili                                                   t
                             ditions.     Looking     separately     at      the      data      for      white      males, coiiipariii-    the    same    diagnosis    with     ti)C                               -.Iicr  tl]211)
                             white females, non-white
                                                                 males,          and          non-White           females, in   this    example    you    see    tire    frcqLICIIC\
                             you will see  that 20 per            cent      of      the       white       males       were diagnostic technique, E.K.G., in patients "k                                            @vented     ti@
                             unconscious     when     they     were      admitted-,      fifty      per      cent      of' dia-flosis of heart disease.
                             the       non-white       males       were       unconscious       when       they       were fit    this    slide,    I    want    yot,    to     look                                      Anotlict
                             admitted@    thirty    per     cent     of     the     white     fei-nales     were     Lifl- frequency  with  @vliicil  tile  test  Was  LISCLI                        j.Is       -.)Ilective     di
                             conscious;     and     fift),     per      cent      of      the      non-white      females. patterns  of use  in  these patients. We li@i@!c                         tile                 nIL'il"io
                             If     you     go     to     the     other     extreme     and     look      at      patients regional  hospitals  into  four        roups  accoicii@@         i@:  nice,          .@iscIll.,ir acL:i
                             with     no     severe     conditions,     YOU     WU]     see     50     per     cent     of size.  We ]lave used the annual ilul-flbers ot-                                      The
                                                                                                                                                                                                     "IS
                             the    white    males;     30     per     cent     of     the     white     females;     none our measure of size. These E.K,G. fFc(iuencie,
                             of    the    Negro    males;     and     20     per     cent     of     the     Negro     fe- t'ore. tabulated from the smallest to the big2c,@                                    This   text    sli
                                                                                                                                                                                                                I atietits Ili  li
                             males.     These     data     may,     of     course,     mean      that      blacks      and Remember we are oniv talkiii@
                                                                                                                                                                ,  about
                             whites     have     different     diseases;      that      we      are      dealing      with with a primary diLtaiiosis of heart disease. Ili II              ill@t!!@,,@t        ilie     se@,,rt%'
                             different     age     groups     in     the     two     races     or     that     the     two hospitals, onIN, three quarters of the patients li@id                                diagnoses       of
                             races    choose    to    go    to     a     hospital     when     they     have     different of art E,K.G. examination. There was evideilc,!                                      Three l@c,-      @'ot
                                                                                                                                                                                                                l@s cli@i@-,]-,osj
                             manifestations       of       disease.       Hospital       admission        policy        is cent  in  the  largest.  In  all   except   the   largest              file
                             another        possible         explanation.         Whatever         the         explanation proportions of patients with E.K.G.                                                  iiiitted  Litid  9
                             of      racial      differences,      you      cannot      compare       admission       out- lower in black (mostly service patients) than it) i. IC                              iiiajoi-    degree
                             comes     unless     you     adjust     for      at      least      the      severity      of (a sizeable percentage of private patients).                                           If    oil    tli
                             disease at the time of admission.                                                           III our data, therefore, the decree of certaii)i                                       rhage.       N@oL,
                             One     other     point     I     mentioned,      that      of      au      relations      of heart  d'                                                                            were Llil@Olls
                                                                                      bg                                    isease  diagnosis is correct is  -oiilg  to
                             disease, is also well illustrated in stroke patients.                                       one kind  of hospital  to another and from oi),@ kijit] @)l'
                                                                                                                                                                                                                per  ceiit   %@@ci
                             In      all       Baltimore       surveys       looking       for       conditions       pre- patient to another.                                                                         Epidciiiiol
                             disposing     to     strokes,     we      have      observed      heart      disease      and To repeat:  tire second question in the epideijiioI()L,.ists                         diagnoses      ,is
                             general      vascular      disease      behind      a      majority       of       pre-stroke mind is:  ""'fiat was the  extent of the effort their                                together   ill   t
                             symptoms.     This     association      shows      up      again      it)      our      lios- into   making   a   dia@ilosi's    and    were    efforts                            scribed   by    co
                             Pita]     admission     survey.     The     numbers     vou     see     in     the      slide alike in different hospitals and among different 1)@iticjiL'S                          We        looked
                             are       from       the       reviewed       records       before       total       s@iniple that Outcome measurements can be compai-ed'@"'                                       to see
                             estimates      were      reconstituted.      The      slide      shows      data       froi-n Nex t  wconsider  the  recording  of  tile  Pli@siciLill's                           as having        die
                                                                                                                                                                                                                hemorrlia-e
                             approximately         4,000         stroke         patient         records.         Different diagnosis  @ifid  Helical  information. What do Illc il@@()@
                                                                                                                                                                                                 I              cerebro@'ascul
                             stroke    diagnoses    are    listed    across    the    top    of    tile    table    ,ilici librarians  do  Nvitli  them.  To get son   ic  estimate  (ii, !II,-                          One-third
                             down     the     side      are      listed      other      majoi-      chronic      diseases. Possible V@iiltiol]S WC Should exi)
                                                                                                                                                                ect front this 1)@il
                             The     numbers     and     per     cent     of     stroke      patients      with      these source. we took two or three troublesome                                             hemorrhage       I
                             other diagnoses are shown in the cells of the tabl( .                                       sent them to record librarians ill a majority ol'I-l@)II)i@@!                          cerebrovascul
                             The       heart       disease       category       shows       the        most        obvious and asked tl)ciii to @@ code" them t'oj Lis.                                         tMrd)   of    addi
                             relationship.      For      every      stroke      diagnosis,       a       Iii-ii       pro- This slide shows tire three dta,ail(,),Ses died the                                  diagnoses       we
                             portion      of      discharge      records      have       i       secondary       diagnosis International Classification  Codes ,iveii  to  flesh         i),,                   noses.
                             of     heart      disease.      More      than      50      per      cent      of      stroke of our   reL:ord librarians. The fir                                                  In    the    pati
                                                                                                                                                                        St One is oil"      tilL@,
                                                                                                                                                                                                                of    those    giv
                            68




       I @vill  speak  about  aaain  later-cerebrovasctilar  accident.   number   had actually  had  a   diagnosis   of   cerebrovasculir
                          c
       Twenty-five out of the  29  librarians  coded  it  as  cerebral   accident, In a  sample  of  death   certificates,   again   more
       hemorrhage  which  is   correct   procedure   by   instructions   than a third had   cerebrovascular    accidents.    Once    more
       given  in  the  index  of  the  classification   manual.   Four   the decision of  the  "coder"   to   put   cerebrovascular   ac-
       librarians  did  not  use  this  code  number.  The  librarians   cidents with cerebral  hemorrhages   and   the   proportion   of
       were less consistent for  the  second  dia-nosis  of  transient   each in  the total  group  of  patients  can  niale  a  lot  ot'
       ischemic  attacks.  The  third  diagnosis  of   chronic   bron-   difference   to   end-point   measurements.   In   our   stroke
       chitis  really  gave  inconsistent  coding  results.  We   were   registry,  we  code  all  diagnoses  separately  so  our   end-
       particularly   worried   about   this   disease   because   the   results for cerebral hei-norrhage  will  probably  differ  from
       average  age   of   admissions   with   a   code   number   for   a majority of others.
       chronic   bronchitis   was   between   30   and    40    years.    I  would  now  like  to  talk  about  a  different  kind   of'
       Obviously  the  group  includes   patients   with   more   than   bias; one I mentioned earlier and one  that  I  did  not  fully
       (other than) chronic bronchitis. In  this  case  we  felt  that   appreciate before  we  started  this  survey.  Our  usual  mor-
       the difficulties  inherent  in  coding  chronic  lung  diseases   bidity data come  from  the  records  of  our  best  hospitals.
       prevented  us  from  learning  about  true   distributions   of   By  "best"  I  mean  the  biggest   hospitals   with   adequate
       that disease from our samples of morbidity data.                  record  keepina  facilities  and  the  most  accessible   diag-
        Another  classification  problem   arises   in   relation   to                 0                                 from      which
                                                                         nostic indices. These are the only hospitals
       collective diagnoses and group outcomes.                          investigators  and  planners  can  easily  get  the   kind   of
        I   mentioned   above   that   the   diagnosis   of   cerebro-   listings  of   record   numbers   and   diagnoses   needed   to
       vascular  accident  had  concerned  us  in   another   context.   collect  morbidity  data.  The  following   slides   illustrate
       The  International  Statistical  Manual  suggests  that   cere-   why this is so. This slide  shows  the  status  of  our  record
       brovascular  accident  be  coded  with   cerebral   hemorrhage.
                                                                C,       rooms  in  Maryland  at  the   beginning   of   1969.   Twenty-
       This next  slide  shows  specific  diagnoses  given  to  stroke   one hospitals  (half)  could  produce-a  computerized  list  of
       patients in hospital  records.  They  are  cross-classified  by   their  admissions  and  used  the   International   Classifica-
       the severty  of  stroke  on  admission.  If  you  look  at  the   tion. Thirteen had  a  card  file  and  used  the  I.S.C.D.  We
       diagnoses of  cerebrovascular  accidents,  you  will  see  that   went  through  these  hospitals  card   files   by   hand   and
       three per cent of  patients  admitted  to  our  hospitals  with   made  lists  from  which  we  could  prepare   samples.   Seven
       this  diagnosis   were   unconscious   when   they   were   ad-   other  hospitals  had   a   card   file   and   used   standard
       @tted and  92  per  cent  were  free  from  any  of  our  three   nomenclature.  For  these  hospitals  we  had  to   develop   a
       major degrees of severity.                                        code  compatible  with   our   selected   I.S.C.D.   categories
         If  you  then  look  at  the  diagnosis  of  cerebral  hemor-   and  we  had  to  go  through  the  card  file   by   hand   to
       rhage, yot@ will  see  that  50  per  cent  of  these  patients   identify  all  compatible   diagnoses   in   the   given   time
       were  unconscious  when  they  were  admitted   and   only   18   period. At the time  we  did  the  survey  two  hospitals  were
       per cent were without all  three  severe  degrees  of  disease.   without  a  filing  system.,  We  sampled  from  all  of  their
         Epidemiologically,   these    are    two    very    different   records  for  one  year  and  read  large  numbers  of  records
       diagnoses  as  physicians  give  them  but  they   are   lumped   to get our balanced  sample  of  patients  with  stroke,  heart
       together  in  the  descriptions  of  groups  of  patients   de-   disease, cancer, diabetes  mellitus,  and  chronic  bronchitis.
       scribed by code numbers in hospital statistics.                   The crux of the matter is that  the  likelihood  of  getting  a
         We  looked  at  our  total  morbidity  and   mortality   data   list  of  patient   discharge   diagnoses   varies   enormously
       to  see  what  proportion  of  the  people  we  were   counting   from the larger  to  the  smaller  hospitals.  An  even  harder
       as having died or been admitted  as  a  result  of  a  cerebral   problem to deal  with,  and  one  that  limits  available  data
       hemorrhage  had,  in  truth,  been  given  the   diagnosis   of
                                                                         more  than  the  actual  mechanization  of  the  index  system,
       cerebrovascular accident.                                         are hospitals that fail  to  identify  which  listed  or  coded
         One-third  of   discharge   diagnoses   coded   as   cerebral   diagnosis was the reason for admission.
       hemorrhage  had,  in  fact,  been  a   primary   diagnosis   of
       cerebrovascular   accidents.   The   same   proportion    (one-   They simply  write  every  listed  diagnosis  into  their  card
       third)  of   additional   non-primary   "cerebral   hemorrhage"   file  with  no  indication  which  one  the  physician   listed
       diagnoses  were   actually   cerebrovascular   accident   diag-   iirst.
       noses.                                                                The next slide  shows  that  the  proportion  of  hospitals
          In the patients' past  medical  histories,  more  than  half   that  can  identify  primary  discharge   diagnoses   increases
       of   those   given   a    cerebrovascular    hemorrhage    code   steadily from the  smallest  to  the  largest  group.  However,
                                                                                                                                       69





                 not all of the  largest  hospitals  identify  primary  diag-       We have tried to look at the patterns of  care  and  floA
                 noses. This failure is a  major  barrier  to  collection  of    patterns  throughout  the  region.   One   of   our   choser
                 evaluation  information.  You  may  want   to   know   about    measurements  was  the  interval  between  onset  of   symp.
                 patients with heart disease.  If  you  go  through  all  the    toms  and  admission  to  hospital,  The  next  slide  shows
                 index cards and count  all  people  admitted  in  a  certain    these intervals.  I  would  like  you  to  notice  the  "not
                 period of time  with  heart  disease,  you  end  up  with  a    recorded" column. About  20  per  cent  of  all  records  in
                 count of everybody who  had  heart  disease  listed  in  any    the sample were  without  information  that  would  help  us
                 ranking  order  among  their   discharge   diagnoses.   This    decide  the  delay  between  onset  of  symptoms  and   hos.
                 specific  problem  almost  doubled  the  staff  work  needed    pitalization.   These   incomplete   records    were    con.
                 to  abstract  information  for  our  survey.  To  make  sure    centrated in the smallerhospitals.
                 that  my  complaints  are  about  systems  and  not  medical
                 records  staff  let  me  first  show  you  evidence  of  the      Any  assumptions  from  these  data  about   patterns   of
                 magnificent  effort  and   cooperation   of   our   regional    medical care  have  to  be  made  with  the  knowledge  that
                 i-nedical records departments. We  asked  for  about  21,000    one in five pieces of information is  missing.  It  is  even
                 medical records.  The  percentages  at  the  bottom  of  the    harder  to  find  information  about  the  places   patients
                 next table describe the few records  the  record  librarians    were discharged to from the hospital.
                 could not produce for our review. It was  a  total  of  less    We  wanted  to  know  where  patients  go  when  they  leave
                 than 2 per cent of 2 1 000 records.
                   The next slide  shows  the  extra  work  we  undertook  to    acute hospitals. From the next slide we  see  that  in  some
                 identify the  diagnosis  for  which  each  patient  was  ad-    hospitals, 50  per  cent  of  the  medical  records  had  no
                 @tted.                                                          useful information on this point. Are  we  going  to  gener-
                   In this slide the "rejected"  records  were  those  pulled    alize our  findings  with  the  Maryland  region-we  cannot.
                 and  reviewed  but  unused.  The  main  reason  for  non-use    The data we have apply  to  only  a  very  small  number  of
                 was that the disease of interest was not listed first  among    hospitals and patients.
                 discharge  diagnoses.  You  can  see  from   our   "control"      We have tried to use other  types  of  morbidity  data  in
                 sample of all admissions other than heart  disease,  cancer,    our  region  to   get   some   baseline   measurements   for
                 stroke, diabetes mellitus, and chronic  bronchitis  that  12    expected  changes  over  time.  They   are   summarized   on
                 per cent of the records were not ii7cluded  in  the  sample.    die next  slide.  We  have  used  death  certificates.  Some
                 There were excluded for the following reasons:                  problems in the use  of  death  certificates  are  mentioned
                  1. the disease was not coded;                                  on the table. We only use  deaths  to  follow  up  the  out-
                  2. the  record  pulled  did  not  match  with  any   record    come of individuals  who  fell  into  our  sample.  We  have
                     number in our sample;                                       been trying to trace deaths in all of the people  that  have
                  3. the  admission  was   either   before   or   after   the    appeared in all of our samples. This is a large scale opera-
                     defined study period.                                       tioii. All names in our samples  have  to  be  matched  with
                 With the  major  RMP  disease  diagnoses  we  had  to  reject   names  that  appear  in  subsequent  mortality  data.   Once
                 as many records  as  we  accepted.  The  difference  between    we get the death certificates, the diagnosis  is  always  in
                 the  12  and  50  per  cent  was  due  to  the   non-primary    question and steps should be taken to get validation.
                 nature of the diagnosis,
                  To   summarizes   available   morbidity   information    is     We  have  tried  to  get  information   about   out-patient
                 biased  towards  large  hospitals.  These  hospitals  differ    visits. Those  of  you  who  use  out-patient  records  know
                 from smaller hospitals in their patient  populations,  their    their two major obstacles: There is  no  way  of  getting  a
                 availability of diagnostic techniques, quality  of  the  in-    list of diagnostic problems unless  they  are  listed  by  a
                 formation in medical  records  and  its  method  of  storage    secretary in a log book or clinic file as patients are seen;
                 and  retrieval.  We  should  recognize  this  bias  when  we    the OLIt-patieilt  records  themselves  have  no  "interval"
                 make  generalizations  about  changing  medical   care   and    diagnoses.
                 service programs on the basis of  local  and  national  mor.     We spent all last  summer  in  outpatient  clinics  getting
                 bidity information.                                             information  about  visiting   patients.   We   found   that
                 One  further  problem   in   using   morbidity   data   from    patients  attend  diabetic  clinics  for  years,  and  their
                 medical records  that  I  will  mention  today  is  that  of    record contains  no  definite  statement  that  the  patient
                 missing information  and  the  bias  it  may  have  on  your    has diabetes mellitus.  The  diagnosis  has  usually  to  be
                 final interpretation of those data.                             assumed.


                 70



      other  problems  met  in   our   surveys   are:   definitive    short-term  general   hospital   bed   needs    for    stroke
      diagnoses are rarely entered. Further,  the  information  on    patients?                     as  stroke  patients  are  con-
      ,vhich we could make a survey diagnosis is limited.              The next question, as far
      I  am  not  going  to  talk   about   functional   end-point    cerned,  is:  Do  we  need  acute  care  beds  for   admitted
      ,measurements  because  I  know  Dr.  Kelman  is  going   to    stroke  patients;  or  for  how  many  patients  do  we  need
      talk about them.                                                acute  care  beds?  We  are  on  the  planning  road  towards
      in  terms  of  disease  measurements,  out-patient   records    getting  the  answer  to  that   question.   Four   neurology
      and physician records have very limited value.                  centers have funds for acute  stroke  units.  They  have  all
      Finally,  I  would  like  to  show  you  some  of  the  ways    agreed  and  have  already  started  to   set   up   standard
      we are using  these  different  kinds  of  disease  measure-    criteria for all centers. These standard criteria will  allow
      rnents to get estimates of regional needs.                      us to describe the patients in  the  same  language  so  they
      One  of  the  questions  we  have  asked   ourselves   about    and their outcomes can be compared        ' The      standard
      our  region  in  general  is:  Should  every  patient   seen    criteria will also ensure that all patients have at  least  a
      (somewhere)  with  astroke  diagnosis  be  admitted   to   a    rwnimum   number   of   standard   diagnostic   tests.   Each
      hospital immediately. We  are  not  talking  about  patients    center. will add its own special tests to  its  protocol  but
      Nvho never appear in any  kind  of  medical  care  facility,    each-has agreed to use a standard basic protocol.
      only  those  who  appear  somewhere  in  the   health   care       Above  and  beyond  this  agreement  to   develop   stand-
      system. If so, how  many  bed  days  would  be  needed.  The    ard information in  the  four  centers,  we  are  working  on
      next slide shows an example  of  the  type  of  construction    the design of  a  randomly  allocated  therapeutic  trial  to
      we  are  making  to  get  this  information.  From  our  in-    allocate patients with different  degrees  of  severity  into
      patient survey we have estimates of  all  the  patients  ad-    our limited number  of  acute  stroke  beds  and  into  other
      @tted to our hospitals in  one  year.  From  a  surveillance    neurological beds. This study  will  identify  the  kinds  of
      of  the  emergency  rooms  of  certain  city  hospitals   we    patients for whom acute  care  makes  a  difference  in  out-
      know  how  many  individuals  with  stroke  diagnoses  visit    come.
      the  emergency  rooms  of  those  hospitals  and  are   sent       This  is  one  of  the  very  tight  end-points  that  Sam
      home.  From  these  two  sets  of  numbers  we  can  get   a    Shapiro was  talking  about  and  one  that  we  believe  has
      total number of people  with  stroke  diagnoses  seen  some-    tremendous implications  for  the  country  as  a  whole.  We
      where in the hospitals in a stated period of time.              want  to  be  able  to  say  how   many   (expensive)   acute
       We have  not  yet  added  into  our  sum  of  patients  the    stroke care beds we need.
      out-patients  with  new  stroke  diagnoses   we   identified       Finally, I would like to discuss  one  figure  I  borrowed
      during our out-patient survey.                                  from  Dr.  Matthew  Tayback  who   is   a   member   of   our
       Now,  what  else  have  we  done.  We  have   completed   a    department. It is a beautiful illustration  of  a  point  Sam
      follow-up study of all  patients  seen  in  emergency  rooms    Shapiro   mentioned   about   the   need    for    comparison
      and not admitted in a defined period  of  time.  We  visited    groups even when  you  are  looking  at  changes  over  time.
      all living  patients  two  years  later  to  find  out  what       Dr., Tayback has  been  looking  at  improvements  in  the
      happened to them since the  initial  emergency  room  visit.    outcome of pregnancy  in  relation  to  maternal  and  infant
      We also visited all  patients  admitted  to  the  same  city    care   programs.   These   outcome   measurements   show    a
      hospitals two years after they  were  discharged  and  asked    beautiful  downward  trend  in   phase   with   program   de-
      them  the  same  question.  We   also   know   whether   and    velopment. (Slide)
      when any patients in  both  groups  died,  and  whether  and       My colleague is wise enough to look at  trends  in  cities
      when  they  were  admitted  to  other  hospitals.  We   know    who  chose  not  to  develop   maternal   and   infant   care
      what they say about their experience  since  the  time  they    programs  during  the  same  years.  Curves  are  shown   for
      ,Went to the hospital when  some  were  and  some  were  not    prematurity  rates  and  for  neonatal  mortality  rates   in
      admitted.  By  putting  together  these  various  pieces  of    nonwhites.  The   hard   lines   representing   cities   with
      information we can look at  all  "recognized  patients  with    maternal and  -infant  care  projects  are  mirrored  exactly
      stroke" and see if there is any evidence of a difference  in    by  trends  in  the  cities  without  programs.  The  initial
      Outcome for similar  admitted  and  not  admitted  patients.    assumption  that  these  programs   are   easy   to   measure
        Our  outcome  measurements  for  this  study   are   death    because they  have  dramatic  changes  over  time  is  proven
      and hospital admissions.                                        wrong. it  is  very  hard  to  measure  the  value  of  these
        This  is  obviously  a  time-consunung  and   slow   study    programs because the other  cities  seem  to  be  doing  just
      but I hope it will give us some basis for estima   ting  our    as well.

                                                                                                                                  71



                     This one pair  of  graphs  illustrates  Sam  Shapiro's  point    were  admitted-then  we  wanted  to  be  able  to   say   there
                  that  I  want  to  emphasize-the  need  for   comparisons   even    has  been  an  improvement.  So  we   wanted   baselines   from
                  when looking at changes over time.                                  which   we   could   measure   improvements   both    in    the
                     I  would  be  glad  to  answer  any  questions  about   other    proces s and in the outcomes.
                  aspects of our studies later in the program.                           3.  We  wanted  to  be  able  to  describe  our  region   in
                                                                                      terms  of  patient.  movements   through   the   medical   care
                                                                                      facilities, consultation, delays etc.  We  felt  this  was  the
                                           Discussion                                 quickest way to get the picture.
                                                                                         The  best  alternative  was  to  take  a  group  of   people
                     DR.    CARPENTER:    That's    fine.    You     noted     the    with  each  disease  and  follow  them   through   the   system
                  complexities of analyzing data       from    existing     medical   for a number of years.
                  care  records,  and  Mr.  Shapiro  said  it  was  a   complexjob       We decided to use a cross-sectional approach.
                  to  devise  new   recordsand   get   decent   information   from       4.   The   last   purpose   was   to   identify   comparison
                  those.  This  is  one  indication  of  the  difficulty  of  end-    groups. We now  have  a  pretty  good  picture  of  the  people
                  result analysis.                                                    seen in all of our health care facilities in the region. If, as
                     Are  there  any  questions  for  any  of  the  panel  members    has  happened,  one  area  sets  up  a  program   for   chronic
                  from the floor?                                                     respiratory disease,  then  from  our  data  we  can  pick  out
                     QUESTION:  With  regard  to   the   stroke   patients,   when    an area that has similar patients  that  doesn't  have  a  pro.
                  you listed other  diagnoses  like  heart  disease,  presence  or    gram and maybe we can make comparisons.
                  absence,  what  criteria  were  used  in  deciding   whether   a       That was our rationalization.
                  stroke patient had heart diseases
                                                                                         QUESTION:   Point   of   information.   I   was    wondering
                     DR.   HENDERSON:   The   data   I   showed   you   are   from    how you could define morbidity.
                  hospital  records  that   were   already   in   existence.   The       DR. HENDERSON: How I define morbidity?
                  diagnoses  we  used   were   abstracted   from   the   discharge       QUESTION:  Yes.  It  came   up   in   your   discussion   a
                  diagnosis.  In  other   words,   we   copied   every   discharge    number of times.
                  diagnosis  listed  in  the  medical   record   onto   our   data       DR. HEN    DERSON:  I  suppose  I   was   just   using   it
                  form.                                                               loosely. In general terms it is a measurement of  illness,  as
                     Our  data  say  that  whoever   wrote   out   the   discharge    opposed  to  mortality  which  is  a  measurement  of   death,
                  summary  in  the  medical  record   listed   this   disease   as       There  are  obviously  different  kinds   of   measurements
                  being present.                                                      of  morbidity.  You  can  describe  the  disease  itself.  You
                     QUESTION-.  The  diagnosis  of   heart   disease   in   these    can  describe  the  use  of  services  by  people   who   have
                  patients  may  have  been  based  on  EKG   findings   or   not?    disease  or  by  people  who  do  not  have   disease   as   a
                     DR.   HENDERSON:   May    have    been    based    on    any-    r-heasurement  of  morbidity.  You  can  use  length  of  hos.
                  thing  the  physician  used  to  make  up  his  mind  that   the    pital stay. You can use measurements of function.
                  patient should be given the dignosis.                                  I used  the  term  in  a  generic  sense  meaning  measure-
                     QUESTION:        Are   these    face-sheet    diagnoses    or    iiients  of  everything  related  to  disease  separate   from
                  extracted from the discharge summary?                               mortality.
                     DR.    HENDERSON:     Discharge     summary.     Not     face       QUESTION:  How  long  did  it  take  you  to  gatlier  thin
                  sheets.  I  had  my  abstracters   copy   the   discharge   sum-    data? And in  the  meantime  did  you  wait  to  start  a  pro
                  maries at length.                                                   gram?
                     QUESTION:   What   was    your    hypothesis    in    getting       DR.  HENDERSON:  No.   Let   me   explain   our   situation
                  involved  with  reviewing  these  thousands  of  records   prior    in  Maryland.  We  have  an  epidemiology  center  which   ha!
                  to the operation of the regional medical program?                   been  busy  collecting  these  kind  of  baseline   data   an(
                     DR.   HENDERSON:   We   had    four    reasons    for    this    which  is  now  working  with  the  directors   of   individua
                  survey:  1.  To  get  information  for  planning.  We  felt   it    projects to set up their evaluation schemes.
                  was really unrealistic  to  plan  to  set  up  new  programs  or       The  center  started  after  the   Regional   Medical   pro
                  extend  programs  unless   we   knew   what   was   already   in    gram  began  and  projects  were  funded  before  any   survey
                  existence.                                                          were set in motion.
                     2.  The  second  reason  was   to   get   baseline   measure-       It is supported  by  RMP  funds,  but  is  administratively
                  i-nents for  evaluation.  If  there   were   improvements   over    in  the  department  of  epidemiology  in  the  Johns   Hopkin
                  time, for   example,   in   doing   more   EKGs   when    people    School of Hygiene and Public Health.

                  72




      It is an advisory and a scientific arm  of  the  program               QUESTION: Or whether a       facility  existed   that   they
      administered  by  the  University  and   it   was   very   iiii-    could  be  moved  to  that  would  have  been   a   physician's
      fortunate  that  the  program  began  before   this   particular    first or second choice?
      activity was funded.  However,  in  spite  of  the  delays,  the       DR.  HEN  DERSO  N:  We  have  not  done   that.   We   have
      result of  our  surveys  (now  being  cleaned  up)  seem  to  be    collected a lot of subsidiary  data.  For  example,  we  did  a
      coming at a good time.                                              survey to  identify  all  of  the  relationships,  both  formal
       In  my  opinion,  (with  its  limitations)   the   region   has    and  informal,  between  and  within  all  our  hospitals   and
      seen a lot of  activity  as  a  result  of  initial  funding  of    between  our  hospitals  and  all  other  institutions.  So  we
      programs.  People  became  interested   and   be-an   to   work.    do  know   with   which   nursing   homes   and   which   other
       Now   I   believe   we   know   the   active   and   interested    hospitals each individual hospital has relationships.
      members  of  our  professional  society  and  we  are   at   the       QUESTION:  Dr.   Henderson,   would   you   care   to   give
      stage  where  we  really  need   some   overall   direction.   I    an  opinion  about  the  necessity  to  have  a  program  which
      believe the Center's  results  are  going  to  be  available  at    would  significantly   improve   the   hospital   systems   for
      the  time  when  some  overall  direction  needs   to   be   de-    data collection  and  data  management  in  view  of  the  fact
      veloped.                                                            that it's terribly expensive and very difficult  to  set  up  a
       How  long  did  it  take  us  to  do  the  record  survey?   It    modern type of information system?
      took us a  year  to  collect  the  in-patient  data  and  another      Do you feel that the data  that's  needed  is  so  essential
      three  months  to  collect  it  from  the  out-patient  clinics.    that this is one of our major problems?
      Over  this  same  period  of  time  we   have   collected   data       DR.  HENDERSON:  Well,   you've   got   to   separate   this
      from  three  sets  of  admissions  to  a   sample   of   nursing    into  the  data  needed  for  patient   care   and   the   data
      homes  so  we  know  about  their  population   and   its   turn    needed  for  overall  planning.  The  speed  at   which   these
      over.                                                               systems have to run is different for the two purposes.
       I think it has been a  fantastically  rapid  job  in  terms  of       The   fast   systems   are   the   most    expensive.    Dr.
      the amount of information collected.                                Williamson knows much more about this than I do.
       We   are   currently   having   a   lot   of   problems    with       I  think  we  need  to  have  our  systems  improved,  there
      analysis  because   the   sampling   frame   was   really   con-    is  no  question  about  it.  The   major   data   problem   is
      founded  by  all  of  these   problems   we   met   in   getting    quality.
      samples  of  records  and  finding  which  ones   were   usable.       Most  people  improving  data  systems  are  really   taking
      We  are  presently  working  hard  at   sample   estimates.   To    no notice of the quality of the data.
      change  our  estimates  we  had  to  look  at  all  of  the  re-       In  my  opinion,  which  is  an  epidemiologist's   opinion,
      jected  records  (five  thousand  of  them)  and  tabulate   the    a  great  deal  of  effort  in  RMPs  across  the  country  has
      reasons  why  they  were  rejected.  We   have   just   finished    gone  into  the   technical   improvement   of   data   systems
      this exercise. Overall it has taken two  and  a  half  years  to    without  taking  any  notice  of  information  that  will  come
      collect, process, and begin to churn out data.                      out of the system in the long run.
        DR.  CARPENTER:  I  think  at  the   rate   things   move   in       Perhaps  we  tend  to  go  the  other  way  and  place   too
      Western  Pennsylvania  data  collected  within  three  years  is    much  emphasis  on  the  exact  meaning  of   the   information
      bound to approximate baseline data.                                 and its accuracy.
        QUESTION:   Were   you   able   to    differentiate    between       There  may  have  to  be   some   approach   between   these
      the care received  and  the  disposition  of  the  patient  that    two  points  of  view   before   we   reach   the   best   data
      actually  occurred  and  what  the  medical  person  in   charge    systems.
      would have wanted for them?                                            But  we   obviously   need   an   improvement   in   medical
                                                                          records systems.
        DR.   HENDERSON:   Not   from    past    records.    We    are       The  biggest  holdup  in  Maryland,  if  you  want  to  look
      doing  that  kind   of   thing   in   evaluation   of   separate    at  the  speed  with  which  information   becomes   available,
      projects. These are retrospective  data  so  they  are  hard  to    is   in   making   the   record   summaries,    getting    them
      validate.
                                                                          completed  and  getting  them  into   the   record   room.   No
        QUESTION:   Not   even   on   discharge    placement,    where    system is going  to  do  that.  You  have  to  get  substitutes
      their first choice of placement would have been?                    for the  physicians  or  give  the  physicians  time  to  write
         DR.   HENDERSON:   You   mean   you   go   to   the   patient    their summaries.
      and  find   out   whether   they   actually   went   there?   We       DR.   CARPENTER:   Dr.   Williamson,   do   you   want    to
      haven't done that. It could be done.                                comment on this?

                                                                                                                                        73






                     DR.     WILLIAMSON:     (Johns      Hopkins      University,    extent   to   which   the   poverty   and   nonpoverty   groul
                  Baltimore,  Maryland):   Yes,   I   agree   fully   with   what    differ in what  is  already  known  to  physicians  about  the
                  Maureen has  said,  and  my  own  bias  is  against  trying  to    health  conditions  and  the   health   conditions   that   ai
                  throw  a  lot  of  money  into  developing   a   fancy   record    found through this health testing program and what
                  system  when  nobody  has  an  idea  for  what   purpose   it's    the medical care behavior of people in light of what
                  going to be used  or  what  kind  of  decisions  are  going  to    known and can be retrieved from the medical records.
                  be influenced by the information you will get out.                    In  every  one  of  the  studies  with  which  I  have  bee
                     I think it's much better to try  to  aim  at  developing  an    associated  in  the  past,.  the   existing   medical   record
                  assessment  function  within  the  medical   care   group   and    systei-n has played a very important role.  In  some  cases  i
                  then especially to Yet  the  physicians  involved  so  that  it    has  been  more   subsidiary   than   in   instances   to   th
                  becomes  a  part  of  the  problem  of  trying  to  attack  and    specialized   effort   to   obtain   information,    overcomin
                  identify what problems  or  priorities  to  aim  at  first  and    some of the problems that existing records pose.
                  then what  kind  of  measure  they  want  to  mal,-e  and  then       QUESTION:  Does  the  HIP  regularly   i-nonitor   or   hav
                  start to  work  backwards  to  the  system  to  say,  "Now,  if    a  quality  control  system  with  regard  to  how   physician
                  we  are  going  to  measure  this  particular   disease,   this    enter  their  medical   records?   What   do   you   do   abou
                  particular problem, we  are  going  to  have  to  have  a  much    keeping up on a certain minimum level of quality?  ,
                  better form, and let's standardize it  so  all  the  physicians       MR.   SHAPIRO:   There's   no    continuous    inonitorijil
                  will use  it,  and  we  can  get  that  and  start  collecting,    system for  quality  of  the  information  being  reported  ii
                  standard data on that problem  to  see  if  we  can  arrive  at    the  medical  records  that  cuts  across  all   records.   Bu
                  some  conclusions  as  to  where  we  can  improve   the   out-    some of you may  recall  the  quality  of  care  studies  tha@
                  come of care for these kinds of patients."                         were carried  out  during  the  1950's  and  early  1960's  ir
                     So I would strongly  stress  going  after  the  function  of    HIP  in  which  the  information  in   the   medical   record!
                  evaluation  and  setting  up   expertise   and   getting   phy-    provided a  critical  source  for  evaluating  physician  per.
                  sicians  and  members  of  the  group   involved   with   that,    formance.
                  rather than taldng some part  of  the  process  and  trying  to       TMs  left  its  mark  on  the  system  and   had   a   ver3
                  bring this up in a  not  very  sophisticated  way  without  the    profound  effect  on  the  way  in  which  records  have  beer
                  balance  of  the  other  parts  of   the   system   that   will    organized and maintained.
                  eventually   lead   to   decisionmaking   and   altering    the       Also,  the  payment  system  within  HIP  does  contain   2
                  system itself.                                                     provision  for  annual  review  of  a  sample  of  records  ir
                     DR.  CARPENTER:  It's   almost   what   you   were   saying,    each  medical  group,  for  new  entrants  into   the   system
                  Sam.  One  needs  a   special   data   collection   system   to    and for entrants  during  the  previous  two  years,  and  the
                  measure  a  specific  end-result.   Existing   systems   seldom    quality of records is judged on  the  basis  of  this  review,
                  work.                                                              and money flows to the ones that meet the criteria.
                     MR.  SHAPIRO:  Well,   by   themselves   they   are   almost       So while we don't cut across  all  categories  of  patients
                  invariably not adequate  to  serve  the  purpose  of  the  kind    in  the  system,  there  is  a  considerable  amount  of   at.
                  of end-result studies that I was describing.                       tention given.
                     But at the  same  time  I  think  that  there  is  a  danger       I might  also  point  out  one  of  the  strengths  of  the
                  even within  the  framework  of  these  large  studies  that  I    record system in our plan  is  the  ability  to  retrieve  in-
                  was describing to  overlook  the  important  role  that  exist-    formation  without  breaking  your   back   going   out   into
                  ing record systems may play.                                       the general coi-nmunity.
                     For  example,  the  multiphasic   health   testing   program       There are 30  medical  group  centers,  and  a  centralized
                  that  I  very  briefly  described  and  that's   just   getting    record  system  provides  a  very  powerful  means   for   ef-
                  underway  in  HIP  is  very  heavily  dependent  for  some   of    ficiently  obtaining  access  to  the   total   medical   care
                  the evaluation  on  what  will  be  found  in  the  physicians'    received in the system.
                  records.                                                              QUESTION:  Many  dollars  have  been   spent   on   studies
                     There is  being  developed  a  retrieval  system  to  obtain    in the name of health  planning,  and  I  was  interested  ill
                  diagnostic  and   physician   and   other   medical   personnel    the suggestion that as a result  of  the  work  you  have  one
                  utilization information from the existing records.                 in  Maryland  you  anticipate  some  of  the  course  of   the
                     One  of  the  questions-it  isn't  an  end-result  type   of    program  might  be  modified  from  some  of  the  results  of
                  question. It's a process type. It's  related  to  process.  But    your study. I wonder if you could  just.  enlarge  on  this  2
                  one of the  important  questions  that  we're  raising  is  the    little bit.


                  74



         DR.   HENDERSON:   Well,   I'm   speaking    here    before            your  own  region  or  health  care  system.  Does  anybody
         ,e have looked at our total  data  even  for  one  disease.            want to claim  credit  for  that?  That  doesn't  mean  you
         @owever, I do believe that it does  point  out  differences            claim good data;  it  just  means  somebody  did  something
         I services, diagnostic services,  differences  between  the            because you showed data to them     Sam,   you    must    have
         ,verity of disease at the time people  get  into  hospital,            had that experience.
         rid also follow-up differences.                                        MR.  SHAPIRO:  Yes.  I   was   waiting   for   Maureen   or
         I  think  we  have  already  made  a  beginning  with   stroke,        Chuck or somebody else.
         Dr  example.  We  have  met  with  neurologists   and   pointed        Yes,  the  breast   cancer   screening   program   has   had   a
         ,ut that there  are  groups  of  patients  who  were  not  get-        very direct effect on what  is  being  included  in  our  multi-
         irig follow-up care. And I think just  looking  at  our  series        phasic   health   testing   program.   We're   going   to   have
         ,f  descriptions  of  patients,  investigations  and  modes  of        mammography   there.   There's   a   move    within    HIP    to
         herapy,  the  neurologists  are   going   to   come   up   with        include  as  part  of  the  general  physical  examination   not
         jeas  about  what  is  needed  to  improve  care   across   the        orfly  palpitation,  which  ordinarily  is  included,  but  also
         @bard.                                                                 mammography.
         That's  not  a  very  specific  answer  I  know,  but  I   have        Now,  this  may  sound  like  a   trivial   affair,   but   mam-
         ooked  at  enough  of  the  data  to  believe  we  are   seein-        mography is a costly procedure.
         remendous variations.                                                  Now,   if   the   information   we   currently   have    hardens
         If  the  neurologists   agree   that   certain   standards   of        over  the  next  couple  of  years,  there's-  Well,  I'm  going
         Jiagnostic   investigation   are   necessary,   and   we   show        to be the optimist  to  say  there's  no  question  in  my  mind
         Variations in frequencies of diagnostic investigations,  it  is        but that there will be  major  efforts  in  many  parts  of  the
         [he responsibility of the region's  neurologists  to  begin  to        country,  including  efforts  among  those   groups   that   are
         set up a program  to  see  that  necessary  investigations  are        concerned  with  the  regional   medical   program's   regional-
         available to all patients.                                             ization  and  expansion   of   services,   to   include   breast
         We  are  trying  to  provide  the  clinical  specialists   with        cancer screening.
         data  they  can  use  to  make  decisions  about  gaps  in  the        In fact, in  your  area,  Abe  Lilienfeld  has  a  project  that
         care.  I  think  we  see  enough  variation  to  predict  there        ties  with  RMP  to  have  every  woman  adn-dtted  to  hospital
         will be enough gaps to keep everyone busy.                             go through mammography.
         QUESTION:   One   of   the   morbidity   figures    that    you        This  program   I   am   sure   he   will   acknowledge   is   a
         showed in the first  slide,  Maureen,  was  prevalence.  And  I        direct consequence  of  the  tentative  findings  that  we  have
         wondered  if  you  agree  with  the   viewpoint   that   preva-        made in the breast cancer program.
         lence should be  one  of  the  last  measures  one  would  ever        DR.     CARPENTER:     Incidentally,     were      the      mam-
         use to  assess  the  effectiveness  of  regional  medical  pro-        mography  cases  usually  curable9  That  is,  the  cases   dis-
                                                       frequency      of        covered ordy by mammography,7
         grams, in that prevalence, which is the
         disease at any one moment of  time,  is  likely  to  go  up  if        MR. SH A PI R 0:  Well,  I'm  not  going  to  be  able  to  give
         regional medical programs are truly effective.                         you  a  direct  response  to  that  because  our   numbers   are
         And  this  is  a  somewhat   embarrassing   finding   that   we        still quite small. But the  histologic  type  of  breast  cancer
         would  probably   not   want   to   show,   although   we   may        picked    up    through    mammography    is    more     heavily
         want to know it ourselves.                                             concentrated  of   the   intraductal   type   where   there   is
         DR.   HENDERSON:   That   is   very   true.   We   are    aware        evidence  outside  of  our  program  that  survival  rates   are
         of it. We are trying  to  get  some  estimates  of  care  needs        much more favorable.
         in terms Of prevalence, not  look  at  the  outcorhe  in  terms
         of prevalence.                                                         At  this   point,   both   those   cases   picked   up   through
         The  MIC  program  is,  again,  a  good  illustration  of  your        mammography   and   those   cases   picked   up   independently.
         point.  In  areas  without  good  facilities  before  the  pro-        on  clinical  examination   have   very   favorable   and   very
         gram began  we  are  getting  an  increase  in  the  stillbirth        similar types of survival.
         rate, and an increase  in  the  mortality  rate,  only  because        DR.  LOGSDON:  I  would  only   add   as   far   as   end-result
         we are finding babies never registered  in  the  past.  So  the        evaluation  of  a  test  that  the   dental   examination   that
         rates are going up as the care initially improves.                     included  oral  cytology  had  such  a   very   low   yield   in
         DR.   CARPENTER:   Let   me   ask   the   audience   and    the        number  of  positive  cases  that  were  thereby  treated   that
         panelist  whether  anyone  has   now   in   their   hand   end-        this  was  deleted  from  the  process  rather  than  adding  to
         result  measurements  which  have  led  to  new  decisions   in        it.
                                                                                                                                               75





                       So that end-result evaluation can delete as well as              DR. HENDERSON: We have one that was not  rea'
                    add.                                                             an  outcome.  That's  why  I  was  not  speaking.  We   did
                       DR.    CARPENTER:    There's    another    good    example    follow-up study of  patients  a  year  after  they  had  be
                    then  of  how  end-results  can  change  the  system  -  end-    discharged  from  three   hospitals   in   Baltimore   with
                    result measurements.                                             diagnosis of stroke. We  wanted  to  find  out  what  medi(
                       Anyone else?                                                  care problems they had had in between times.
                       Do you  want  to  give  us  an  example  of  how  some  of       One of the reasons for doing  this  study  was  that  t'
                    the  end-result  measures   you   have   made,   John,   have    Maryland  Heart  Association  wanted   to   develop   stro,
                    motivated  either  your  own  institution  or  your  Regional    programs  and  wanted  to   know   the   needs   of   strol
                    Medical Program  to  undertake  health  care  a  little  dif-    patients living in the community.
                    ferently?                                                           We  found  that  many,  many  patients  said  that   thi
                       DR.  WILLIAMSON:   I   guess   the   two   most   dramatic    could not get to their usual medical care facilities  to  g
                    illustrations might be, first, our  heart  failure  study  at    their blood pressure measurements, their        ills, and ,
                                                                                                                                    p
                    Baltimore City Hospital, where we took a  look  at  a  range,    the  kinds  of  month-to-month  care   patients   with   tt
                    of outcomes from case  fatality  rates  to  people  who  were    kind of chronic disease require.
                    still out of work a year  after  leaving  the  hospital  that       They  could  not  get  there   not   because   they   we
                    should not have been out of work.                                paralyzed and couldn't be gotten  out  of  bed  but  beeau@
                       . And having found that the  results  did  not  meet  some    they could not  speak  well  enough  to  feel  confident  I
                    very stringent criteria we  set  up,  the  administration  of    travel or because they were  too  insecure  or  unstable  I
                    the hospital  was  impressed  but  didn't  do  anything.  But    go without an escort.
                    then they did  okay  some  more  studies.  And  then  one  of       And as a result  of  this  study  the  Heart  Associatic
                    my graduate  students  took  and  replicated  the  same  type    was  given  a  van,  and  it  now   has   a   transportatic
                    of  study  in  another  area  and  found  the  same  kind  of    program-the van driven and staffed  by  volunteers.  It  h;
                    rate.  For  example,  the  case  fatality  rate  was   almost    started to offer a free service to  needy  patients  in  tt
                    double  that  which  they  predicted  under  the   worst   of    metropolitan  area  to  take  them  to  their  medical  cai
                    circumstances.                                                   facility if they have no other means of getting there.
                       We identified that the problem  had  been  that  the  care       So we  did  have  a  particular  effect.  It  wasn't  RM
                    given during the time they were at  the  hospital  was  great    sponsored, but it was a community organization.
                    but it was that year after they left. Then this  other  study       The process  of  doing  evaluations  has,  in  fact,  ha
                    found  the  same  thing  -  they  then   appropriated   money    numerous effects  on  programs.  The  simplest  to  describ
                    and hired some  new  staff  and  set  up  what  they  call  a    is in our coronary care units. We have been oo 'ng
                    follow-up clinic to follow  the  patients  after  they  leave    coronary care units throughout  the  region.  AU  the  unil
                    even  through  the  hospital  may  not  have  responsibility.    have  beautiful  patient  information  forms  which  includ
                    They still wanted to find out  what  could  they  do  to  see    all  kinds  of   measurements.   Few   of   them   actuall,
                    that these patients get to another  physician,  to  see  that    measure and record weight.
                    they do fill their prescriptions, to see that they are  going       We  have  been  abstracting  information  from   one   c
                    to be followed.  With  heart  failure  there  are  disastrous    these units for some time, and they are  now  beginning  ti
                    results if they  don't  take  certain  medications  and  have    make much  better  attempts  to  get  complete  records.  W
                    certain medical care.                                            are having a real effect  on  the  recordkeeping  systems.o
                       And this has resulted in, I  think,  quite  an  innovative    the unit.
                    approach to this whole organization  of  the  clinic  system,       Again we have been going  to  hospitals  looking  at  th
                    and we are rather pleased  with  what  seems  to  be  happen-    performance  of  nurses  who  have   been   through   strok
                    ing.                                                             education programs. We  look  for  care  plans  and  whethe
                       Now,  the  payoff  will  be  to-wliich  we  want  to   do-    care plans changed after the nurses attend the course.
                    repeat the study  and  now  see  if  we  find  any  different       The process of  evaluation  inevitably  affects  ordinav
                    results as  far  as  outcomes  go  or  see  if  we  are  just    prograi-ns. This is not decisionmaking;  it  is  a  sort  0
                    measuring  something  where  there  are  other  factors  that    infiltration from the bottom.
                    might explain this.                                                 DR.   CARPENTER:   Very   good.    Does    anyone    els
                       But this  is  a  definite  change  that  occurred  in  the    have any examples?
                    whole clinic system as a result of these systems.                   Are  there  other  matters  then  that   came   up   thi
                       DR. CARPENTER: Good.                                          morning  in  the  discussion  of  the  problems  of  healtl

                    76




      status ineaSLirenici)" and the need  to  choose  prop-,  prob-  slii-nmaries of  the  essential  information  we  want  for  both
      lenis for the considerable effort required?                     patient care and research.  We  currently  have  a  girl  who  is
        DR.  MARGULIES:  You   know,   I   was   inipressolu   by   a over-edLicated for  the  position  working  with  us  to  set  up
      couple of things this niorninc,. One  of  them  is  the  sense  the  content  of   a   training   program.   We   are   hopefully
      of reassurance I got that in the years  that  I  lia,.,-  left  going  to  add  another  couple  of  candidates  in  the  spring.
      medical  practice  the  problem   of   medical   records   has      We are coiiil)ariiio the girl's  summaries  aoaitist  medical
                                                                                          t,                       LI
      remained so stable that I don't have  to  relearn  an%,thin(y.  residents' in one special area  after  the  other.  A  successful
      it's about where it was.                                        program   Would   be   a   great   step   forward   in   speeding
        But the other  thin(,  that  I  realty  wanted  to  raise  on tliin-s up.
                            Z,                                            LI
      the basis of the experience of tire people  who  lia%  -  been      DR. MA R G U L I 1; S: Of course, this still confines  you
      describiii- their efforts in evaluation and nieaSLirii-,2 Out-  to  what  you  can   do   in   improving   medical   records   in
              C,                                           -
      come is how important  they  feel  this  issue  of  ii-@edical  hospitals. And  on  a  continuing  basis,  as  you  pointed  out,
      records is. Obviously You feet that it is very ilillik-)rtallt. you  have  had  to  confine   your   observations   to   isolated
        This then raises the basic question in my iiiiiiL@  in  aii%@ incidents and, in fact, to  patient  response  on  the  basis  of
      kind  of  evaluative  procedure  of  liaviiia   adequate   in-  their own experience.
      fori-nation. And if you are goin- to measure            %,oti       DR. HENDERSON: Right.
      are  doin-  and  measure  the   effects   of   any   kind   of      DR. LEWIS: I'd like to jump in and comment.
      activity, whether it's reaionalization  or  clini,:q-l  pro-        I  think  besides   Maureen's   prograi-n   with   medical
      cedL[rC,  whatever  it  may  be,  as  you  ver%,  co-rectl@@    records  librarians  your  comment  raises  the  issue  as  to
      pointed out, you  have  to  have  somethin-  that  %oti  can    whether  or  not  ambulatory  care,  traditional  or  radical,
      compare with something else.                                    can  ever  be  evaluated  without   a   problem-oriented   ap-
        For Regional Medical  Pro@rams  that  could  ver\  easil@@    proach to recordkeeping.
      be a major goal-to took at the capacities which         have        DR. MA R G U L I F S: That's really what I'm  getting  at.
      for influencing medical records, for introduciii- stability,    Can it be? I doubt very much it can.
      consistency, and so forth.                                          DR. HENDERSON: No, I do not think it can.
        Now, that's one aspect of it,  but  you  also  pointed  to        DR.  LEWIS:  The   second   point-and   it's   even   more
      another which is of real concern, and that  is  the  Margins    subtle  than  that-is  the  problem  of   distinguishing   be-
      perceptions  of  medical  record  librarians  of  how   they    tween  the  actuarial  content  versus  the  contractual  ele-
      perform in a record system.                                     ments of the medical record.
        Have you pursued this  particular  issue  further  and  do        Let  me  put  it  back.  Sam  and  many  of  us  would  be
      you have some advice for LIS?                                   interested in data that allows us to do a life table  kind  of
        DR.  HENDERSON:  Well,  we  have  been  pursuinz   it   in    following of  what  happens  in  time  on  patients  from  the
      several  ways:  One,  throu-h  setting   up   meetin2s   and    actuarial kind of prognostic point of view.
      instruction. As a matter of fact, it is not really  instruc-        In fact,  however,  when  one  looks  at  medical  records
      tion. Interestingly enouch it is easier to  use  the  Inter-
                              C,                                      in  tracing backwards the history, one is a  prisoner  of  the
      national Code than  a  standard  nomenclature.  We  have  to
      11 unlearn"  the  record  librarians.  We  have  tried  to  en- kind  of  medical  information  which  that  physician   chose
      courage them to change their use of codes.                      to write down which  really  was  in  part  @  fulfillment  of
        We also have a  pilot  study  setting  up  an  educational    his contract with the patient.
      program for a new kind of person  that  we  are  callinc,  a        And one has a  highly  biased  view  of  the  world,  much
      medical records summarizer.  I  said  earlier  that  medical    of which  serves  to  remind  everybody  who  will  read  that
      record summary  is  one  of  the  biggest  hold-ups  in  the    record that he was in fact doing a good  job  as  he  saw  his
      record system. We have had  in  our  research  pro-rams  for    job with that patient.
      many  years  medical  record  abstractors   who   are   very        This sort of contractual,  or  legal,  ethical  reason,  I
      competent  and  can  abstract  a  medical  record  perfectly    think,  is  one  of  the  more  serious  problems  which   has
      well, getting all the detailed information we need.  We  are    been  cited  by  Garfinkel   and   others   outside   of   the
      now trying to see whether we can  train  an  assistant  with    medical  care  system,  and  one  which  raises  the  question
      this capacity who can be  used  in  a  service  function  to    as to  whether  or  not  professionals  can  record  actuarial
      Summarize the medical record to  the  physician's  satisfac-    information without  the  kind  of  super-structure  that  has
      tion so that he will sicn it. This would really gj%,-  us  a    been built in special long-term  studies  to  get  information
                             b
      much more rapid flow of  records  and  we  will  get  better    that is other than almost a self-fulfillment prophecy.

                                                                                                                                  77




               DR.  CARPENTER:  Let  me   see   if   I   can   take   a   more have  an  influence  on   hospital   records   that   you   can
               positive  point  of  view  about  the  medical   records.   Our perceives
               data  was  obtained  from  medical   records,   and   we   were MR.   SHAPIRO:   We   don't   have    the    experience    yet.
               curious as  to  what  we  could  find  out  about  the  medical We  are  going  to   become   operational   in   November.   So
               records, how bad or how good were they.                      that's  a  very  easy  question  to  answer.  We  don't   know.
               We  tried  to   find   certain   expected   correlations.   One But this issue is one part, one phase of our evaluation.
               would  think  if  a  patient   came   in   comatose   that   he I  want  to  comment  on  the   quality   of   records   issue.
               ought  to  die  more  frequently  than  someone  who  came   in What Chuck has  referred  to  as  the  actuarial  approach  can
               alert. And this kind of correlation, in  fact,  we  could  find be thought of  in  terms  of  prospective  studies.  There  are
               in  the  records.  But  maybe  the  correlation  is  so  strong enormous  difficulties  even  under   the   best   of   circum-
               that  even  with  a  much   muddleheaded   recording   in   the stances   when   you   try   to   use   information   collected
               charts it is evident.                                        during a previous period.
               If  you  get  past  the  diagnosis  sheet  and  look  at   what You   have   the   problems   of   reconstituting   a   popula-
               the doctor wrote in  the  record  you  can  learn  some  inter- tioii. You have problems  related  to,  again  even  under  the
               esting things.                                               best  of  circumstances,  absence  of  information   that   did
               For  instance,   in   the   county   we   studied,   the   Sig- not appear to be terr.ibly relevant initially.
               nificance of coma  with  no  neurologic  signs  is  really  not This  in  no  way  detracts  from  the  importance   of   major
               adequately  recognized.  Often   if   the   spinal   fluid   is efforts to  improve  medical  records,  and  we  too  are  get-
               examined  and  blood  is   found,   the   diagnosis   of   sub- ting  involved  in  new  approaches  to  improve   quality   of
               arachnoid hemorrhage is  not  made.  And  if  spinal  fluid  is records.
               not examined, the urine may not be examined either.          I  want  to  emphasize  that  in  every  research  project   in
               It   may   show   unrecognized   4-plus   sugar   and    4-plus which   outcome   measures   have   been    used,    we    have
               acetone.                                                     depended  on  the  HIP  medical   records   in   one   way   or
               So by  getting  past  that  face  sheet  into  the  details  of another.  The  record  has   not   supported   completely   the
               care,  somebody  who  is  adequately  trained   can   learn   ainvestigation,  but  without   the   record   we   would   have
               fair amount.                                                 been in terrible  difficulties.  Even  in  the  "purest"  types
               We  are  now  in  the  process  of  saying  to  the  people  in of outcome  studies,  existing  good  records  systems  can  be
               our  study  county,  "Some  of  you  lose  more  patients  than of invaluable assistance.
               others.  The  difference  is  not  related  to  age,  sex,   or DR.  LEWIS:  If  I  may  make  a  comment,  since   the   issue
               certain measures of severity."  We  also  can  say,  "it  looks has arisen, I think there  is  a  tendency  to  confound  tech-
               as  though  you're  not  all  doing  an   adequate   neurologic nology with validity, or neatness with validity.
               exam.  Generalists  lose   many   patients   without   definite For   example,    Maureen's    comments    about    PAS    hos-
               neurologic  signs  who  are  diagnosed   as   stroke.   Similar pitals  who  had  a  printout,  but  had  no   better   records
               patients  (without  clear  signs)  who  are  treated   by   the than  those  who  didn't  support  this.  I'm  sure   some   of
               internists die less ftequently."                             you are  aware  that  lots  of  people  are  pushing  automated
               These  and  other  data  lead  us  to  conclude   that   though Wstory-taking   and   computerized   forms    so    that    the
               hospital  records  are  imperfect,  they  do   contain   useful physician gets a very neat  printout.  The  issue  of  validity
               data.                                                        seems  to  have  been  totally  overlooked  in  a  good  number
               QUESTION:   Was   there    any    evidence    that    hospitals of these projects,
               that take part in the PAS program of the  Committee  on      Whether or  not  it  looks  neat  and  comprehensive  is
               Professional Hospital Activities keep any better records     one thing, but whether or not it      means    anything,
               than those who don't take part in that program?              whether or not anything has really been measured that is
               DR.  HENDERSON:  No.  No,  we  looked  it   that.   The      of any value, is another.
               only difference was they could supply us with a printed
               index. That was very helpful.                                           How to Measure Health Status
               DR.  CARPENTER:  Are  any  of  your   hospitals   using                 HOWARD R. KELMAN, Ph.D.
               any kind of automated history?
               DR.HENDERSON:No.                                              What I   would like to talk  about  are  some  ways  ir
               DR.   MARGULIES:   Does   the   utilization   of    the      which - and I think for this audience this will not neces
               screening program, the automated multiphasic screening,      sarily come as anything new  or  unique  -  others  hav(

               78




     looked  at  health  status  and  have  tried   to   measure   it    you  get  away   from   biologic   measures   of   how   people
     focusing   principally   around   the   measurement    or    the    function, the more  the  function  of  the  individual  is  in-
     determination  of  disability  and  related  kinds  of  measures    fluenced  by  non-biologic  factors  such  as  their  immediate
     of discomfort or dissatisfaction.                                   social environment, their  aspirations,  their  past  histories
        I  think  it  was  probably  Kerr  White  who  first   coined    and future desires.
     the five  "D's"  of  measurements  of  health  status  -  death,       So  that  what  we  might  try  to  attribute   to   medical
     disease, disability, discomfort, and dissatisfaction.               care maybe gets  less  and  less  influenced  by  what  medical
        And it seems to me to be  as  good  a  way  as  I  can  think    care  can  do  and  more  so  by  what  the  patient's   social
     .of to define the  different  kinds  of  ways  in  which  health    situation is like.
     can be thought about and determined.                                   I  wanted  to  put  that  out  to  begin  with   because   I
        Our   speakers   this   morning   have    concentrated,    or    think  that  sometimes  we  make  the  assumption   -   and   I
     focused  I  should  say,  most  of  their  discussion  and   at-    think  I'm  as  guilty  as  anybody  else  -  that  whether   a
     tention  on  measures  of,  and  utflization  of  measures   of,    person can or cannot walk or %vill  or  will  not  go  to  work
     mortality and disease or  sickness,  and  I'd  like  to  talk  a    is due solely to whether  he  feels  and  actually  is  healthy
     little bit about the third D and maybe get  a  little  bit  into    or appears to be healthy.
     some of the other D's.                                                 For  example,  there  are  questions  as  you  know  in  the
        Of course, I  couldn't  help  but  get  the  feeling  follow-    national  health  survey  which  ask   people   whether   their
     ing  the  discussion  early  this  morning  that,   why   bother    activities have been restricted due to illness. Well,  this  is
     even to  begin,  when  we  have  so  much  ground  to  cover  in    a loaded question, it  seems  to  me.  It's  something  that  I
     terms  of  defining  really  what  the  RMPs  are  supposed   to    think  we  need  to  consider  with  regard  to   measures   of
     do, to begin  with,  and  to  achieve  and  develop  a  kind  of    disability.
     apparatus  for  assessing  these  largely  undefined  or  global       The  other  thing  I  think  we  need  to  think  about  are
     objectives.                                                         the data  sources  for  information  of  this  sort  which  are
        But  I  suppose  if   we   waited   until   objectives   were    different than those stressed by the previous speakers.
     clearly  delineated  and  everybody  was   really   sure   about       There is  less  dependency  here  on  hospital  records  and
     what  they  wanted  to  do,  we  might  not  even   be   meeting    those  kinds  of  reporting  systems  with  their  degrees   of
     here.                                                               unreliability and uncertain  validity,  and  more  reliance  on
        Let me go on  a  little  bit  further  and  talk  about  dis-    a  hard  source  of  information  -  namely,  the  patient   or
     ability measures and  why  and  how  it  might  be  utilized  in    somebody who cares for him.
     RMP  programs  -  which  I  know  very  little   about   because       Now, I know  that  it  has  been  traditional  to  think  of
     my connections with RMP        have  been   rather   peripheral,    measures  of  social  functioning  as   relatively   soft   and
     That is, I've been approved but not funded.                         measures  of  morbidity  and  mortality  as  relatively   hard.
        I  suppose  it's  worth  starting  out  by  asking:  Why  get    But  I'm  convinced  by  what  the  first  two  speakers   told
     concerned  with  disability  or   discomfort   or   dissatisfac-    me this morning about  how  really  soft  the  latter  kind  of
     tion? What has that got to do with medical care?                    information is  -  and  I  would  say  I'll  put  my  bet  down
        I  leave  the  obvious  answer  to  that  to  you  to   think    on the patient.
     about  for  only  two  seconds,  because,  for  a   variety   of       But, quite seriously, I think  the  whole  question  of  the
     reasons,   we   have   become   increasingly   concerned    with    reliability of patient in terms of  asking  him  how  he  feels
     the  social  and   economic   and   psychological   consequences    and what he's able to do or  not  do  for  himself,  or  asking
     for  living  of  individuals  who  survive  medical   care   (or    his  relative  or  asking  somebody  who   has   observed   the
     chronic fllness) and what  is  done  to  them  or  for  them  or    patient  either  in  a  treatment  program  or  on   a   visit,
     on their behalf.                                                    whether  it  be   a   visiting   nurse   or   an   occupational
        The   increasing   concern   with   chronic   or    long-term    therapist or  an  interviewer,  does  pose  technical  problems
     illness  and  the  consequences  of  that  for  individuals   in    of reliability and validity which  are  related  to  but  some-
     terms of their ability  to  function  physically,  socially  and    what  different  than  the  kinds  of  problems  that  we  have
     psychologically has led  to  the  desire  to  regard  disability    heard about this morning.
     as a  sequela  of  long-term  illness  and  how  this  might  or       Now, in thinking  about  this  subject  and  in  some  prior
     might not be affected by the care that people receive.              conversations  with  Dr.   Carpenter,   he   inquired   as   to
        One  of  the  major  problems  I  think  we  face  in  trying    whether  there  were  some  kind  of   standard   measures   of
     to look  at  disability,  to  measure,  to  define  it,  and  to    disability, social  functioning  -  you  didn't  use  the  word
     then try to relate it to medical care, is that the further  .       but I did "happiness" - those kinds of things.

                                                                                                                                      79




                    There  are  "standard  measures,"  and  each  person   who    Security Administration  with  regard  to  disabled  persor
                 does  a  study   develops   his   own   "standard"   measure.    and how they function in the community.
                 There are good and sound reasons for this.                          Essentially, the kinds of information that  they  collec
                    One of them is that it is  exceedingly  difficult  to  get    are geared to basically well or "non-sick"  populations,  s
                 any  real  consensus  that  goes  beyond  the   confines   of    that how relevant it is to populations of sick people
                 perhaps an advisory group    'about  what  you  mean  by  dis-   something you really -have to decide for yourself.
                 ability  and  what  you  mean  by  social   functioning   and       But with regard  to  the  question  you  raised  earlier
                 whether  any  of  these  things  have  anything  to  do  with    Maureen, about denominators,  I  think  this  is  where  th
                 the program that was  supposed  to  influence  any  of  these    sort of information may prove  to  be  helpful  if  you  ea
                 states.       -                                                  use the  current  information  and  if  you  find  that  th
                    The other problem  is  that  what  might  be  regarded  as    numbers are adequate for  the  population  you  are  talkin
                 disability in a  person  with  physical  impairments  is  not    about.
                 necessarily going to  cover  the  same  kind  of  ground  for       Now, one of the problems, of course, is  that  these  ai
                 presumably well people out in the community.                     usually national surveys, and depending  upon  the  size  c
                    So that if you are interested  in  a  small  increment  of    your  local  community,  you  may  only  have  a  sample  c
                 change, let's say, in whether a person  can  now  dress  with    six or eight people in this national study.
                 or  without  some  kind  of  assistance  because  they   have       In any event, you may be  able,  with  the  aid  of  ver
                 sustained  some  kind  of  motor  or  neurologic  impairment,    competent people, to relate the  local  population  you  ai
                 that would not  necessarily  be  an  appropriate  measure  or    concerned with to adjusted rates based  on  t  ese  nation,
                 question to  ask'  of  somebody  who  is  out  there  in  the    sources. I think it's something  that  we  don't  ordinaril
                 community  and  who  is  unemployed  for   one   or   another    think too much about, at least in this area.
                 kind of reason.                                                     Now, some of  the  kinds  of  things  that  they  try  t
                    If  you  wanted  to  develop  a  battery  of  measures  ot    coflect information.  on  in  this  survey-  I'll  just  ru
                 social,  physical  and  psychological  functioning   to   run    through it. very briefly. I'm sure  that  many  of  you  ar
                 the  gamut  from  patients  who  may  be   nearly   or   com-    familiar with it.
                 pletely  bedbound  to  those   who   are   both   fully   am-       They  ask  questions   about   days   lost   from   wort
                 bulatory and who work  quite  effectively  as  physicians  or    wholly or  partiafly-the  extent  to  which  the  individu,
                 legislators or RMP coordinators,  this  is  as  yet  a  quite    has restricted activity days I guess is the actual term tha
                 formidable  task  to  get  anytl-iing  beyond   the   crudest    they use-whether  there  are  mobility  limitations  preser
                 kinds of information.                                            or whether the person in a sense is either confined  to  th
                    I think the other point that needs  to  be  made  is  that    house or can get about without any  kinds  of  difficulties
                 those of us - and there are many of  us  in  this  room  that       And they also inquire, very interestingly  it  seems  ti
                 I  recognize  and  many  who  are  quite   expert   in   this    me, about the person's social role activities - that is, th
                 field-don't view  these  measures  really  as  replacing  the    occupational information, if  the  person  is  a  housewife
                 more traditional  and  hard-to-get-at  and  harder  kinds  of    or if  a  child,  whether  there  have  been  any  activit:
                 information   centering   on,   you   know,   mortality   and    restrictions with regard to'those roles.
                 morbidity,  but  really  try  to  see   these   measures   as       I'm not sure what  they  do  about  people  who  are  6:
                 perhaps  other  kinds  of  ways  in  which  the  benefits  or    and over, because 'we have no real social role  defined  fo
                 lack  of  benefits  of  programs   can   be   documented   or    those individuals unless one can call retirement a role. Si
                                                                                  I think part of the  roblem  in  talking  about  people  ag
                 tested.                                                                              p
                    What are some of the ways       in  which  disability  has    65 years and.  over  is  that  they  would  probably  scor,
                                                                                  pretty low  on  these  scales.  They  don't  work,  perhaps
                 been thought about and how        have   some   people   been    They may never get out  of  the  house.  (I  shouldn't  sa,
                 going about it? Perhaps a word or two on that.  ,                11 never.")
                    I  have  already  referred,  I  think,  to  the   National       Now,  ranging   from   those   types   of   very   globa
                 Health Survey, and I think it's  important  particularly  for    measures- and  I  again  want  to  emphasize  that  the  in
                 persons  concerned   with   broad   population   groups   and    dividual is asked and his response  categorizes  him  -  t(
                 planning for their care and meeting  their  care,  like  RMP,    say whether limitations or restrictions in activity are  du
                 to  be  aware  of  the  kinds   of   information   that   are    to  illness  limitations.  Nobody  exan-dnes  him.   Nobod,
                 produced  not  only  out  of  the  National  Health   Survey,    decides a pfiori. The  individual  categorizes  himself  ii
                 but  also  more  recent  studies  conducted  by  the   Social    terms of his response.

                 80




         So  that  you  have  disabflitv  measures  of  that  type  on   been   based   on   information   obtained   from    professional
      one end of the  continuum,  if  I  can  put  it  that  way,  and   people  who   know   the   patient.   Sometimes   it's   a   team
      at the other  end  of  the  continuum  you  have  a  variety  of   making  a  judgment  based   on   their   experience   with   the
      different  kinds  of   measures   of   function   which   center   patient,  coming   up   with   a   group   judgment   about   the
      around  a  core  of  what  have  come  to  be  defined  as   ADL   patient  or  individual.  In  other  scales  it's  a  single  in-
      activities   -   tofleting,   dressing,   feeding,   ambulation,   dividual  who  knows   the   patient,   who   may   have   worked
      transfer  activities.  You   name"   it   and   you   can   find   with  the  patient,  or   who   maybe   sees   the   patient   or
      "scales" for it.                                                   former  patient  out  in  the  community  -  a   nurse,   perhaps
         And   these   kinds   of   measures   have   been   developed   or   an   occupational   therapist.   Sometimes    the    patient
      essentially to look  at  rather  severely  disabled  people,  or   himself or a relative is the source of information.
      those   with   potentiality   of   becon-dng   quite    severely      it  may  be  almost  like  splitting  hairs,  but  we,   some-
      disabled, who  require  a  great  deal  of  care  and  who  have   times  seem  to  take  these  three  rather   disparate   sources
      rather  profound  limitations  in  the  ordinary  activities  of   of  information  on  particular  individuals   and   throw   them
      daily living.                                                      all together  as  though  they  possessed  similar  qualities  of
         And  you  will  find  on  this  end  of   the   continuum   a   reliability and validity -  and  of  course  they  don't  really.
      variety of different kinds of scales,  all  of  which,  or  many      However, with all  of  these  problems,  as  I  said  earlier,
      of  which,  have  proved  to  be  quite  useful  in   terms   of   judicious  selection  and  use  of  these   scales   has   proven
      evaluating change in  patient  status  over  a  period  of  time   quite  valuable  in  terms  of  determining   whether   a   given
      or over a  period  of  exposure  or  lack  of  exposure  to  one   program  is   having   some   appreciable   effect   on   raising
      or another kind of treatment programs.                             levels  of  function  of  disabled  individuals  or  on   whether
         The scales  vary  in  terms  of  the  actual  dimension  that   it has reduced their need for assistance.
      they are cutting.  But  what  they  really  are  trying  to  get      Particularly   with   regard   to   individuals   in   nursing
      at, it seems to  me,  is  the  extent  to  whici  not  only  the
      individual  can  perform  at  one  or  another  level  but   the   homes  or  who  require  great  amounts  of   nursing   care,   a
      extent  to   which   this   performance   is   based   on   some   small   increment   of   gain   from    dependence    to    inde-
      increment  of  care  or  assistance,  whether  this   assistance   pendence, let's say, in  an  activity  like  tofleting  can  mean
      may be  given  by  someone  in  the  home,  a  relative,  or  by   a great deal over a  period  of  time  in  an  institution  where
      somebody  in  the  treatment  institution  -   the   extent   to   many  in  the  population  may  require  a  great  deal  of  care
      which  the  individual  can  perform   this   particular   func-   and assistance in terms of toileting.
      tion, dressing,  tofleting,  transfer,  etc.,  independently  or      Certainly  I  don't  need  to  remind  this   group   that   a
      dependently.                                                       small increment of  gain  in  tofleting  in  a  patient  who  has
         And   the   scale   endpoints   usually   range   from   some   to be taken care  of  at  home,  while  it  may  only  reflect  a
      level  which  is  either  "unable  to"   or   "completely   bed-   jump  from  3  to  2  on  the  scale  position,  may  reflect   a
      bound"  to,  "can  do  by  self"  or  "requires   no   kind   of   great  deal  more  in  a  home  situation  if  someone  has   the
      assistance."                                                       responsibility for the care of that individual.         I
         Now,  investigators  have   usually   used   a   battery   of      I  think  then  that  one   of   the   other   problems   with
      these scales, and in some  of  these  scales  different  weights   these  scales  is  the   fact   that   a   stepwise   jump   from
      are  assigned  to   different   functions   -   ADL   functions.   position  4  to  position  3,  while  it   looks   mathematically
      Other  scales  do  not  assign  different  weights  but   rather   neat, may  not  have  the  same  kind  of  social  meaning  as  a
      give  equal   weight   to   performance   on   each   of   these   jump from 4 to 3 on another scale.
      functions. Still  other  kinds  of  scales  are  concerned  with      But these  are  generally  problems  of  scale,  and  I  don't
      whether the patient needs help or doesn't need help.
         Some  years  ago   in   a   study   we   had   underway,   we   think they are specific really to this kind of problem.
      found  several  different  ways  in   which   disabled   people,      When  we  move  from  this  more  or  less  traditional   area
      could  be  scaled  or  the  scores  manipulated,  and  we  found   of definition  of  disability  or  disability  determination  and
      that where there  was  change  each  of  these  scales  revealed   its  application  either  to  broad  populations   or   to   more
      pretty well who was going to be changed.                           narrowly defined  clinical  or  patient  groups,  into  the  area
         Where there  wasn't  any  change,  it  didn't  really  matter   of  discomfort,  into  the  area  of  dissatisfaction,  into  the
      which of these scales were used.                                   area  of  social  functioning  with  regard  to  let's  say   the
         Now,  one  of  the  other  things  I  should   mention   with   family or  the  community,  we  get  into  terrain  that  is  not
      regard to these scales is that  they  have  for  the  most  part   nearly as well worked over.

                                                                                                                                        81




                                                                                    rnents Dr. Henderson made about followup studie
                   I guess in large part we don't really think about  or  try    corn
                to affect family relationships, if I can  put  it  that  way,    that we ought to be interested in things like  whether  the.,
                when we think about stroke patients.                             patient is now better or less able  to  communicate,  to  use
                   I  suppose  the  connection  between  whether  the  stroke    the medical care system, to  manipulate  it  to  their  own,@
                patient  will  now  get  along  better  or  worse  with   his    benefit. Maybe this ought to be, if  it  isn't,  one  of  the
                spouse, and the application on  the  other  hand  of  medical    kinds  of  things  we  ought  to  be  aiming  at  with   sick
                measures to first see if you can keep the  person  alive  and    people.
                then to make living a little more livable for the  person  in       Their  whole  knowledge  of  what  is  wrong   with   them
                biological terms in distant.  Life  saving  does  and  should    and  what  they  might  do  about  it,  I  think,  represents
                take precedence. But we pay a  lot  of  attention,  at  least    another area that might  be  thought  about  with  regard  to
                on paper, to  social  well  being,  and  maybe  we  ought  to    looking at some of the kinds  of  programs  that  have  been,
                begin to  think  of  broadening  some  of  our  concern  into    or that ought to be, developed.
                some of these areas.                                                What I'm suggesting is that for a variety  of  reasons  we
                   I shouldn't want to leave you      with   the   impression    may not  be  able  to  affect  very  basically  the  biologic
                that there aren't studies of social well  being  of  well  or    functioning  or  biologic  status  of   many   disabled   in-
                sick people and that there  aren't  studies  of  family  well    dividuals. We may be  better  able  to  affect  some  aspects
                being  or  cornpatibflity,  community.participation   and   a    of the individual social situation,  his  social  or  psycho.
                variety of other kinds of social  measurements;  e.g.  social    logical functioning, or the function  of  those  around  him.
                isolation, work satisfaction, work performance.                     I  don't  know  why,  for  example,  the  National  Health
                   But what I'm suggesting is  that  in  terms  of  at  least    Survey doesn't ask  at  least  for  information  from  family
                some of the kinds of programs  that  we  are  talking  about,    members, and what their, input is in terms  of  care  of  the
                it may be well  to  think  not  only  of  scales  which  more    sick person.
                directly seem to be related to biological  efforts  centering       That is, if the individual replies that he is not able  to
                around  disability,  but  also  scales  which  seem  -   only    work because of illness 'oughtn't  we  to   get   information
                seem  -  less  related,  a  little  more  remote   from   our    on whether the social  role  of  some  other  individual  has
                interests - for several reasons.                                 been altered as a result of that? Is that not really part  of
                   One is our own bias. That is, it may  very  well  be  that    the disability picture that we all  see  pretty  often?  Does
                while  we  may  be  increasing  the   person's   ability   to    this now mean  that  somebody  else  in  the  family  is  now
                function  independently   in   one   or   another   area   of    working?  For  somebody   that   is   disabled   and   cannot
                activity, this  may  have  quite  deleterious  effects,  when    work,  does  this  now  mean  somebody  else  in  the  family
                this  person  gets  home,  on  his  family.  We  don't   know    situation's work role has been affected?
                that unless we look at it or think about it.                        What I'm  suggesting  then  is  a  kind  of  broader  view
                   The reverse may also be true.  We  may  have  very  little    that we might think about with  regard  to  the  plethora  of
                success, for one or another reasons, in  terms  of  basically    effects that programs  that  have  been  developed  ought  to
                affecting  the  physical  level  of  functioning  of  an  in-    be looked at in terms of status or benefit.
                dividual, but perhaps the application  of  other  aspects  of       Now,  the  obvious  retort  to  that  is  that   you   can
                the program has  had  beneficial  consequences  in  terms  of    extend the concept and idea  of  health  status  to  a  point
                how  the  family  may  now  function  or   how   the   person    where it begins to be so diluted  as  to  lose  its  meaning.
                may function in other kinds of areas.                            But I don't think that  some  of  these  kinds  of  questions
                   I think part of the problem in  moving  into  these  areas    that I have raised or some of the areas of  inquiry  that  we
                is twofold. One is to make, as  we  all  do,  some  kinds  of    ought to be undertaking are that far afield for us  at  least
                decisions  out  of  the  plethora  of  dimensions   of   psy-    to think about.
                chosocial  functioning,  those  which  have  some   kind   of       There are also  areas  in  which  there  is  enough  meth-
                more  plausible  relationship  to   medical   care   programs    odologic  experience  and  technique  and  enough   familiar-
                than others.                                                     ity in the health field in terms  of  sample  survey  that  a
                   And I  think  here  that  we  do  have  a  wide  selection    ready transplantation - and I  use  that  word  advisedly  in
                of- "scales" is hardly the  word  I  think  to  use  in  this    this particular context - of these  kinds  of  efforts  would
                regard -  but  dimensions  out  of  which  scales  that  have    appear to be appropriate, with some cautions.
                been developed or can be developed can be applied.                  And  I  want  to  end  up  with  noting  some   of   these
                   Certainly  it  seems  to  me  that  with  regard  to  sick    cautions and then see  what  you  have  to  say  about  these
                people,  and  particularly  with  regard  to  some   of   the    kinds of things.


                82




          These  kinds  of  studies  -  you  have  heard  how  much  it       I think  you  said  that  it's  often  worth  measuring  dis-
       costs from Dr. Henderson. Well, going out and             inter-    ability  and  I   was   fascinated   that   you   talked   about
       viewing  people  or  sending   out   people   into   the   corn-    measuring,  family  disability,  not  just  patient  disability.
       rounity  to  ask  and  get  detailed  information          about    By  the  time  we  get  that  on  the   front   sheet   of   the
       activities  of  daily  living  is  more  expensive.  It's   more    medical record, we'll be quite far down the line.
       expensive  to generate, to reduce, process and              make       DR.   KELMAN:   Not   in   your   or   my   lifetime.   Well,
       available information from this source than it is          using    maybe yours.
       available  hospital  records  with  all  of  their  limitations.       DR.   CARPENTER:   It's   interesting   you    pointed    out
       And I want  to  mention  that  because  it's  a  consideration,.    that sometimes  it's  hard  to  understand  the  validitv  of  a
       we all do think about even today.                                   measure   of   death   unless    you    know    a    concomitant
          The other thing I  think  one  has  to  think  about  is  the    measure  of  disability.   Dr.   Stoneman   pointed   out   that
       practicality of obtaining  many  of  these  measures.  I  recall    probably   those   of   our   patients   whom   the   internists
       in  one  study  that  I  was  involved  in  we  were  concerned,    appear  to  have  saved  went  home   comotose   and   wet   the
       in  addition  to  getting  physical  and  social   measures   of    family bed for ten  years  before  they  finally  died.  And  so
       functioning,   with   getting    psychological    measures    of    it  is  necessary  to  measure  both  death  and  disability  to
       functioning.  And  this  involved  first  obtaining   and   then    understand  the  value  of  their   treatment.   By   the   way,
       sending  a  highly  qualified,  highly  trained  clinical   psy-    disability  on  discharge  was  the  same  for  both   physician
       chologist  into  nursing  homes  with  a   suitcase   which   he    groups.
       opened  up  and  then  did  his  testing   in   front   of   the       QUESTION:  As   you   were   speaking   of   different   ways
       patients.                                                           of measuring  health  status,  one  thing  struck  me.  I  think
          This  whole  apparatus  -  and  I  won't  even  get  involved    some  of.  these  measurements  have  to  be   reproducible   if
       in  terms  of   the   development   of   this   procedure,   ran    we are going to use them in evaluation.
       anywhere.ftom one to three hours.                                      In  evaluation  evidently  we  are  going  to  pick  them  up
          Well, I'm not  suggesting  that  these  very  detailed  kinds    at one point  in  time  and  then  later  on  pick  them  up  in
       of  measures  on  memory,  on  judgment  and  recall   are   the    order   to   evaluate   programs.   How   can   we    pick    up
       kind of thing  that  ought  to  be  done  routinely,  but  there    reproducible measurements?
       may   be   some   programs   where   this   kind   of    measure       DR.  KELMAN:  I   think   many,   if   not   all,   are   re-
       would  be   entirely   appropriate   and   may   be   the   most    producible. I think the  question  is  whether  it's  a  measure
       relevant  criteria  of  brenefit  for  some  kinds  of  patients    you  want  to  get  and  whether  it's  relevant  to  your  pro-
       and therefore  shouldn't  be  excluded.  But  it  is  expensive.    gram.
          Again  one  or   two   comments.   The   data,   of   course,       For  example,  it  is  not  difficult  to  ask  one  or  more
       considering again the source,  is  highly-  well,  I  was  going    times or of one  or  more  points  in  time,  not  difficult  to
       to  say  highly   reliable.   Relative   to   other   forms   of    get  from  the  patient  the  answer  to  a  question,  "How  do
       information  on  mortality  or  morbidity  it  is  no  more   or    you  feel?"  I  think  the  prior  question  is:  Do  you   want
       no  less  subject  to  problems  of  reliability  and   validity    that piece of information?
       than these  data  are,  although  the  problems  are  different.       It is  not,  I  think,  difficult.  Now,  some  of  the  con-
          And,  finally,  I  would  end  up  with   just   a   reminder    siderations  you  would  have   to   take   into   consideration
       that, as I  said  earlier,  the  further  one  moves  away  from    are: How stable is that feeling  state?  Is  this  going  to  be
       physical  and  biologic  measures  of  function,  the  more  the    something that he is telling  me  right  now  and  is  it  going
       actual  functioning  of  the  individual  and  the  patient   is    to be based on what  has  happened  to  him  in  the  past  five
       going  to  be  influenced  by  things  other   than   what   was    minutes, or is this a more  or  less  enduring  state  of  being
       done to him medically  or  what  his  biologic  status  is.  And    that I am concerned about?
       this  presents  a  problem  in  terms  of  evaluating   program-       One  of  the  things   in   the   program   may   have   been
       matic impact.                                                       directed towards altering  favorably  the  feeling  states,  the
          I  think  I'll  stop  there  and  ask  if   there   are   any    moods,  the  emotional  status,  whatever  you  want   to   call
       questions that  I  could  try  to  answer  or  any  points  that    it, of certain kinds  of  patients.  So  that  may  not  be  the
       you would like to have me try to elaborate on.                      most efficacious way of getting information.
                                                                              But  you  can  get   reproducible   information   by   asking
                                                                           those kinds of questions.
                                 Discussion                                   Now,  whether  they  are  the   kinds   of   questions   that
                                                                           relate to the kinds  of  information  you  are  seeking  is  the
          DR.    CARPENTER:    Thank    you    very    much,    Howard.    prior question.
                                                                                                                                         83





                   This is true also of measures of  social  function  and           and I think that is one of  the  parameters  you  have  to
                measures of activities of daily living which are a  little           watch pretty carefully.
                more enduring than the example I cited but I  really  used              DR.   KELMAN:   I   agree    completely.    And    it's    my
                that just to make a point.                                           impression that most  of  the  scales  you  get  into  with  the
                   DR.   WILLIAMSON:   Howard,   if   you   were    to    recom-     disability  measures,  whether  it's   vocational   or   occupa-
                mend  to  us  one  key  reference  in  the  literature  on   the     tional  or  activities  of  daily  living,  are  really   geared
                validity  and   reliability   of   disability   measures,   what     towards   estimating   how   dependent   or   how    independent
                reference would you recommend?                                       a  person  is  either  occupationally,  vocationally,   socially
                   DR. KELMAN: On the validity and reliability?                      )r physically.
                   DR.   WILLIAMSON:   This   question   of   looking   at   the        DR. RIKLI:  I'd  just  like  to  add  one  point  there,  and
                reliability  and  validity  or  usefulness   and   general   ap-     that is about disability. A man may be       missing  a  leg  or
                plicabihty  of  these  measures.  What   literature   could   be     rrdssing an eye  or  have  many  other  disabilities  which  are
                pulled out that  would  get  us  going  in  studying  this  more     really  compensated  for  and  are   not   really   of   serious
                thorougwy?                                                           concern  to  society  when  a  person  makes   the   adjustment.
                   DR.  KELMAN:  I   think   one   of   the   places   I   would     But if  they  are  unable  to  adjust  and  have  a  dependency,
                start  is  with-  I  forget  the  author-but  it  was  a   mono-     then they become a serious concern.
                graph   put   out   by   the   National   Center   for    Health        DR.    KELMAN:    Right.    And    your    comment    reminds
                Statistics.                                                          me  of  something,  namely  that  we  have  to   distinguish   I
                   DR. WILLIAMSON: Sullivan?                                         think  between  an  impairment  such  as   this   and   a   dis-
                   DR.  KELMAN:  Sullivan  I  think  the  name   is.   I   think     ability.  They  are  separate  things.  There  are  many  of  us
                that's  a  good  reference  to   start   out   with   not   only     who  function  with  a  whole  variety  of   impairments   quite
                because of  the  kinds  of  questions  lie  raises  and  how  he     well.
                tries  to  relate  disability  to  the  broader   questions   of        That  is,  if  I  were  an  engraver,  with   my   level   of
                health status, but  also  because  I  think  he  has  an  excel-     impaired  vision,  I  might  be   quite   disabled   occupation-
                lent bibliography.                                                   ally. But in terirs  of  the  kind  of  vocational  situation  I
                   I  think  the  article  by  Ellinson  in  the  Hatidbook   Of     am in now, I'm not at an.
                Medical  Sociology  on   sociomedical   measures   or   measure-        These are  other  problems,  and  this  is  part  of  what  I
                ment  problems   is   an   excellent   discussion   of   method-     was  trying  to  get  across  in  terms  of  the  point  I  made
                ologic problems.                                                     about  when  you  begin  to  move   away   from   the   biologic
                   DR. WILLIAMSON: Levine's?                                         functioning   of   individuals   to    estimating    how    they
                   DR.   KELMAN:   Right.   In   that   book.   I   think    you     function  in  social  terms  and  in  social   situations.   The
                would  do  well,  if  you   haven't   already,   to   write   to     biologic  becomes  less  influential.  Not  uninfluential,   but
                Murray Wylie and get some reprints from him.                         less influential.
                   And  I  think  with  that  a  person  would  be  well   armed        I'm   glad   you   raised   the   point   of   distinguishing
                and  well  acquainted  with  not  only  the  problems   of   the     measurement  of   disability   from   measurement   of   impair-
                application  of  these  kinds  of  measures  but  their   poten-     ment.   They   are   both   important,   but   they   are   dif-
                tial and actual utility.                                             ferent  kinds  of  things.  We  sometimes  tend  to  think  that
                   DR.   CARPENTER:   You   can   also   look   up   Kelman   in     when   we   measure   impairment   we   are    measuring    dis-
                the  literature.  That  will  get  you  a  long  way  down   the     ability, and vice versa, but'we are not.
                road.                                                                   DR.   CARPENTER:   It's   hard   to   get    the    diagnosis
                   DR.  RIKLI:  A  small  observation.  Those  five   D's   that     adequately on  the  front  sheet  of  the  chart  and  a  little
                you  attributed  to  Kerr  White  -  I  have   heard   them   on     easier I think to get survival  indicated  on  the  front  sheet
                many occasions  -  are  most  useful  in  taking  a  project  or     of  the  chart.  Are  there  any  obvious   measures   of   dis-
                a program and running down those five.                               ability  on  discharge  that  could  be  coded  on   the   front
                   And  as  you  talked  about  disability,  you  talked   about     sheet of a chart?
                the  independence.  It  seems  to  me  that  probably  a   sixth        DR.  KELMAN:  I  think  there  are   a   number   of   things
                "D"   might    be    dependency    -    financial    dependency,     that would be very useful  to  try  to  get  hi  some  standard-
                emotional   dependency   and   physical   dependency.   And    I     ized  fashion.  I  think  it  would  be  extremely   useful   to
                think  that's  probably  the  greatest  concern  of   people   -     know a  few  pieces  of  information.  (I  say  that  as  though
                when  a  parent  or  uncle  or   aunt   becomes   dependent   on     it's so easy  and  so  simple.)  The  extent  to  which  an  in-
                them   in   some   manner.   And   dependency   is   measurable,     dividual is  able  to  perform  certain  limited  activities  of


                84




      daily living-and  God  knows  why  on  any  hospital  record             DR.  CHARLES  LEWIS:  But  looking  at  institutions  to
      you can't find out how much  education  the  person  has.  I          examine the feasibility of recording information, I  think-
      wish we could get that in. And where  the  person  is  going          the real question is the one you  raised:  If  you  have  a
      to, more  explicitly  than  "discharged,  improved."  If  we          marvelous   language   which   is   somehow   or    another
      could  routinely   know   whether   the   person   is   returning     codable in  a  series  of  digits,  so  what?  How  will  it  be
      home or to some alternate living situation,  I  think  would          accepted? How will it be involved in  medical  records?  To
      be a way of starting out.                                             what  extent  will  it  actually  influence  patient  care,  or-
         Again,  if  you  can  do  that  in  some  kind  of   standard-     ganizational behavior, interorganizational behavior?
      ized  fashion,  fine.  But  to  have  the  ward  clerk  or  some-       But  I  think  that's  maybe   a   remote   question   because
      body  else,  you  know,  just  scribble  down  some   things   is     the real  issue  is  that  there  is  no  way  of  communicatin-
      not worth the bother.                                                 this between institutions, between patients.
         DR.  LEWIS:  I  was  looking  for   some   of   the   partici-       This  is  an  attempt  to  try  to  standardize  -   to   deal
      pants  in  a  four-center  contract  from  the  National   Center     with  Howard's  original  point  that   everybody   starts   out
      now, and I guess there isn't anyone here.                             by inventin- a new wheel.
         The   Harvard   center,   Western   Reserve,   Syracuse    and       DR.  KE  LMA  N:  There  was  an  attempt  -   some   of   you
      Johns  Hopkins,  have  a  contract  with  the   National   Center     may be familiar with it. I   haven't   heard    what    happened
      for  Health  Services  Research  to  develop   a   classification     to  it,  but  it  was  called   "rehabflitation   codes."   That
      system for patients that deals with three levels:                     effort  involved   a   number   of   advisory   committees   who
           The  problem  of  the   patient,   the   actual   management     for  years  tried  to   develo   a   common   way   of   codiii-
                                                                                                        p                                  t,
      of the patient.                                                       relevant  information  for  patients   in   rehabilitation   and
         9   Second,   the   problem   of   institutional   management,     related kinds of  programs,  institutions  and  facilities.  And
      in  terms  of  length  of  stay-the  usual  issues  that  an  or-     t  ey  deve  oped  reams  and  reams  of   material.   I   don't
      ganization or institution is concerned about.                         think it was ever used much by anybody.
         o  And  a  third  level  of  coding  which  has  to  do   with       I  don't  really  know  why  it  wasn't,  because  there  were
      interorganizational  needs  -   in   reality,   what   the   com.     many,  many  places  and  many,   many   people   and   advisory
      munity  has  to  furnish  patients  in  the   way   of   extended     committees   that   worked   on   development    of    it    and
      care facilities, etc,                                                 worked very hard, and a  lot  of  it  was  good.  Maybe  all  of
         The  'classification  is  in  the  early  stages,  first   de-     it was good.
      veloping  a  common  language,  and  is  designed  to   work   at       But  I  guess  there's  a  different  set  of   problems   in-
      several  levels   like   any   taxonomy   which   progresses   to     volved  in   developing   these   beautiful   codes   and   then
      deeper   levels   in   which   information   is   going   to   be     trying to take that and  translate  it  operationally  in  terms
      obviously less and less available.                                    of  some  ongoing  system  like  a  medical  care  system.   And
         This is  an  attempt  at  the  kind  of  classification  which     I really don't know what happened to it.
      would  lead  to  a  series  of  codes  some  place  on  a  record
      that  would  describe  a  functional  disability,  to  an  insti-
      tutional problem and interinstitutional problems.                      The.Relation of Process and End-Result. Evaluation
         And,  as  I  say,  I  don't  see  anybody  from  one  of   the                     CHARLES LEWIS, M.D.
      groups here.
         DR.   CARPENTER:   And    this    language    will    describe      I   want   to   approach   this   from   the   stance   of   an
      activities of daily living?                                           operator,   somebody   who   has   to   make   decisions   about
         DR. LEWIS: Part of it will. In  essence  it  looks  at  a  lot     evaluation  data  as  well  as  someone  who  is   supposed   to
      of the  kinds  of  things  Howard  has  been  talking  about.  It     be providing it.
      tries  to  take  into  consideration  all  of  this.  They   have      And  I  will  assume  at  the  beginning   that   we   evaluate
      been  reviewing  the  literature  trying   to   develop   between     things  in  order  to  change  things,  not  as  some  form   of
      four  institutions  a  common  language  so  -that   when   some-     self-amusement,  (w@ch  it   does   turn   out   to   be   some-
      body  says  they   are   impaired,   for   example,   in   mental     times),  but  in  order  to  provide  some  guidance  for  those
      status, they Will  now  know  it  is  coded,  in  terms  of  dis-     who  would  like  to  really  change  the  way  things  operate,
      oriented by place, by time, by person, etc.                           if they need changing.
         It poses a real problem in numerical taxonomy.                      Now,  I'd  like  to  restate  very,  very   simply   what   was
         MR.  SHAPIRO:  Well,   Chuck,   does   this   effort   contem-     said  more  eloquIently  this   morning.   Something   -   and   I
      plate major changes in the contents of hospital r         ecords?     have decided to call it a  condition,  not  a  problem,  not  an

                                                                                                                                          85






               event, and  not  an  in-put  -  just  a  condition  at  a  time        The second factor is the state of  the  art  of  evaluation
               zero, whatever time  you  care  to  choose  that  to  be  -  is    in general.
               usually measured in some kind of units.                                If we  present  those  who  are  coordinators  of  programs
                  And  the  units  are  hopefully  relevant  and  possible  of    with  end  results  which  say  "it  worked"  or   "your   pro-
               being  measured,  assessable,  and  hopefully  available.   And    gram is good" - that's all they want.
               I  think  we  would  like  these  to  be   valid,   replicable,        But if one  is  going  to  present  someone  with  informa-
               practical, and sensitive.                                          tion which is other than socially acceptable,  it's  useful  to
                  For  operators,  the  thing  we  are   currently   concerned    be able  to  tell  them  what  processes  went  wrong,  because
               with is that the condition needs altering,  or  else  there  is    this  provides  alternatives  for  strategies   in   terms   of
               a  question  of  it  being  altered.  After  looking  at   this    restructuring programs.
               condition   we   do   something,   not   just   anything    but        Donald  Campbell  and  others   have   talked   about   the
               something  specific,  and  that  the  thing  we  do  is.   also    problems  of   reforms   as   experiments,   and   the   social
               measurable or assessable.                                          legislation that has  been  enacted  to  create  social  change
                  Having   done   something,   a   whole   bunch   of    messy    and  why   evaluation   of   these   programs   has   been   so
               things happen that are  called  processes.  And  I  would  just    difficult.
               say that  one  man's  process  is  another  man's  end  result,        If you had  a  million  dollars  riding  on  a  program  in
               that  somewhere  in  here  people  may  choose  to   stop   and    which  it  was  announced  a  p?iofi  there  were   no   alter-
               say, "That's all I'm interested in."                               natives  to  success  except   through   this   approach,   you
                  And  this  is  particularly  true  I  think  in  looking  at    have  some  idea  of  why  individuals  resist  evaluation  (at
               continuing  education,  in  which  maybe  all  we  want  to  do    the risk of going out of business).
               is show they were sitting in the room.                                 The failure to specify  strategies,  alternate  strategies,
                  The  next  thing  we  may  decide  we'd  like  to  know   is    for experimental programs creates a problem.
               that they sat in the room and learned something.                       Perhaps  one  of  the  few  ways  we  can  deal  with  this
                  Then we'd like to  know  if  they  took  it  home  they  did    type  of  program  is  by  looking  at  the   nature   of   the
               something with it.                                                 processes that  went  on  while  reaching  an  end  result  and
                  Now, as I have  just  indicated  here,  most  of  the  times    presenting  these  data  to  those  who  have  to  make  policy
               when  we  are  concerned  with   process   we   are   concerned    decisions.
                                                                                      This is particularly important, I think, if  one  is  going
               about the number  of  things  that  are  done,  the  number  of    to  institutionalize  experiments   1  programs  -.  that   is,
               things  used,  the  nature  of  things  done  or  not  done  in    change  the  way  people  do  things.   The   transfer   of   a
               terms  of  quality.   Basically,   process   evaluators   count    program  which  seems  to  produce  results  into  a  different
               heads,  or  something,  or  the  use  of  things.  People   who    setting is  difficult.  Uffless  one  has  some  idea  of  what
               look  at  disability,  deaths,  and  so  forth,   as   in   the
                                                                      I .         went on.
               morning's discussions, are concerned with end resuits.                 Maybe this is related to  some  of  the  problems  in  dis-
                  The major point I'm going  to  make  -  I  hope  -  is  that    ability evaluation.
               it is difficult to  affect  change  without  doing  both,  that        I didn't stop and spend as  much  time  as  I  should  have
               end-result and  process  evaluation  need  to  be  carried  out    here  talking  about  the  measurement   of   "do   somethina."
               conjointly if one is going  to  be  an  applied  evaluator  and                                                               C,
               attempt to use results to redirect efforts.                        I  think  there  are  probably  more   "good"   programs   that
                                                                                  have  succeeded  because  the  "do  something"  was,  in  fact,
                  Let me  just  point  out  some  of  the  other  things  that    a  phantom  treatment  that   never   got   done   than   other
               by some of these terms I think relevant.                           kinds,  in  which  something  rather   dramatically   happened.
                  The  use  of  evaluation  data  depends  upon  two  sets  of        It's  very  useful  to  know  what  it  was  you  did  that
               factors:                                                           made a difference,  and  I  would  just  suggest  as  you  look
                  One,   organizational   factors.   Organizations   need   to    at the  literature  (as  one  moves  from  clinical  trials  of
               maintain  themselves.  They  need  to  perpetuate  the   status    drugs  where  we  are  sure  we  injected  "something",  to   a
               quo,  their  prestige   and   individual's   vested   interest.    program  in  which  we  install  a  new  kind  of  health  man-
               Evaluation basically  questions  the  reason  for  being  in  a    power)  that  we  really  don't  take  the  same  consideration
               certain  business,  and  doing  certain  things.  Fear  of  the    to  standardize  the  dosage,  the  blood  levels,  and   other
               consequences of change,  change  in  rank,  or  change  in  the    things  that  we  are   concerned   about.   The   process   of
               structure  of  an   organization   are   certairdy   sufficient    evaluation  begins   with   knowing   what   the   experimental
               causes to reject evaluation data.                                  treatment was.

               86



            I have selected out of the literature  eleven  papers  that     1. The first one was a  sub-study  that  came  out  of  ti-ic'
            are concerned with evaluation,  and  I'd  like  to  comment     national  hatothane  study  of  the  incidence   of   hepatic,
            on them and then talk about the  kind  of  evaluation  that     necrosis  with  halotliane.  This  was  a  report  of  iiisti.
            was done in 'each of  them  and  what  it  would  mean  (to     tutional  differences  in  post-op  death  rates.   Among   34
            ine) in terms of trying to implement these results.             hospitals, the end results (death  rates)  in  surgery  varied
            Let  me   begin,   though,   by   oversimplifying   certain     by a factor of 27.  They  were  subsequently  adjusted  for  a
            classes of end, and process, evaluations that these  papers     few things like  age  and  sex  and  other  things,  and  that
            represent.                                                      difference  is  resolved   to   10-fold.   There   were   sub-
            The first one and the simplest type  is  the  reporting  of     sequently  readjusted  for  severity  of  procedure,  and  the
            end results - end results in a group of patients or  insti-     difference collapsed to 3-fold. This  is  the  kind  of  study
            tutions  that  describe  the  fact  that  different  things     which says the  death  rates  in  hospitals  are  different  -
            happened.                                                       nothing else  -  and  if  we  age-adjust  and  do  some  other
            The second class or  type  of  paper  looks  at  variations     things that we know how  to  do,  they  are  still  different,
            in end results between groups or  amon-  -roups  or  within     but we don't really know why.
            groups, as a function of patient or doctor characteristics.     2.  The  second   paper   is   by   Leon   (Gordis)   on   the
            Type  three  is  similar  to  type  one  but   related   to     evaluation   of   a   program   for   preventing    adolescent
            process evaluation. These  describe  what  happened  -  the     pregnancy. This is  a  paper  that  looked  at  a  program  in
            processes that  were  carried  out  and  how  they  varied.     wl-lich  teenage  girls  who   were   sexually   active   were
            This ranges all the way from results of chart audits and  a     treated in  a  special  clinic  by  social  workers,  by  phy-
            whole bunch of things that are done  to  people  or  thincs     sicians, gynecologists, and  placed  on  oral  contraceptives.
            that are used.                                                  The design  then  was  to  follow  these  girls  to  determine
            A  little  more  complex,  and  fourth,  is  the  study  of     how  many  of  them  stayed  under   treatment   month   after
            processes  of  care  as  a  function  of  certain  provider     month.  About  50  percent  dropped   out   of   the   program
            characteristics. This is an attempt to  describe  the  dif-     within the year. The characteristics  of  those  young  ladies
            ferences in  the  way  processes  were  carried  out  as  a     who  did  not  stay  in  the  program  versus  those  who  did
            function of the professional's  background,  training,  and     were coi-npared.
            so on.                                                          3.   The   next   paper,   an    evaluation    of    community
            The fifth type, is a look at  both  process,  or  treatment     nursing services in the care of the  mentally  ill,  was  done
            done to somebody,  and  the  end  results  of  that  treat-     by  Tayback.  It  looked  at  what  happened  when   a   bunch
            ment, without  any  comparison  to  other  similar  events.     of  patients  discharged   from   mental   institutions   were
            In the sixth  class,  there  are  two  processes  -  one  I     provided services by visiting nurses in  the  home,  in  terms
            have listed as "C" for  control,  in  which  there  was  no     of  a  criterion  called  rehospitalization.  The  result  was
            treatment, and an examination  of  the  end  results  among     that  there  wasn't   any   difference   among   control   and
            two populations or groups with different  kinds  of  treat-     experimental patients.
            ments.                                                           The  paper  raises  some  interesting  questions  as  to  why
            I'm staying away, in this  discussion,  from  the  kind  of     there  wasn't  any  difference.  I  think  from  the  descrip-
            complex  experimental  designs  that  many  of   us   would     tion, I might  point  out  there  wasn't  any  standardization
            like to carry out and are  very  comfortable  with  in  the     of  treatment.  One  really  didn't  know   quite   what   was
            laboratory, i.e. cross-over, factoral designs. Because they     being  done  and  how  this   might   have   varied   or   how
            don't  come  along  very  often  in  the  business  we  are     certain  subgroups  of  women  might   have   had   a   better
            involved in.                                                    prognosis  than  others.  In   terms   of   looking   at   the
            There  are   some   other   kinds   of   quasi-experimental     probability  of  rehospitalization  as  a  function   of   the
            designs that are possible such as a time series observation     patient, this is another kind that  fits  in  second  category
            that  was  pointed  out  this  morning,   regression   dis-     also.
            continuity designs, etc.  I  refer  you  to  the  paper  by       4.  The  fourth  papers   comments   on   genetic   counsel-
            Donald Campbell in  the  American  Psychologist,  for  dis-     ing. And  if  any  of  you  know  any  other  studies  of  the
            cussion of these.                                               Iefficacy  of  genetic  counseling,  I'd   appreciate   knowing
            With   this    very    crude    and    perhaps    debatable     them. This is about the only one I have come by.
            classification I'd like to go over eleven papers. I  really       Famflies who had had one or more defective          children
            didn't choose these with any bias,  except  that  they  il-     for  whom  the  genetic  inheritance  patterns   were   known,
            lustrate these types of evaluation.                             were provided  counseling  services  (not  further  described)
                                                                                                                                         87





                and  then  followed  forward  for  a  period  of   time.   Ap-   cedures. It  looks  at  processes  as  a  function  of  certain
                proximately  60  percent  of  the  patients  went  ahead   and   variables in the structure of medical care.
                had another child.  It  would  suggest  about  40  percent  of      7. The next paper is by Thompson and his group            at
                this counseling, however it  was  done,  had  some  effect  on   Yale on end result measurements of the quality               of
                further childbearing.                                            obstetrical care in two U.S  'Air Force hospitals.
                   Here again there was  no  discussion  of  the  effects,  no      Thompson   looked   at   two   Air   Force   hospitals   and
                discussion of the characteristics of patients.  It  represents   perinatal  mortality  by  race,  and  found  out  that  in  one
                a  straight-forward  statement  that  so  many  children  were   hospital, the black perinatal  rate  was  higher,  but  in  the
                born   who   had   major   congenital   anomalies   or   minor   next hospital the white prematurity rate was higher.
                congenital   anomalies    to    families    who    had    been      He  went  back  and  looked  at  utilization  of   care   by
                counseled.                                                       trimester  of  pregnancy  and  found  out  that  all  of  these
                   5.  The  fifth  paper  presented  is  from  San   Francisco   ladies were using prenatal  care  rather  early.  It's  a  very
                data  on  the  neighborhood  clinics  for  a  more   effective   facinating paper  because  the  more  you  read  it,  the  more
                outpatient  treatment  of  tuberculosis.  This  was   prompted   you have trouble reconciling some of the results.
                by  some  observations  that  (in  San  Francisco)  about   80      8.  The  next  paper  measured  the   quality   of   medical
                percent  of  alcoholics,  (50  percent  of   blacks   and   20   care through vital statistics.  This  is  a  comparative  study
                percent  of  Chinese)  broke  their  appointments  to  the  TB   of  appendectomy  rates  in   the   hospital   regions   around
                outpatient clinic.                                               Rochester,  New  York.   There   were   large   variations   in
                  The  public  health  department  went  into  each  of  these   rates  at  which  appendectomies   were   performed.   And   no
                neighborhoods,  organized  clinics  with  the  help   of   the   relationship  was   found   between   rates   of   appendectomy
                local  citizenry.  The  compliance  rate   with   broken   ap-   and  deaths  due  to  appendicitis  -  an  example  of  looking
                pointments, sometimes  used  as  a  measure  of  satisfaction,   at a process, and the variations  in  process  as  they  relate
                dropped to about 5 to 10 percent.                                to an end result.
                  The  interesting  thing  about  the  paper  is  that  nobody      9.  The  next  study  of   comprehensive   outpatient   care
                reported  whether  or  not  there  were  any  readmissions  or   in rheumatoid arthritis  is  one  of  the  ones  that  deserves
                active cases of TB.                                              reading if you're going to read  any  of  these.  In  this  one
                  This  is  a  discussion   essentially   of   processes   and   Dr.  Katz  does  several  things.  He  defines  the   condition
                change in  processes  related  to  the  structure  of  a  pro-   that he's trying to deal  with.  He  measured  disability  with
                gram, which, oddly enough,  did  not  look  at  the  payoff  -   all   of   the   problems   that   Howard   Kelman    mentioned
                which  is whether  or not     'any   of   these    tuberculous   earlier this  morning.  He  describes  the  processes  of  care
                patients   complied   with   their   medications,   or    were   for  a  group  that  got  physical  t@erapy,  nursing,   public
                readmitted to hospitals.                                         health    nursing,    comprehensive    team    approach,    and
                  6.  The  next  paper  is  a  study  of  variations  in   the   describes  it  very  well.  He  measures  outcome,  significant
                incidence of  surgery.  This  was  a  study  which  looked  at   changes  in  disability,  as  a   result   of   applying   com-
                all  Blue  Cross  subscribers  in  the  state  of  Kansas  and   prehensive  care  for  ambulatory  patients   with   rheumatoid
                looked  at  the  incidence  of  certain   common   operations,   arthristis.
                T'&A,    appendectomy,    etc.,    in     various     economic      I 0. The next one is a  study  that  we  did  in  Kansas  on
                subregions  of  the  state,  defined  so  they'd   be   fairly   continuing  medical  education.   This   is   a   study   which
                homogeneous in nature.                                           basically  looked  at  the  tremendously   aggressive   program
                  The "Glover"  effect  or  variation  in  rates  for  tonsil-   in  continuing  education  that  had  been   mounted   at   the
                lectomy was  reconfirmed,  as  was  a  34  fold  variation  in   University  of  Kansas  for  over   30   years   with   circuit
                rates  for  appendectomy,  cholercystectomy,  and  a   variety   riders,  with  regional  courses  and  with   conferences   and
                of other  procedures.  The  rates  for  surgery  were  studied   seminars held at the medical school.
                as a function of  the  availability  of  sur-eons,  beds,  and      It was an  attempt to  look  at  the  participation  of  all
                general  physicians  in  the  area.   The   percent   of   the   physicians in  the state for each year  at  risk  over  a  ten-
                variance  of  these  rates  that  could   be   explained   was   year period.
                rather   phenomenal.   For   appendectomy,   70   percent   of      We took a   look at the predictors of  use,  as  a  function
                the  variation  could  be  explained  by  beds  and  surgeons.   of physician   characteristics,   and   found    among    other
                  It has some interesting implications,  but  it  doesn't  say   things that it's  related  to  being  near  a  regional  center
                anything  about  the  consequences  of  these  surgical   pro-   (having it avaflable); being a specialist, and being  a  recent

                88




       graduate, but not at  all  related  to  place  in  class  on    that  was  discovered  when  it  was  found  out  that   the
       graduation.                                                     deaths occurred after discharge.
          II. The last is  from  the  nurse  clinic  study  at  the       Let me tell you  about  one  that  I'm  willing  to  talk
       University  of  Kansas  by  Barbara   Resnick   and   myself    about, and it's a  negative  one,  about  how  process  in.-
       which looked  at  activities,  events  and  the  outcome  of    formation,  and  perhaps  some   outcome   data   influenced
       ambulatory care in  which  a  population  of  patients  with    program  planning  in  the  Kansas  Regional  Medical   Pro-
       defined   chronic   illness   were   previously    examined,    gram. Perhaps  we  can  get  a  postscript  from  Bob  Brown
       described  in  fair  detail,  were   randomized   into   two    who is now in charge of the program.
       groups.  One  went  back  to  medicine  clinic,  the   other       In  1967,  the  very  start  of  the  program,  we   like
       received care by nurse practitioners.                           everybody else  were  trying  to  get  people  involved  and
          The critical  incident  technique  was  used  to  try  to    trying to  convince  everybody  it  was  their  program.  No
       measure some of the activities of  the  nurse  clinic,  some    one believed this.
       of  the  things  that  John  Williamson  and  Paul  Sanazaro       We were always saying, "If you just  bring  us  projects,
       piloted. We  looked  at  outcomes;  death  rates,  in  which    we'll help  you  get  them  funded."  And  they  brought  us
       there was no difference;  the  level  of  disease,  no  dif-    one from an area in  Kansas  that  has  some  problems  with
       ference. There were significant  differences  in  disability    economic  growth,  where  the  population   was   relatively
       rates at the end  of  one  year  of  care  under  these  two    aged, the physicians likewise,  and  no  younger  physicians
       systems; the nurses' patients were far less disabled.  There    were  goinal  and  there   were   lots   of   rehabilitation
       were significant differences  in  discomfort  and  satisfac-    problems.
       tion levels. This paper attempts to look  at  processes  and       Some of the people in that  area  said,  "We  want  funds
       outcomes.                                                       to train assistants in occupational therapy and  PT  assist-
          If I  were  presented  with  the  data  on  institutional    ants, because we have a junior college,  and  we  can  train
       post-op  death  rates  I  would  say,  "I  don't  understand    these people,  and  then  they'll  provide  our  rehabilita-
       why our hospital is  either  so  good  or  so  bad."  But  I    tion."
       don't  have  any  answers,  and  if  we  were  good  I'd  be       We said, "Fine. We need some data to support it."
       happy.                                                             We had done a survey and were  quite  aware  this  was  a
          I think that regarding the second  paper,  evaluation  of    very disabled population.
       the  program  for  preventing  adolescent   pregnancies,   I       We also took a look  at  the  occupational  and  physical
       would say, "This looks good, but I really  can't  tell  what    therapy facilities in hospitals in  this  nine  county  area
       you're doing to these young ladies, and I really can't  tell    and found without  exception  all  of  them  were  operating
       if anything is happening. Therefore, I  think  you'd  better    at less than 50 percent capacity.
       try to measure what you're doing to  them  a  little  better       We interviewed a  sample  of  about  50  percent  of  all
       if you want me to pick  up  the  tab  for  this  kind  of  a    practicing physicians  in  this  nine  county  area  and  we
       program after the grant support wears off."                             C,
          For  the  third  paper,  an   evaluation   of   community    sent our young ladies to them, and they asked:
       nursing services, I think  the  comments  would  be  as  for       "Have  you  seen  anybody  who  needed  occupational   or
       the previous study.                                             physical therapy?"          little  probe  to  explain  what
                                                                          And then there was a
          This  comment  on  genetic  counseling.  I   don't   know    occupational therapy was.
       what   you   can   say   when   you're    confronted    with       The next question was, "Did they get care?"
       information that says patients don't do  what  doctors  tell       The final one was, "Do  you  think  we  need  riiore?"  -
       them to do except begin to deal with  their  patients  in  a    to which the answer was always yes.
       little more sophisticated way.                                     When we took this data back,  we  were  able  to  say  to
          For  the  TB  clinic  study,  this  looks  good  on   the    the  people,  "Look,  you  have  lots   of   problems,   end
       statistical sheets,  but  did  anybody  get  TB?  Again  the    results that need to be changed;  but  you  have  facilities
       lack of outcome data creates major problems.                    that  are  being  underused.  There  are  occupational   and
          This  morning  when  Bob   said,   "Does   anybody   here    physical therapists  who  are  going  to  leave  their  jobs
       have  end  results  that  influenced  decisionmaking?",  Sam    because they don't have any work to do."
       Shapiro  talked  about   mammography,   and   someone   else       If we look  at  who  creates  demand  for  rehabilitation
       mentioned  dental  cytology.  And  John  talked  about   the    services  (doctors)  and  talked  to  them,  we  found  that
       heart failure study and the creation of a  follow-up  clinic    they (the doctors) were not  aware  of  the  need  for  this
                                                                                                                                  89





                 service  and  had   identified   patients   for   whom   these    certainly the end result standing by  itself  in  conjunctior
                 services should be prescribed.                                    with the particular kind of setting in which it  was  carrie@
                   We   didn't   try   to   make   any   interpretations.    We    out  has  been  of,  an  enormous  importance  in   assessing
                 presented  this  to   influential   citizens   whose   comment    the impact by prepaid  group  practice's  impact  on  health,
                 was, "It looks  like  we  have  a  job  to  do  with  our  own       So while I want to repeat that,  of  course,  process  i,;
                 doctors."                                                         terribly important, there  are  on  occasion  very  important
                   I don't know, Bob,  whether  there  is  still  pressure  for    practical  considerations  that   make   it   extraordinarily
                 this. But I think that in  one  case  we  were  able  to  show    difficult if not impossible to get at process.
                 that by looking  at  the  processes,  that  is,  why  patients       The reverse is  true  too.  An  advance  in  understanding
                 who  need  care  do  not  get  it,  we  were  able  to   avoid    process with some  implied  benefits  from  process  with  nc
                 spending some money at least at that time.                        ability to get at the end result is also worthwhile.
                   I have  asked  some  of  the  experts  around  the  room  to       DR. LEWIS:  I  think  that's  an  excellent  example.  And
                 give  me  some  feedback  on  some  questions  that   I   have    the  question  has  always  occurred  in  my  mind:  If  this
                 raised. I think I'll start by asking  Sam  Shapiro.  It  seems    sort of care system is related to these  kinds  of  outcomes,
                 to me that one of the reasons  you  have  been  so  effective,    then why have the, let's say, perinatal  and  infant  mortal-
                 Sam,  in  influencing  programs  is  that  you   really   have    ity  social  gradients  in  the  United  Kingdom   not   been
                 been looking at end  results,  but  also  describing  to  your    totally  eradicated  by  the  emergence   of   the   national
                 own  group  the  processes  that  they   were   pursuing   and    health system?
                 carrying them right along with you.                                  MR.  SHAPIRO:  Do  you  want  to  get  into  a  discussion
                                                                                   of that?
                                                                                      DR. LEWIS: No, sir.
                                         Discussion                                   MR.  SHAPIRO:  Look,  in  a  system  like  HIP,  we   know
                                                                                   that there are very important gradients by  social  class.  I
                   MR.  SHAPIRO:  Yes.  Well,   Chuck,   I   almost   have   to    don't want to get into that issue because I  think  it  openq
                 say "of course."                                                  up a new, highly complicated issue.
                   The influence of an  end  result  observation  is  going  to       DR. KELMAN:  Well,  I  would  like  to  go  a  little  bit
                 be very heavily affected by  the  ability  to  understand  the    further and reject if I can, just for  the  sake  of  a  con.
                 process  by  which  you  achieve  the  end  result,   and   as    troversy,  your  emphasis  on  process  evaluation.  I'm  not
                 much attention has to be paid  to  the  issue  of  process  as    against it. Let me say that  like  everybody  else,  I'm  for
                 the end result.                                                   motherhood and  all  of  that.  No,  these  days  you're  not
                   The only  reservation  that  I  would  have  is  that  there    supposed  to  be  for  motherhood.   I'm   not   opposed   to
                 are  occasions  when  it  becomes  incredibly   difficult   to    process  evaluation.  However,  I  think,  Chuck,   that   at
                 tease out of the situation  anything  but  very,  very  global    least as I look over much of the evaluation  literature,  I'm
                 descriptive  information  about  process.  But  yet  the   end    struck  by  the  fact  that  we  have   many   more   overall
                 r                              irm one.  And  I  have  a  very    de
                 esult in itself can be a very f'                                     scriptions of program  and  process  and  visits  than  we
                 specific situation in mind.                                       have end-result evaluations.  As  I  look  over  the  process
                   Some time  ago  we  looked  at  the  question  of  perinatal    kinds of things - and this may be strictly  personal,  but  I
                 mortality and prematurity in HIP  in  contrast  to  the  rates    don't think it is -  they  raise  no  questions  in  my  mind
                 amon-  patients  of  private  physicians  in   the   community    about  program.  However,  when  I  look  at  outcome   eval.
                      Z,                                                           uations with or  without  process,  they  at  least  raise  a
                 and did all the  necessary  standardizing.  We  came  up  with    question  and  would  give  me  some  pause  about  programs.
                 a finding of lower mortality and prematurity in HIP.
                   And the next question  we  raised  was:  What  is  there  in       Now, I don't agree with the  kind  of  response  that  you
                 HIP that produces this type of  result;  in  other  words  can    made to the first study - that if it's  good,  fine,  and  if
                 we identify the  process  of  care  responsible,  as  well  as    it's no good, let's forget  about  it.  I  don't  think  that
                 other factors?                                                    would  be  an  appropriate   response   to   outcome   result
                   But, it  was  just  not  possible  for  us  to  examine  the    where you may not know the process or channel.
                 process by which  people  received  their  prenatal  care  and       I can give you  an  example  of  a  study  we're  involved
                 the  other  circumstances  in  the  process  of  medical  care    in  where  the  outcome  was  negative.  We   had   excellent
                 that mic-,ht have influenced this result. I  think  the  whole    descriptive  material  on  the  process.  Nobody   paid   any
                 cause  of  reducing   infant   mortality   would   have   been    attention to  it  because  it  was  a  negative  finding.  So
                 advanced if we had been  able  to  get  at  the  process,  but    that's one point.

                 90




         The  second  point  I  would  make  has  to  do  with   the       tant separate  question,  and  I  wouldn't  treat  them  as
         utility of evaluation. I think  that  obviously  there  are       necessarily intertwined.
         factors aside from  the  presence  or  absence  of  process       The  other  thing  is  that   my   own   -hard   evaluation
         information   that   would   make   the    acceptance    or       experience - I mean in terms  of  doing  long-term  studies
         rejection of an outcome result affect its  acceptance.  It'       - has been in mental health, which is  a  little  different
         there is no question to begin with about  the  program,  if       from a lot of other studies.
         everybody is sure that is the only thing  that  can  affect       But  we  did  a  study  where  we  were  looking   at   re-
         family planning, if this is the only alternative to patient       habilitation  of  chronic  VA  patients  with  control   in
         neglect, then I submit this is not a question to be studied       an experimental ward  and  reached  a  conclusion  on  most
         or to be evaluated and an evaluation is  strictly  eyewash.       of our variables that the experimental ward  was  a  little
         It  would  seem  to  me  that  what  is  really  wanted  is       better and on one variable it was worse.  And  in  a  sense
         documentation of the efficacy of  what  people's  faith  is       that was hard, you know. I mean the data  was  as  good  as
         in something, and I  submit  this  is  not  an  appropriate       you ever get in psychiatry, which is a little weak.
         condition for evaluation of either process or outcome.            But then  I  think  the  creative  part  of  this  in  some
         DR.  LEWIS:  Let  me  respond  and  say  that  when  I  was       sense came in a bunch of us  sitting  around  the  table  -
         making comments about  these  papers,  I  hope  you  didn't       by  a  "bunch"  this  included  some  patients   too,   in-
         lose the fact that I have been in and out of  character  in       cidentauy - and trying to  figure  out,  "Well,  gee  whiz,
         this discussion, one of which is a  political  animal  con-       we thought we were  going  to  get  big  differences."  And
         cerned with getting things done and trying  to  keep  peace       yet we were only getting very small differences.
         and run an organization.                                          What  was  the  process?  And,   furthermore,   what   were
         And  mai@f6at's  what  all  this  is   about   -   interor-       processes that didn't exist in either ward that might  have
         ganizational conflict and the ways one deals with it  using       been instituted that one might want  to  carry  forward  in
         evaluation information.                                           further experimentation?
         It seems to  me  that  the  majority  of  people  who  want       That's a very soft set of procedures.  I  think  it's  very
         to evaluate something, Howard,  come  at  it  the  way  you       important that this be done.
         just said: "We have a good thing. Wouldn't it  be  nice  to       I agree with  one  of  the  comments  that  was  made  that
         show it?"                                                         there's a great tendency to get  so  embroiled  in  process
         DR. KELMAN: "For you to prove it."                                because outcomes tend to  b'e  more  difficult  to   measure,
         DR. LEWIS: "For you to prove it."        I   think    some-       that you end  up  patting  yourself  on  the  back  as  the
         times the most fascinating opportunity   for     evaluation       process looks pretty good.
         comes  serendipitously  that  way.  And  you  can  say  "We       DR. LEWIS: Let me restate. I  have  tried  to  say  that  I
         don't  do  that  kind  of  evaluation,"  wherever  you  are       think  both  have  to  be  done  whenever  possible  -  but
         locally enshrined, or you  can  say,  "Okay,  buddy,  we'll       there are circumstances in which  only  one  or  the  other
         have a go at it but let's be prepared  to  take  the  worst       can  be  done  and  appropriate  circumstances  when  maybe
         answer you are prepared to hear."                                 only one or the other should be done.
         It  seems  to  me  that  evaluation  almost  could  not  be       But I don't think there is  such  a  thing  as  process  or
         separated - just a personal opinion -  from  the  political       end results. And this gets to be an ideology, and it really
         and the ethical  context  in  which  it  is  performed  and       breaks  down  between   the   denominator   and   numerator
         without the consequences  to  those  who  are  involved  in       people  in  the  world,  those  who  are  concerned   about
         it. That may  be  a  little  more  philosophical  than  I'm       groups and  don't  give  a  damn  about  cases,  and  those
         supposed to be.                                                   who are only interested in  what  happens  with  the  case.
         DR.  Fox:   Two   comments.   I   agree   with   what   you       And these two subcultures have  always  existed.
         say. However, I think  that  one  must  separate  two  very       MR.   SHAPIRO:   Present    company    excepted.
         important issues. One is  the  bureaucratic  and  political       DR.  LEWIS:  I  don't  want  to-  I'll  run   up a flag  in
         pressures to prevent good evaluation.                             rwnute. But I think, quite honestly, this  is  one  of  the
         Now, that is a very important  product.  In  fact,  I  tend       problems in trying to diffuse this issue of  what  are  you
         to  believe  the  primary  reason   why   good   evaluation       going to do, because it really is related to personal orien-
         doesn't take  place  is  more  for  that  reason  than  the       tations about how you see care.
         reason that technology doesn't exist.                             DR  CARPENTER:  Dr.  Brown,  there  is  a  luU   here.   Do
         The   second   aspect   though,   the   relation    between       you want to give us that followup? Are  they  still  trying
         process and outcome studies, is itself  a  terribly  impor-       to train occupational therapists in way-out Kansas?
                                                                                                                                     9 1





                     DR. BROWN (Coordinator, Kansas Regional  Medical                   It was the most elementary kind of analysis which w(
                  Program): Well, it's a very complicated thing, and there          are an used to.
                  has  been  a  great  deal  of  study  of  the  situation.  It's       And   the   conclusion   of   the   meeting,   which   was-
                  essentially where it was at that time.                            When,you  think  about  it,  a  year  ago  we   didn't   reaII5
                     Another  similar  thing,  however,  Chuck,  having   to   do   know as  much  then  as  we  do  now.  The  conclusion  of  th(
                  with  changing  conditions.  It's   the   phenomenon   we   see   meeting  was  that  we  had  to  get  a  handle  on   ways   tc
                  with  the  home  health  care  service.  If  the  nurse   makes   describe  Regional  Medical  Programs,   from   my   point   oi
                  rounds in a  hospital  with  the  physicians,  she  builds  her   view, that  would  be  able  to  focus  in  on  what  kinds  ol
                  clientele for the visiting nurse association very  rapidly.  If   transformations  were  taking  place   in   the   health   car(
                  she is  at  headquarters  and  doesn't  go  into  the  hospital   system through Regional Medical Programs.
                  and make her own, she  doesn't  get  referrals,  which  is  the       You know - big deal - that's the conclusion.
                  same  -  which  has  to  do  with  awareness   of   physicians,       Well, we haven't really got  any  further  than  that,  and
                  you  know,  of  whether  everything   is   really   lovely   or   yet  I  feel  when  I  go  looking  at  the  budget  su@missior
                  where it isn't.                                                   when it comes  in  to  me  and  I  have  to  make  some  recom.
                     The  same  with  the  PT.  Since   they   don't   know   and   mendations  that  I  can't  really  justify   Regional   Medic@
                  have  personal  experience,  they   really   think   everything   Programs budget just  on  terms  of  additional  trainees  thii
                  must be all right and they really don't need it.                  year   or   whatever.   That's   not   really   what    Region@
                     It's  a  complicated  problem  hooked  up  with  our   whole   Medical Programs is about any more.
                  educational process  in  the  state.  So  they  haven't  really       And  I  don't  know  what  kind  of  indicators   to   use,
                  made any progress.                                                This is a tough question.
                     DR.   CARPENTER:   It    was    effective    evaluation    I       DR.  CHAR  LES  LEWIS:  To  drop   back   and   say   some.
                  gather.                                                           thing here since I'm  out  of  the  RMP  business,  isn't  this
                     DR.  LEWIS:  We   didn't   spend   some   of   Dr.   Brown's   the  whole  problem  since  1966,  that  the  RMP   was   based
                  money anyway'.                                                    on  a  promissory  note  which  could   never   be   delivered,
                     DR. BROWN: They still want it.                                 which was really the  elimination  of  heart  disease,  cancer,
                     DR.   CARPENTER:   Well,   could   we    get    some    dis-   and  stroke,  and  some  of  us  had  a  strong  feeling   that
                  cussion  around  the  question,  "should  end-result   analysis   besides providing "improving  the  care  of  the  patients"  it
                  be  undertaken  by  every  region  funding  a   coronary   care   was  really  about  regionalization,  and   the   establishment
                  program?"                                                         of  relationships,  and  the  introduction  of  change   within
                     MRS.   BLAXALL   (Budget   Examiner,    Office    of    Man-   the  system  which  occurs  only  under   certain   conditions.
                  agement and Budget):  I  don't  know  if  we  want  to  specif-       It sure helps to have a little  money.  It  helps  to  have
                  ically limit it to that. But  a  year  ago  we  had  a  session - some doctors who are hurting.
                  with  Pete  Peterson  and  Karl   Yordy   and   a   couple   of       I think it's fascinating  that  we  have  focused  most  of
                  people -  the  Assistant  Secretary  for  Planning  and  Evalu-   our  attention  on  university  medical  centers,   which   are
                  ation  -  and  Don  Schon  was  there  and  a   couple   others   about  the  last  things  in  the  world  that  are  going   to
                  from his firm.                                                    change  because  of  the  density  of  prestige   and   popula-
                                                                                    tion.
                     Ane,  the  whole  point  of  the  meeting  was  to  try  and       I think if one  really  wants  to  see  innovation  in  the
                  get  a  handle  on  the  kinds  of  evaluation  criteria,   in-   medical  care  system  today  you  go  to   the   small   towns
                  dicators, whatever you wanted  to  call  it,  that  the  Bureau   any  place  in  the  country  and  you   find   nurse   practi-
                  of the Budget  might  use  not  so  much  in  evaluating  in  a   tioners  and  physicians'  assistants  and  i-nergers  of  hos-
                  hard sense but perhaps even  describing,  the  process  of  the   pitals  and  all  sorts  of  interesting  things  that   aren't
                  activities  of  Regional  Medical  Programs   in   the   budget   making the New York Times.
                  appendix, for example.
                     We were using  such  things  as  the  process  indicators  -       But I suspect if one were going  to  invest  a  little  RMP
                  how  many   participants   in   the   training   program,   how   cash, one could very  easily  facilitate  regionalization  out-
                  i-nany  regions  were  operational,  just,   you   know,   just   side  of  those  sorts  of   procrustean   things   that   have
                  indicators, nothing  that  really  explained  anything  related   probably already died  but  the  message  just  hasn't  got  to
                  to  Schon's  systems   transformation   model,   not@ing   that   the brain yet.
                  gave  any  flavor  of  Regional   Medical   Programs   in   the       MRS.  BLAXALL:   That's   right.   I   agree   with'   your
                  description.                                                      statement.

                  92




        DR@ LEWIS: A lot of people don't.                             measurable.  The  diversity  -  the  major  strength  of  the
        MRS.  BLAXALL:   But   it   doesn't   help   me   in   the    law's pert-nissiveness toward local innovation   -      makes
     question I  have.  This  is  a  tough  question  we're  still    for such difficulty of expression that  it  now  becomes  the
     working  on.  For  example,  does  that   mean   instead   of    bane  of  the  evaluator's  existence.  Having  no   national
     using  the  old  indicators  that  we  should  focus  in   on    decision that a priority, for example, for coronary  care  is
     anecdotal elements?                                              acceptable, he has less clear  evidence  as  to  whether  his
        DR.  LEWIS:  No,  there  are  end  results  that  can   be    Region has placed significant priority on such care.
     measured I would assume.                                           DR. LEWIS: I  think  if  your  programs  had  written  real
        If you would like to talk about the availability  of  care    objectives and not statements  of  vague  goals,  they  might
     for populations and the  provision  of  care  to  populations    have been evaluatable. And  it's  like  teaching,  you  know.
     that don't have any care, as a  byproduct  of  RMP,  I  think    If you just  tell  them  what  you  want,  which  we  usually
     that  can  be  measured  -   providing   that's   what   your    do,  it's  a  mess.  Writing  educational  behavioral  change
     objectives were.                                                 objectives is a very difficult job.
        But  there  have  never  been  any  objectives  except  to      DR.  HASTINGS:  It  occurs  to  me  maybe   we   have   got
     "improve  the  quality  of  care  for  patients  with   heart    a  new  definition  of  what  RMP  is  really  about.  If  we
     disease, cancer, and stroke" - starting  at  where  the  care    make the  assumption  that  RMP's  real  business  is  social
     was probably the best.                                           change, if we  are  supposed  to  be  changing  things,  then
        .MRS.  BLAXALL:  Does  this  get  back  to   the   question   perhaps we  should  shift  our  statement  of  what  our  ob-
     then that I hear when  I  go  around  and  talk  to  some  of    jectives   are   from   disease-related,    medically-related
     the  regions,  "Who's  making  the   objectives   for   RMP?"    criteria as listed in each of these articles,  as  enumerated
     Washington  or  our  local  RAG?  Is   that   the   kind   of    in each of  these  articles  that  you  just  discussed,  and
     question that you're getting towards?                            frankly say that we're in the business  -  that  we're  in  a
        Because if there aren't any  concrete  objectives  at  the    political business, an  organizational  business  instead  of
     national level, which is what  I  suppose  I  have  to  worry    being in a task-related business, that  we're  in  the  busi-
     about, then -                                                    ness of changing a system.
          DR.  LEWIS:   I   think   RMP   when   it   emerged   in      And if we define ourselves that  way,  then  it's  possible
     1966, for some  of  us  that  really  got  seduced  into  the    to state objectives that one  can  measure,  different  kinds
                ocess  without  knowing  what  is  going  on   and    of objectives that people have tried to measure.
     planning pr
     found  ourselves  operational  before  we  really  knew  what      But  if  that's  what  we  are  about,  maybe  that's  what
     was going on, we had been at  that  time  fascinated  by  the    we should be doing.
     fact that this was a program in search  of  objectives,  that      DR.  LEWIS:  It  would  have  been  nice  if  the  original
     there was an  enormous  amount  of  money  to  be  spent  for    law hadn't said in it as long as it  doesn't  interfere  with
     doing something, but  no  one  ever  defined  from  hierarch-    current patterns of  practice.
     ical quarters up there what  was  expected  of  regions,  and      DR.    CARPENTER:    But    interference     and     change
     regions grew  depending  upon,  essentially,  the  philosophy    aren't the same.
     of the coordinator or the parent institution.                      DR. HENDERSON: That's right.
        And  at  that  time  I  think  many   of   us   felt   the      DR. ST 0 N E M A N: I think there is a real  gap  that  has
     taxonomy    of    FW.    There     were     hardware-oriented    developed in this conference. I think  it's  been  there  all
     regions   and   software-oriented.   There   were    disease-    the time. I think Dr. Lewis alluded to it.  It  concerns  me.
     oriented  and   there   were   people-oriented.   They   were    I'm  sure  it  concerns  many  other  program   coordinators.
     centralized  and  there   were   decentralized.   They   were      I think a lot of us    were  seduced  into   RMP   by   the
     clearly  determinable  by  the  nature  of  the  people   in-    bright hope of local initiative  and  local  decision  making
     volved in the original programs.                                 and  system  building   within.the context of the  law  as  it
        I do  not  know  whether  it  has  changed  or  not.  This    was written, with perhaps a few  liberties  with  the  inter-
     was  the  equivalent  of  the  identity  crisis  which  over-    ference clause.
     whelms the teachers of preventive medicine annually.               . But we did  develop  regional  advisory  groups.  We  did
        DR.   CARPENTER:   You   know,   it's   interesting   that    develop systems. We  did  spend  a  couple  of  years  teach-
     now  we  are  stuck  with  really  so  many  objectives  that    ing them what the law says  and  what  it's  all  about.  And
     there are  people  who  say  we  don't  have  any.  Each  in-    we did do this on the thesis that  unless  we  put  a  system
     dividual region has a large number  of  objectives,  some  of    together  that  could  work  together  we  were  never  going
     which  are  immeasurable,   some   of   which,   tho.ugh,   are  to be able to move  the  system  in  any  effective  way.  We
                                                                                                                                 93





                  have  begun  to  make  some   progress   toward   doing   that,        This may be just a time lapse. But I  do  hot  think  the
                  but we aren't there  yet.  I  don't  know  all  about  all  the     you  should  say  or  anybody  should  say  that   we   cannc
                  other regions. I know we're not there yet.                          do it. Given  enough  money  and  the  proper  input  it.  ea
                     The  law  is  being  renewed.  It's  written  by   Congress.     be done. But  it  cannot  be  done  except  by  coflaboratio.
                  It's  still  virtually  the  same  language  except  for   some     between  many   kinds   of   experts   with   background   an
                  kidney  wording  and  a  few  other  minor  changes.  And   yet     training in the sciences needed for the purpose.
                  the  Bureau  of  the  Budget  and  others  in  Washington   are        DR.  STONEMAN:  I  know,   but   given   the   fact   tha
                  coming through with a clarion  call  that  we're  going  to  be     each region is doing its own thing, 1:f'you will, even  give@
                  judged  on  whether  we're  agents   of   social   change   and     the kinds of that you describe-and I  listened  very  closer,
                  whether  we  can  materially,  with  the   dollars   we   have,     this morning, very interestedly-at  $200,000  for  the  firs'
                  affect the health status of the nation very soon.                   year how long with  that  kind  of  a  data  base.would  tha'
                     Now, we spent  all  day  finding  out  nobody  can  tell  us     regional  medical  program  have   to   go   with   operation
                  how to measure  health  status  to  begin  with.  So  we  can't     activities directed  toward  the  soft  spots  and  gaps  tha
                  evaluate   that   pursuit   except   in   individual    program     you  identify  and  develop  before   you   can   come   bacl
                  activities, and that's  out.  We're  not  supposed  to  measure     with a  continual  status  evaluation  that  will  answer  th
                  activities  as  much  as  we  do  broad  program.  The   people     question that she asked-for one region?
                  back  home  still  think  we're.  working  under   Public   Law        DR.  HENDERSON:  I  can   in   part   answer   your   que,@
                  89-239 and renewals.                                                tion. I cannot give you a time  limit.  But  I  can  tell  you
                     Now, it seems  to  me  that  there  is  an  obvious  question    problem  about  the  whole  program  that   I   think   extend
                  here that I  hope  will  be  addressed  before  the  meeting  is    this  time.  Because  of  the  insecurity  of  funding,   fron
                  over.  I  don't  think  we  can  do  what  we  have  been  asked    year  to  year,  our  unit  has   no   full-time   professions
                  to do until we do  what  we  set  out  to  do  -  put  a  system    person.  No  one  with  enough  epidemiological   and   statis
                  together. And I  don't  think  we  can  do  it  by  Fiat  within    tical experience  to  orga@ze  this  kind  of  center  can  a,
                  the  next  four  months  or   within   the   next   12   months,    that stage in  their  career  afford  to  go  full-time  on  @
                  probably not in less than several years.                            program  without  surety  of  continuity  and  funding.  So  i
                     A  d  this  comes  back  to  the  question  the  young   lady    the program had a more  stable  base,  it  could  be  done  ii
                  asked  about  -  what  do  we   put   down   to   justify   your    much  shorter  time  because  you  would   get   people   work
                  existence'! I don't  think  we're  going  to  with  $94  million    ing at the job full time.  The  very  nature  of  the  progran
                  this  year  produce  enough   product   in   additional   health    is extending the length of time  it  takes  to  do  evaluation
                  care  delivered  to  amount  to  a  minuscule  fragment  of  the       DR.  LEWIS:  I  think  just  to  reintroduce  Buck  Rikli'@
                  total systems production.                                           question as we have come full  circle,  it's  whether  or  not
                     Maybe  we're  going  to  produce  a  process  that  can   put    the kind of data that we  are  talking  about  will  influence
                  us in a position  to  do  something  about  that,  but  I  can't    planning and operation.
                  give you much more justification than that.                            MR.  SHAPIRO:  I  don't  see  how  you  can   answer   thal
                     DR.  HENDERSON:  I  want  to   just   try   to   remove   one    question-in a kind  of  global  way  any  more  than  I  could
                  misconception I think I heard.                                      possibly grapple with the global  way  of  stating  the  issue
                                                                                      of  changing  medical  care  systems.   You   can   think   ir
                     I   would   not   say   we   cannot   get   measurements   of    terms  of  a  change  of  medical  care  systems  involving  a
                  health status. I say we can.  I  tried  to  say  that  it  is  a    total approach. This is a  $65  billion-a-year  industry.  And
                  difficult task and it takes experts  in  many  fields  to  apply    anybody   who   thinks   that   RMP   is   going   to   change
                  their knowledge and do it efficiently.                              medical care  systems  in  a  very  fundamental  and  decisive
                     I  think  you  have  seen  that.   There   are   experts   in    way just doesn't  know  what's  going  on.  It's  unthinkable.
                  several  kinds  of  measurements  here  today.   We   have   all       But you could define  medical  care  systems  in  clusters,
                  tried to say that it takes a lot of effort, a lot of skill,  and    in smaller units, in  a  dimension  which  you  can  begin  to
                  a lot of  skilled  personnel  focusing  on  doing  the  specific    grapple with.
                  kinds  of  evaluation.  I  do  not  think  the  RMPs  have   had       I  hate  to  come  back  to  our  own  experience  and  our
                  people  with  the  right  kinds  of  expertise  in  their   pro-    own  aspirations,  but  the  program  that  I  was  describing
                  grams  to  start  off  with-for  good   reason.   The   majority    this morning  in  coronary  care  is  directing  itself  at  a
                  have  been  planners  and  people  who  had  to   get   programs    categorical disease, but to be  effective,  the  way  we  view
                  implemented   and   were   well   versed   and    became    well    effectiveness, it means  a  change  in  a  system.  Hopefully,
                  versed in these aspects.                                            through  a  demonstration  of  the  kind  we  are   projecting

                  94



          there  can  be  an  influence  on  a  much  broader   segment   of        before we can  begin  to  decide  how  we  are  going  to  eval-
          the   community   in   developing   approaches   to   a   specific        uate them.                 still   not   clear   to   me    what
                                                                                    And I must confess it's
          disease.                                                                                           going  to  use   for   that.   I   have
          So I think  that  there  is  a  danger  of   stating   issues   in        evaluative methods we are                    ifig    many     of
          such  broad  terms   that   it   becomes   absolutely   impossible        got some strategic concepts of why I'm do
          to cope with them.                                                        the things I'm  doing,  but  they  are  steps   along   the   way   to
          DR.   KELMAN:   Well,   again   I   think   if   you   came   away        what  has   been   discussed   in   terms   of   more   profound
          ftorn  the  discussion  all  day  with  the  idea  that  we  can't        changes in  the  system  than  the  reorganization  of  a  given
          measure health status, then really we failed.                             subsystem within  our   coronary   care   process.   I   hope   that
          I  don't  think  we  here   could   allow   you   to   slide   out        clarifies what I said to some extent.
          from  taking  a  hard  look  at  RMP   easily   by   saying,   "We        DR.   BROWN:   It   gets   back   though   to   this    business
          can't  measure  health  status   so   therefore   RMP   can't   be        about process and  end  results.  If  you're  going  to  try  to
          evaluated in those terms." It's not appropriate.                          define  how  many  people's  lives   you   saved   or   so   on,
          The  discussion  we  have  heard  thus  far   initiated   by   the        that's  going  to  take  a  very  long  time  and  may  not   be
          young  !ady  in  the  back  is   very   similar   to   many   dis-        possible  and   probably   isn't   even   important.   But   the
          cussions   I   have   been   in   after   a   program   has   been        process  is  important,  the  process  by  wl-dch   subregional-
          launched   and   they   say,   "Well,   we'd   better    get    an        ization or regionalization occurs.
          evaluator  in  here  to  tell  us  what  we're  doing  because  we        Now,  that  may  be  hard  for   people   in   the   Bureau   of
          don't  really  know."  And  I  think  that's  pretty   sad   after        the  Budget  to  measure,  but  that's  their  problem  as  well
          all this time.  I  cannot  for  the  life  of  me  understand  how        as ours, because  that  is  where  maybe  the  $96  million  can
          we could get into  the  sorry  state  of  spending  all  of  these        have  some  influence  on  what  is  happening   in   terms   of
          millions of dollars setting  up  all  of  these  regional  offices        the whole.
          and then come around and say, "Well          , I   really    don't        Now,  that's  about  as  global  as  I  could   make   it,   and
          know  how  to   judge   whether   one   or   another   region   or        within  that  there  are  55  sub-sets  and  probably   25   ap-
          one  or  another  unit  should  get  more  or   less   funds   for        proaches within those sub-sets of 55    .regions,    and     then
          what it wants to do."                                                     within that there  are  a  lot  of  other  smaller  things  that
           This   is   an   extremely   dangerous   kind    of    situation,        Dr. Shapiro refers  to  which  are  terribly  important,  but  I
          tying  it  back  into  some  of   Chuck   Lewis'   comments,   for        don't  know  how  you  measure  those  in  terms  of  lives  you
          an  evaluator  to  operate   in,   because   he   or   she   can't        save.
          possibly  win  in  such  a  situation.  In  other  words,   you're        DR.   KELMAN:   Could   I   be   antagonistic   and   ask    why
          putting  the  evaluator  in   the   position   of   defining   the        it's important  to  have  all  these  subregional  clusters  and
          objectives  of  the  program.  Do  you   really   want   that?   I        paraphernalia?
          don't think you do.                                                       DR.    BROWN:    It's    a     mechanism     because     someone
            DR.  CARPENTER:  Bill,   you   stimulated   a   lot   of   this.        feels  that  there  n-dght  be  a   better   way   or   a   more
            DR.STONEMAN-. Yes,l'dlike torespond.                                    economic way or something to deliver health services.
            I  didn't  mean  to  sound  like  an  evaluation  nihilist.  The        DR. KE L M A N: I'm asking an outcome question.
          thing  that  bothers  me  is  that  we  have  had  for  some  time        DR.  BROWN:  I  was  struck   with   this   business   here   of
          now  some   very   broad   and   general   aims   for   RMP   out.        the  neighborhood  health  clinic  where  the  analysis  of  the
          @ned  which  are  extremely  vague,  and  if  I  overstated  their        report says  that  95  percent  of  the  patients  get  followup
          vagueness  and   the   unlikelihood   of   their   immediate   ac-        contrasted  with  only  -  what?  -  10  percent  or   20   per-
          complishment,   I   apologize.   But   I   apparently   made   the        cent.  Therefore  the  neighborhood  health  center  is  a  good
          point.                                                                    thing?
            If  we  are  going  to  go  to   program   evaluation   at   the        DR. KELMAN: I don't know if it's good.
          regional  level  instead  of   concerning   ourselves   with   in-        DR.   BROWN:   Well,   nobody   knows,   but   that's   one   of
          dividual  project  activities,  then  I  would  submit  that  most        the objectives  it  seems  to  me  we're  hearing,  one  of  the
          of  the   evaluative   techniques   that   were   described   this        Roals of  the  regional  medical  program.  Access.  Isn't  that
          morning  are  more  appropriate  to  project  evaluation,   if   I
          can  use  that  term,  than  they  are  to   program   evaluation.        access?  It  doesn't  make  any  difference  whether  the   out-
                                                                                    come  was  better  for  the   patient.   Nobody   measured   it.
             Then it's necessary  for  us  to  hold  our  own  feet  to  the        But if we could guarantee access, that     ,s  politically   im-
          fire  in  terms  of  setting  some  precise   program   objectives        portant right now.
                                                                                                                                                   95





                    Now, I'm  not  saying  that's  good  or  bad.  I'm  just       services to migrants all over the country. But  because  ol
                 saying if you take stability of data you could  say,  well,       the lack of solid evaluation  information  and  because  ol
                 here are X number of  people  who  did  not  get  followup.       the lack of gras,,-roots support, this program is  in  real
                 Now they  get  followup.  Therefore,  you've  improved  the       jeopardy right now of being lost in the shuffle of  another
                 system.                                                           bill that was passed. And if I was any kind  of  prophet  I
                    Maybe all you have done  is  added  a  component  to  it       would say that the same thing  could  well  happen  to  the
                 that costs you money.                                             RMP.
                    DR.  CARPENTER:  I  suppose  the   fear   is,   Bob,   that       DR. CARPENTER: Dr. Fox?
                 although that  is  politically  important  today,  it  doesn't       DR. Fox:  I  think  Martha  and  1,Would  like  to  respond
                 sound as though it's  going  to  stay  politically  important,    to some of the comments.
                 whereas  whether  or  not  there  is   increased   access   to       For  those  who  don't  already  know,   this   is   Martha
                 improved  health  care  may  have  a  little  longer   staying    Blaxall who is a budget  examiner  in  the  Health  Branch  in
                 power as an argument.                                             the Office of Management  and  Budget.  She  also  helps  riie
                    MR.   SHAPIRO:   Let   me   give   one   example   briefly.    write  speeches  for  places  like  here  and  ropes  me  into
                 Then I've got to leave. And  I'm  going  to  oversimplify  the    interesting meetings.
                 situation.                                                           I  think  a  couple  of  points  have  been   raised.   The
                    During  World  War  11,  there  was  an  EMIC   program   -    problem  of  insecurity  of  funds,  for  example,  has   some
                 emergency  maternal  and  infant  care.  Nobody   thought   in    validity. The issue of lack of  goals  may  or  may  not  have
                 terms  of  an  evaluation  of   that   program.   There   were    validity. I think that can be carried too far.
                 millions  of   women   who   were   delivered   through   this       I  wonder,  for  example,  whether  you  were   at.   lunch
                 program.  After  the   war   that   program   was   abandoned.    and listened to Dr. Margulies' speech     ' He      enumerated
                 There was  no  supporting  evidence  that  could  be  used  to    certain things that were as clear as  they  are  going  to  be
                 sustain  a  program   that   roughly   corresponded   to   the    enumerated,  and  if  you   people   don't   understand   what
                 EMIC program. There are a         lot  of   people   who   are    they are, then I don't know what else can be done.
                 convinced  that  some  form  of  EMIC   program   would   have       You  also  heard  in  the  morning  that  the  concept   of
                 been  maintained  after  the  war  and  hopefully  would  have    themes  versus  specific  objectives  was  talked   about   by
                 resulted in further reductions in  infant  mortality  in  this    Don Schon,  and  I  haven't  heard  anybody  dispute  that  as
                 country  instead  of  the  long  sustained  period  of   small    a concept.  You  know.  The  messages  that  you're  going  to
                 decreases if  those  responsible  had  taken  the  trouble  to    get  will  consist  of  themes.  You'd  scream  if  you   were
                 think through the importance of evaluation.                       given specific objectives in terms  of  numbers  of  this  and
                    There  is  currently  a  program  in  maternal  and  infant    that type of unit that you must engage in.
                 care,  and  there's  a  huge  amount  of  money  being  poured       We  have  heard  that  you  can't  measure  health  status.
                 into that program. I don't  believe  that  that  program  will    Well, you know, I made  a  big  point  of  this  yesterday  in
                 continue in the long run unless it can  prove  itself  in  one    my  talk,  and  presumably  you  heard  that.  Not  that   you
                 way or another.                                                   can't  measure  health  status,  but  that  you  won't  get  a
                    I think  in  the  RMP  there  are  very  similar  types  of    single measurement of  the  impact  of  RMP  tied  up  in  one
                 situations. I don't care  how  carefully  you  regionalize  an    cost-benefit measure. We're aware of that.
                 ambulance   service    to    respond    to    coronary    care       On the other hand, there  are  things  that  can  be  done.
                 emergency   situations.   You   may   have    a    beautifully    I sure learned a heck of a  lot  today.  It  (the  panel)  has
                 operating  program.  But   unless   somebody   can   establish    some of the best information of what  the  state  of  the  art
                 whether  or  not  that   program   is   really   accomplishing
                 something in terms  of  outcome,  that  program  is  going  to    ui measuring health status really is.
                 be chopped. That's the rationale behind outcome.                     Let me tell you  some  of  the  things  that  I  think  one
                    DR.    LOGSDON:    Could    I    just     comment     about    can   expect.   I   think   one   can   expect   movement   in
                 another  program  that  had  a  similar  type  of  outcome  in    directions. What those  directions,  the  precise  directions,
                 the  migrant  health  bill   that   was   passed   and   which    ought  to  be,  that's  up  to  you  people  again.  You  know
                 operated  on  a  budget  much  less  than  this,  about   one-    the themes. What are some of die sys'L,@i@                  ii
                 fifth of the amount,  and  was  passed  primarily  because  of    duplication i-ti facilities being eliminated or  new  duplica-
                 Steinbeck, his  writing,  and  some  special  interest  groups    tion being prevented?
                 that  were  able  to  get  enough  support  in  the  Congress.       We're on the  verge  of  entering  into  the  kidney  field
                 And   this   program   provided   health   and   environmental    for big. Are we going to  have  the  same  Iliascos  there  we

                 96




      have had  in  open  heart  surgery?  If  we  do,  then  maybe    good things that we  think  we  can  do,  and  these  are  our
      the program should be questioned.                                objectives.  This  is  what  we   think   is   reasonable   to
         We have  also  heard  examples  here,  and  I  have  heard    measure  us  by.  Here  are  some  measures  that   might   be
      them,  anecdotally,  of  important  situations  where   duph-    tempting from your  point  of  view  but  we  think  they  are
      cation of facilities has been prevented.                         unrealistic because -
         We  know  manpower  is  important.  You  know.  Is   P-NIP        And  I  think  the  regions  have  to  come   forth   with
      doing  something  to  rationalize  the  introduction  of  new    honest information, not with snow jobs.
      manpower in the project areas?                                       Now, in a sense,  things  are  bad.  There's  uncertainty.
         These are meant  to  be  seed  money  projects.  Are  they    But the  uncertainty  isn't,  I  contend,  anywhere  near  :is
      engaging  in  projects  that  are  real  projects  that   are    bad as what your statements make us believe.
      absorbed into the  regular  system  after,  say,  a  two-  to        DR.  JESSE  B.  ARONSON:  I'd  like  to  ask   the   ques-
      five-year period?                                                tion as to  why  in  all  of  these  discussions  of  measure-
         One can look at  core  staff  and  ask  whether  they  are    ments  we  haven't  brought  in  or  I   have   heard   really
      developing  a  regional  strategy  that   intuitively   makes    nothing about the measurement of the cost factor.
      sense or is it  a  case  of  just  responding  to  individual
                                                                           We  know  that  we  are  far  from  getting   cost-benefit
      requests and interest groups that come in?                       studies.  We   certainly   can   get   cost-effectiveness   of
         We have heard statements  that  the  evaluator  can't  set
      objectives. Well, this is true in a purist sense. But if  the    process. And if  .we are  going  to  start  measuring  process
      evaluator  can't  help  the  decisionma-ker  set  objectives,    without  measuring  costs,  I  don't  think  we're   measuring
      can't start to ask questions that  assist  the  decisionmaker    process in any realistic sense, in any case that will  in  any
      in setting objectives, then the evaluator ought to  be  fired    political sense certainly be realistic.
      -  and  I  really  mean  that  -  because  that  may  be  the        And I  think  we  ought  to  put  more  of  our  thinking,
      most important thing that he can fulfill.                        and  we  ought  to  have  examples  of  studies,   where   the
         And I know  to  some  extent  the  regions  have  to  come    cost-effectiveness   of   process   becomes    an    essential
      forth and say, "Look,  within  these  themes  these  are  the    element in our whole measurement system.


















                                                                                                                                   97






                                           WORKSHOP ON PROGRAM EVALUATION

                                                                    Participants
       Harold W. Keairnes, M.D. - Moderator                                      Robert K. Ausman, NI.D.
       Coordinator for Evaluation                                                Deputy Director
       Tri-State Regional Medical Program                                        Florida Regional @tedical Program
       Avery M. Colt                   chusetts                                  Jack E. Thoi-nson, Ed.D.
       Field Associate for Eastern Massa                                         Coordinator of Evaluation
       Tri-State Regional Medical Program                                        California Regional 'Medical Program

       Claire G. Farrisey
       Special Projects Coordinator                                              Robert Beckman, Ph.D.
       Tri-State Regional Medical Program                                        Director of Research
       Osler L. Peterson, M.D.                                                   Nassau-Suffolk Regional Medical Program
       Associate Director for Data Collection
       Tri-State Regional Medical Program and                                    William R. Thompson
       Acting Chairman                                                           Deputy Director
       Department of Preventive Medicine                                         NVashington/Alaska Regional Medical
       Harvard Medical School                                                      Program

       Dean J. Siebert, M.D.
       Associate Coordinator for Dartmouth                                       Paul E. White, Ph.D.
        Medical School                                                           Division of Behavioral Sciences
       Tri-State Regional Medical Program                                        Johns Hopkins School of Hygiene and
                                                                                   Public Health

       Ruth B. Mott
       Research Associate for Data Collection
       Tri-State Regional Medical Program                                        Jeannette Forsyth, Ph.D.
                                                                                 Project Administrator, Information Support
       James J. Dunlop, Ph.D.                                                       System
       Arthur D. Little, Inc.                                                    Tri-State Regional Medical Program


                   Approaches to Program Evaluation                              Medical   Programs.   After   talking   about   the   impact   of   the
                         H. W. KEAIRNES, M.D.                                    anniversary   review   guideline   on   local   programs,    I    aid,-
                                                                                 "How   do   you   expect   the   evaluation    activities    to    con-
                                                                                 tribute   to   the    development    of    these    program    applica-
         Evaluation is assuming a larger          role  in    the    planning    tions?" This is the dialogue that followed:
       and    management    of     Regional     Medical     Programs.     The        MR.  LAWTON:   The   program   application   and   program   itself
       new   procedures   for    anniversary    review    program    applica-            has to demonstrate that  it  can  manage  the  process  in  its
       tions  and  in-depth  site  visits  indicate  that   increased   local            own  region  of  good  health  service  problem  solving.   The
       autonomy    in    management    of    activities    and    funds    is            (evaluation  technique for  doing  that  must  not  only  exist
       contingent   upon   a   clear   understanding   by    Washington    of            in the region but must  be  visible  in  the  application.  The
       yesterday's    achievements    by    the    program.    Under    these            region  has  to  know  how  to  apply  and  use  the  technique
                                                                                         and how to use the  results  of  the  evaluation  technique.  I
       conditions,   past   performance   is   equally   as   important    as            think it's a good  circle  involvement.  You  have  to  develop
       future   plans.   Evaluation,   whether   done   formally    or    in-            and put  down  a  technique  that  helps  you  do  your  job  -
       formally  -  if  done  at  all  -  helps   build   the   bridge   from            better.
       the past to the future.                                                       DR.  KEAIRNES-  You  talk  as  if  evaluation  has   something   to
                                                                                         do with planning.
          Recently   I   tape-recorded   a   brief   interview    with    Mr.        MR. LAWTON: I  find  them  hard  to  separate.  I  think  that  the
       Robert    Lawton,    Deputy    Director    of    Tri-State    Regional            credibility factor is extremely  important  here.  I  think  if
                                                                                                                                                      99




                         You are going to do  good  things  for  patients  and  good       But none  of  these  differences  negate  the  value  of  ti
                         things for patient care through rationalization,  then  you    game  movie  and  the   process   of   planning   for   tomorro
                         have  to  demonstrate  that  what  you  did  yesterday  had                         at   happened   yesterday.   What   folloN
                         some merit and improved patient  care  -  so  that  evalua-    on the basis of wh
                         tion is an on-going  thing.  Today's  planning  and  tomor-    is  a  description  of  the   concepts   and   methodology   f,
                         row's results are pretty dependent on  yesterday's  evalua-    taking an  RMP  game  movie  that  will  allow  a  clear  asses
                         tion.
                                                                                        ment  of  the  performance  of  the  teams  involved   in   Wil
                      In  the  game  of  improving  patient  care,  that's   another    ning  or  losing  the  game  of  rationalizing   and   improvil
                   way  of  saying  that  evaluation  is  part  of  the  process  of    the process of medical care delivery.
                   winning.  Planning  and  action  are,  or  should  be,  based  on
                   experience.    Evaluation    involves    the    systematic    de-                Concepts of Information Support
                   scription  of  these  experiences  and  the  associated  achieve-
                   ments.  If  done  well,  evaluation  can  supplement   the   gut-      Evaluators  in  Regional  Medica  Programs   play   the   rol
                   level  feelings  that  play  such  a  prominent  role   in   most    of cameraman, not  coaches  or  players.  In  their  role,  the
                   decisionmaking    about    the    future.    Unfortunately    de-    must  keep  the  camera  focused  on  the   crucial   activitie
                   cisionmakers  have  functioned  so   long   without   systematic,    on  the  playing  field  if  the  coaches  are  to  have  usefu
                   evaluation  that  many  feel  that  they  can  win  without   it,    game  movies.  Evaluators  have  not  been  hired   as   judges
                   or, at least, by paying no more homage to it.  '                     Only  those  persons  whose  decisions  influence  the  fate  o
                     The  model  for  winning  through   the   use   of   evaluation    an  organization   can   really   be   considered   as   judges
                                                                                        Evaluators  are  hired  to  provide  information  to   decision
                   has  been  established   by   that   multi-million   dollar   in-
                                                                                        makers  so  that  their  judgments   are   not   made   on   in
                   dustry   -   professional   football.   Each   week   each   team    complete,   inaccurate   or   biased   information.   In   thi!
                   records  the  process  of  their  winning  or   losing   in   the    sense,   they   are   concerned   much   more   with   INFORMA.
                   game  movie.  The  coaches  and,  to   a   lesser   extent,   all    TION SUPPORT than with judgmental evaluation.
                   members  of   the   team   spend   many   hours   reviewing   the
                   game   movie.   They   evaluate   every   plan   and   the   per-      This   concept   of   information   support    makes    sense
                   forinance  of  every  member  of  the  team.  Those  plays   that    only  when  the  decisionmakers  utilize  the  information.  11
                   worked well  will  be  used  again.  For  any  play  that  didn't    no  one  but  the  cameraman  sees   the   game   movie,   then
                   work,  decisions  will  be  made   about   the   performance   of    the  plans  for  next  week's  game  will  be  based   on   the
                   each  player  and  the  appropriateness  of  the  play.  On  this    rather  undetafled  and  unsystematic  recollections   of   the
                   basis plans  are  made  for  practice  and  for  the  next  game.    coaches  and  players.   Similarly,   taking   two   weeks   to
                   And  then  they  practice.  There  is  little  mercy  for   teams    develop the film  destroys  its  usefulness.  If  the  film  is
                   that  continue  to   make   the   same   mistakes   in   decision    available and utilized, then it must be  of  such  quality  and
                   making  and  performance  that   were   obvious   in   the   game    content that  the  coaches  and  players  find  it  useful.  If
                   movies.  Of  course  they  have  to   take   into   account   the    they feel that. it is useful, they will  utilize  the  informa-
                   limitations  of  their  personnel  and  their  system   and   the    tion  in  their  planning  and  decisionmaking  and  they  will
                   new  challenges  presented  by  the  next   opponent.   In   next    request  that   the   service   be   continued.   In   Regional
                   week's  game,  if  they   have   successfully   evaluated,   cor-    Medical  Programs   information   support   services   can   be
                   rected  and  planned,  they  will  win.   And   they   may   even    justified only  when  there  is  utilization  of  the  informa-
                   win  over  a  team  that   has   superior   personnel   and   re-    tioii  and  requests  for  additional  infori-nation   by   the
                   sources.                                                             decisioiimakers.
                   The   task   of   a   broad-base   social   change   organization
                   such    as    Regional    Medical    Programs    appears     more                   Decisionmakers in RMP
                   complex  than  that  of  a  professional   football   team,   but
                   only  superficially.   They   both   have   the   same   over-all    Who   are   these   decisionmakers    in    Regional    Medical
                   objective  -  winning.  RMP's  goal  line,   however,   is   less    Programs  that  correspond  to  the  coaches  of  the   profes-
                   well   defined.   There   are   many   more   ways   of   scoring    sioiial  football  teams?  One  of  the  important  differences
                   points.  The  process  of  moving   down   the   field   involves    between  the  two  games  is  the  larger  number  of   players
                   many   more   players.   The   opportunity   for   fumbling    is    and  decisioni-nakers   involved   in   RMP   activities.   RMP
                   much  greater.  The  rules  and  the  officials  are  much   more    dccisioiiniakers.fall into several important groups:
                   difficult to identify. The  fans  are  often  not  interested  in    1.  Coordinators  or  directors   -   the   senior   executives
                   paying  to  see  the  team  win.  And  there  are  no  time  outs    %vho are  responsible  for  the  implementation  of  the  plan-
                   during the game or between games.                                    iiing and operational activities of the program.
                   100




      2.  Planners  -  the  comnlittee  i-nembers  and  core  staff   to resist the activities  directed  towards  rationalizing  the
      ersonnel who deten-nine  the  direction  -  objectives  -  of   health care system.
      le program  and  the  activities  that  will  move  the  pro-   The   evaluator   in   focusing   his   infort-nation   support
      rain and the region in that direction.                          services must first know the location  of  the  goal  line  and
      3. Project directors  and  officers  -  the  core  staff  and   the rules of  the  game.  Then,  if  he  understands  who  the'
      )reject  personnel  who  manage  the   process   of   project   key players are and how  they  participate  in  the  game,  lie
      levelopment and operation.                                                    able   chance   of   providin-    a    meaningful
                                                                      stands a reason                          t,
      4.  Grantors  -   the   members   of   local   and   national   service; that is,he   will   make   the   appropriate    observa-
      tdvisory groups and the staffs  of  granting  agencies  whose   tions on  the  appropriate  players  during  the  entire  game.
      recisions   determine   which   activities    and    programs   Being  guided  by  the  decisionmakers  in  this   process   of
      )ecome funded.                                                  focusing  his  observations  improves  his  chances  of  making
      5.  Consumers   -   both   professional   and   lay   persons   a game movie that  the  decisionmakers  will  find  useful.  If
      whose support determines  the  success  or  failure  of  most   the decisionmakers  will  not  provide  the  assistance  or  if
      broad-based social chance programs.                             their  assistance  is  not  sought,  making  the  ai-ne   movie
                                                                                                                   9
      If  the  decisions  of  all  these  people  about   how   the   becomes  an   irrelevant   exercise.   Fortunately   for   both
      game  should  be  played  are   correct,   Regional   Medical   decisionmakers  and   evaluators   there   are   some   general
      Pro-rams stands a -good chance  of  winning  the  battle  for   guidelines to follow.
      rationalizing  the  medical  care  system  through  voluntary
      mechanisms. The thesis  of  this  paper  is  that  meaningful   Location of the Goal Line-
      information based on  past  experiences  and  provided  in  a   Problems and Objectives
      useful  manner  will  improve  all  crucial  decisionrnaking.    Each  problem  in  the   medical   care   system   defines   a
      Of  course,  evaluative  information  becomes  only  part  of   different goal line.  Setting  objectives  is  the  process  of
      the decisionmaking  process  and,  by  itself,  cannot  over-   specifying  which  goal  lines  should  be  crossed.   Planning
      come   problems   in   communication,   resources   or   con-   specifies the activities which if carried out  should  lead  to
      straints  that  also  influence  the  decisionmaking  process   crossing the goal lines.
      and its restflts.                                                 Analysis  of  published  studies,   surveys,   reports,   and
                     Work of the Evaluator                            applications gives the first level  view  of  the  problems  of
                                                                      a  health  care  system  in  a  geographic   area.   Interviews
       Meaningful information forms the       context    for    the   with all  classes  of  decisionmakers  and  other  key  persons
      Work  of  the  evaluator.  He   must   understand   how   the   are   required   to    understand    the    relation    between
      game  of  Regional  Medical  Programs  is  played,  who   the   described   and   perceived    problems.    The    degree    of
      players are, what direction the team  is  heading,  and  what   concensus  or  agreement  on  high  priority   problems   gives
      the coaches want to  see  before  he  knows  where  to  focus   some  indication  of  the   potential   cohesiveness   of   the
      the  camera.  For  example,  focusing  on  the  wide  flanker   medical care system for problem solving.
      while he sits on  the  bench  during  a  defensive  play  may      Obviously   the    Regional    Medical    Programs    cannot
      be the same as  focusing  on  the  evaluation  of  a  project   cross all possible goal lines or  solve  all  the  problems  of
      when  the  decisionmakers  really  need  to  understand   how   the  medical  care  system   simultaneously.   Objectives   and
      all the  components  of  the  program  are  working  together   priorities  help  direct  the  team  towards   those   problems
      to  further  the  task  of   winning   the   game.   Narrowly   that  most  need  to  be  solved  or  are  most   amenable   to
      'focused observations have  limited  value  to  understanding   solution.  Published  objectives  may  or  may   not   be   the
      the  total  game  process.  Indeed,  focusing  on  the  wrong   true   operational   objectives.   Discrepancies   arise   when
      area may prevent  the  coaches  from  observing  the  process   operational objectives are  perceived  as  being  not  socially
      of  scoring.  Meaningful  information  that  is  useful   for
      planning the next  game  depends  on  a  description  of  the   acceptable  or  when  there  is   lack   of   concensus   among
      entire field including  the  play  of  all  members  of  both   decisionrnakers   about   desired   objectives.    Such    dis-
      teams and the success of both  teams  in  crossing  the  goal   crepancies  make  it  more  difficult  to  mobilize   resources
      line. It includes all the projects  -  both  operational  and   to accomplish the objectives.
      Planning - as well as all  non-project  activities  of  staff      Public  objectives  can  be   determined   from   documents.
           comniittees.  It  also  includes  everything  that   the   Operational   objectives   can   be   determined   by    direct
                team - the forces for the status  quo  -  is  doing   interviews  with,  and  by  secondary  interviews  about,   key
                                                                                                                                    101






                     decisionmakers.  Following  these  processes  allows  de-                             distribution of key persons in relati on to  proble
                     scription of the nature of the  objectives  and  of  dis-                             institutional   relationships   as   characterized
                     crepancies between published and operational  objectives.                                exchange,  domain,  domain  conflict,  and   jo
                                                                                                              planning activities
                     Data Source:  documents                                                              history of previous change efforts
                                   direct interviews
                                   secon ary interviews.                                     Record of Team Performance -
                     ;knalysis:    nature of problems and objectives                         Results of Previous Resource Allocations
                                   consensus on problems                                        Local   Regional   Medical   Programs   have   up   to    thi
                                   discrepancies between published     and    operational    years   of   experience   as   operational   programs.    Unles!
                                      objectives                                             game  movie  exists,  this  description  of   team   perfon-naii
                     Rules and Playing Conditions of the Game -                              will  have  to  be  primarily  performance  statistics  that   ,
                     Resources and Constraints in the                                        generally   available,   such   as   the   number   and    types
                     Medical Care system                                                     plays,  number  of  yards  gained,  and   the   number   of   fi
                                                                                             downs,   penalties,   and    scores.    Recollections    of    t
                        The  Regional  Medical  Program's  task   lies   in   a   setting    players  give  some  clues  to   the   process,   but   they   a
                     created  by  existing   institutions   and   their   services,   key    subject  to  bias.  Nevertheless  this   information   is   part
                     persons   both   lay   and   professional,   existing    legislation    planning for tomorrow.
                     and  regulations,   and   financial   resources   both   fixed   and       The   players   in   the   Regional   Medical   Program   gar
                     flexible.    General    socioeconomic     conditions,     population    can  be  considered  to   be   staff,   committee   and   adviso
                     ,distribution,      transportation      patterns,      communication    group  members,  and  all  other  persons  in  the  medical   ca
                     systems  and  educational   resources   are   also   part   of   the
                     milieu.    Describing    these    facts    makes    apparent     the    system.  It  is  important  to   identify   through   interviews
                     playing conditions of the game.                                         the members  of  the  team,  their  skills  and  attitudes,  th(
                     I  The  constraints   in   the   system   are   created   by   legal    assignments   in   the   change   process,    and    their    p(
                     forces,   institutional   relationships   and   history.    Legisla-    formance  record.  Their  skills  relate   to   their   trainiii
                                                                                             their  position  in  their  institutions,  their   concern,   ai
                     tion,   regulations    and    guidelines    may    be    found    in    their  commitment.  Their  assignment  as  well  as   their   p(
                     published  documents,   but   their   impact   and   their   ability
                     to  respond   to   new   problems   can   be   learned   only   from    formance vary with the activities.
                     administrators  who   have   had   to   work   within   and   around      Identifying all  the  activities  or  plays  that  are  carri(
                                                                                             out  is  perhaps  the  most  difficult  task  facing   the   ev;
                     them.  Institutional   relationships   can   be   characterized   by    uator.  There  are   so   many   simultaneous   activities   wi-
                     patterns  of   1)   institutional   exchange   of   board   members,    vague  starting   points,   a   paucity   of   progress   report
                     staff,    clients,    and    communications,    2)     institutional    confusion as to  who  is  participating,  and  a  lack  of  agre
                     domain   for   clients   and   resources,   3)    domain    conflict    ment   on   when   the   play   is   completed   and,   therefor
                     both  actual  and  perceived,  and   4)   participation   in   joint    when  it  is  appropriate  to  measure  progress.  The  easy  w,
                     planning   activities.   Historically   the   fate    of    previous    out  is  to  restrict  one's   concern   to   funded   operation
                     change   efforts   and   the   general   responsiveness    of    the    projects.   That   is   appropriate   if   operational    projec
                     system  to   new   problems   and   new   resources   suggests   the    account  for  90  percent  of  core  staff   and   project   sta
                     rules  which  influence   the   success   of   all   future   change    time  and  budget.  Unfortunately  that  is   rarely   the   cas
                     efforts. These are the rules of the game.                               The  whole  spectrum  of  activities  that  must  be   identific
                       This   information,   although   crucial   to    evaluation,    is    include   operational   projects,   planning    projects,    con
                     the  keystone  of  planning.  It  makes  clear  the   condition   of    niittee  activates,  central  administration  services  includir
                     the  playing  field  and  the  rules  of   the   game.   The   eval-    communication,       research,      data     collection,      an
                     uator   should   watch   for   ignorance   or   raisperceptions   of    evaluation, conferences, developmental negotiations.
                     ,the  conditions  and  rules  by  persons  playing  for  the   local      Once  an  activity  has  been  identified  the  players,   the
                     Regional Medical Program.                                               assignments,  their  performance,  and   the   effectiveness   (
                     Data Source: documents                                                  the  activity   should,   if   possible,   be   identified.   Tb
                                  interview                                                  performance   of   individuals   relates   to   how   well   the
                                                                                             carried  out  their  assignments.  The   effectiveness   of   th
                     Analysis:   identification of key  persons,  institutions,  and  re-    activity  asks  not  only  how  many   yards   were   gained   -
                                     sources                                                 short  term  estimate   of   progress   usually   based   on   a

                     102





     chievei,nent  of  project  objectives   -   but   also   whether   the     patterns   and   should   not   result   in   the   players    becoming
     Jay  or  activity  resulted  in  a  first  down  or  the  crossing  of     so defensive that they will not participate.
     goal  line  -  a  long  term  description   of   the   resolution   of       The    ongoing    mechanism    for    recording     and     reporting
     ny   of   the    specific    problems    on    which    the    program     evaluative   information   depends   on   the   philosophy    of    the
     bjectives focused.                                                         coordinator,   the    evaluator,    and    the    core    staff.    But
       Resource  allocation   is   akin   to   selection   of   plays   and     participation    in    the    evaluative    process    will    probably
     he    assignment    of    players.    Effectiveness    in    achieving     result  in  more  effective  utilization   of   the   information.   In
     rogram   objectives   is   obviously    related    to    having    the     his  assigned  role,  the   evaluator   should   be   responsible   for
     ight   players    and    the    right    play.    Resources    include     surveillance   of   documents,    especially    n-dnutes    of    meet-
     ersonnel  time  and   funds   which   are   directly   accessible   to     ings,   application   for    planning    and    operational    projects
     4e   local   regional   medical    program    plus    all    available     and  reports  of  projects   and   studies   in   order   to   maintain
     ersonnel   time   and   dollar   resources   in   the   region    that     some  general   structure   for   all   evaluative   observations.   He
     ould    potentially    be    mobilized    towards    achievement    of     may    supplement    Ws     observations     by     interviews     with
     ,rogram objectives.                                                        persons  involved   in   the   various   activities,   by   participant
                                                                                observations    in    committee    meetincs,    planning     activities
       Restrospectively     many   details    of    resource    allocations                                              0
     nd   player   performance   are   lost.    Effectiveness    both    in     and   consultations,   and   systematic   reports   from   core   staff
     noving  down   the   field   and   in   scoring,   however,   is   ap-     and   project   directors.   Involvement   of   many   of   the   staff
     iarent  because significant  gains  are       t, usually       obvious.    in   reporting   participant   observations    and    their    analysis
     )etails   become   more    important    when    progress    has    not     provides   an    opportunity    to    train    them    in    evaluation
     ieen  obvious.  In   this   circumstance   winning   in   the   future     concepts     and     the     use     of     evaluation     information.
     )bviously    depends    on     developing     a     more     effective     Although   discrete   segments   of   the   program   may    seem    to
     .Uocation   of    resources    because    new    players    and    new     require   specialized   research    or    project    evaluation    such
     [ollars are not usually available.                                         activities   are    not    a    substitute    for    ongoing    program
                                                                                evaluation.    Program    evaluation    requires    the     identifica-
       Data     Source: documents                                               tion  of  all  activities,  all  the  players  on  the  teams  or  some
                      interviews                                                other   major   catecory.   The   performance   of   the   players   in
                                                                                each  of  the  activities,   the   success   of   the   activities   in
       Analysis:      descriptions  of persons involved  in                     making   procress   down   the   field   and   drawing   first    downs
                         Regional Medical   -Program activities                 and   the   effectiveness   of   the   whole   mix   of    plays    and
                      identification of activities                              players   and   short-term   achievements   in    moving    the    pro-
                      identification of effectiveness                           gram  across   the   goal   line   in   scoring   gains   against   the
                      description of resource allocation                        problems that exist in the region.
                                                                                   In  this  context  evaluation   itself   is   one   of   the   major
       Once  the      location  of   the   goal   line,   the   rules   and     activities    of    a    program.    Effectiveness    of     evaluation
     ?laying  conditions  of   the   game,   and   the   record   of   tewn     activities   can   be   judged    from    its    influence    on    the
     Performance   have   been   developed    by    the    evaluator,    he     decisionmaking    and    the    planning    processes.    Indeed,    if
     ias  two  o-bvious  tasks:  first,  to  report  this  information   to     evaluation    cannot    be    demonstrated     to     contribute     to
     his   organization,   and   secondly,   to   set   up    an    ongoing     winning  the  game   it   cannot   be   justified   as   an   important
     mechanism   for   recording   and    reporting    evaluative    infor-     activity   of   the    Regional    Medical    Program.    Effectiveness
     mation.   Both   of   these   processes   depend   upon    the    spe-     and   relevance   must   guide   the   entire   process   of   observa-
     cific  conditions  and   needs   of   his   program.   He   must   re-     tion,   analysis   and   reporting   of   evaluative   information    -
     member   that   the   information   should    be    considered    con-     that  is,   effectiveness   and   relevance   to   the   decisionmaking
     fidential  and  that   the   coordinator   of   the   program   should     of    all    classes    of    decisionmakers    from    consumers    to
     have  complete  control   over   the   use   of   his   analyses   and     Congressmen.
     reports.   The   evaluator   should   work   closely    enough    with         The     importance     of      involvement      was      summarized
     the  coordinator  so  that  the  results   are   made   available   in
     a   concise,   meaningful,    useful    form,    but    with    enough     quite  well  by  Mr.   Lawton   in   our   interview   when   I   asked
     accompanying   detail   for   use   by   other    decisiomnakers    if     him  if  he   had   an   opinion   about   what   proportion   of   RMP
                                                                                effort  should  be   put   into   evaluation.   Let   me   close   with
     desired   by   the   coordinator.   The   evaluator   in    developing         response:
     the   information   should   recognize   that   the   reports   should             "No, I don't think I have.
     be   constructive   and   not   destructive.   The   reports   'should             I  see  it  working  in  this  way,  an  evaluation  component,
     allow    an     opportunity     for     development     of     winning         such as  you  and  your  associates,  but  in  addition  I  @k  out
                                                                                                                                                    103




                   geographic  and  medical  school  coordinators  have  to   apply activities  of  the  program  and  the  effects,  both  observed
                   evaluation in th@ work  and,  if  they  need  it,  get  educated and  anticipated,  of  these  activities.   Program   evaluation
                   about   evaluation   technique.   Our   program   iniplementors, -,niphasizes  a  comprehensive  approach   that   accounts   for
                   whether  they  are  in  the  RMP  orkanization  or  in   outside
                   health institutions have to have an appreciation  of  evaluation all activities  supported  by  the  resources  of  the  program,
                   also, because I find it impossible to separate  evaluation  from not  just  discrete  operational   projects.   Ideally   program
                   the objectives of programs or projects.                         evaluation   provides   a   mechanism   for    monitoring    the
                      I would hope that there is a  little  bit  of  evaluation  in progress  and   the   effectiveness   of   all   activities   in
                   everybody - including me."                                      furtlierin-  the   purposes   of   the   RMP.   These   purposes
                                                                                   include  the  philosophy,   goals,   and   objectives   of   the
                            Program   Evaluation    Workshop    -                  program  as   a   whole,   as   well   as   the   more   limited
                                       A Case Study                                objectives  of  each  discrete  project  or  activity.   Program
                            HAROLD     W.     KEAIRNES,      M.D.                  evaluation  information  can  then   lead   to   better   under-
                                                                                   standing  of  the  most  effective  way  to  allocate  financial
                   This  paper  describes  the  educational  content  that  was    and  personnel  resources  within  the  program  and  to   main-
               developed  by  the  faculty,  the   educational   methods   that    tain quality control of all supported activities.
               were   used   in   a   workshop   session   sponsored   by   the       When  used  for   learning   or   planning,   program   eval-
               Re-ional  Medical  Program   services,   observations   on   the    uation  attempts  to  assist  the  staff  and  advisory   coiii-
               educational  process,   and   implications   of   the   workshop    mittees   through   a    continuous    feedback    process    to
               experience  for  training   and   evaluation   and   development    improve  the  quality  of  their  decisions  and  actions.  This
               of program evaluation methods.                                      is  the  traditional  planning  -  action  -  evaluation  cycle.
                                                                                   Maximum  benefit  occurs  when  all   the   actors   participate
                                EDUCATIONAL                 CONTENT                at least partially in the entire process.
                   The  evaluation  of  Regional  Medical   Programs   at   the    Role of E@,altiator i?i Program Evaluation
               local  level  is  an   open-ended   story.   Formal   guidelines
               have  not  been   established   because   they   are   evolving.       The  evaluator   concerned   with   program   evaluation   is
               Their  evolution  is  based  on  a  process  that   takes   into    the keystone for the following processes:
               account   the   multiple    uses    of    evaluation,    extreme       I .Identification  of  the  uses  for   program   evaluation.
               variability  in  judgmental  criteria   and   significant   dif-       2. Identification  of  those  individuals  and   groups   who
               ferences  in   the   philosophy   and   approach   of   the   55          will use pro.-ram evaluation -information.
               regions.  The  open-ended  nature  of  the  legislation   -   PL       3 .Identification  of  appropriate  evaluation  criteria   as
               89-239  -  requires  this  evolutionary  approach.                        described by each user for each use.
                                                                                      4. Identifyin-  his  own  role   in   obtaining   information
                   There are three  basic  uses  for  evaluation: 1) jtistifica-
               tion 2) control and 3) learning or planning.                              that  allowsitidgment  on  the  established  criteria  for
                   NVhen used for  justification,  evaluation  must  deal  with          each use.
                                                                                      5. Carryinc, out his role.
               judgmental  criteria  that  have  been   established   and   are                  0
               regularly   utilized   by   various   groups,   e.g.    National    In the process he  must  avoid  substituting  his  criteria  for
               Advisory  Council,  RMPS,   local   and   national   legislators    those of the users o    f   program    evaluation    information
               and  political  figures,  organized  providers,   and   consumer    and  must  be   sure   that   the   information   is   accurate,
               groups.  Usually  these  criteria  are   unwritten   and   often    reliable, and  relevant  to  the  criteria,  the  uses  and  the
               unartictdated, but  they  do  exist.  In  general  these  groups    users.
               make  judgments  based  on  their  private   interpretation   of       The evaluator's role in program       evaluation   is   quite
               the  intent  of  the  legislation,  and  these   interpretations    different from that in project evaluation.      'Project    eval-
               form  the  criteria  for  their  judgments.  Their   interpreta-    uation  is  concerned  with   discrete   activities   based   on
               tions  may  vary  considerably  from  that  of  core  staff  and    objectives that are  or  can  be  well  defined  and  agreed  to
               RAGs  of  local  programs  and   often   may   not   allow   for    by  the  project  staff  and  the  funding  agency.   Here   the
               problems  of  feasibility  and  practicality  in  carrying   out    evatuator's  role  is  to  apply  systems'   concepts   tempered
               the legislative mandate.                                            by  econon-dc,  educational,  epidemiologic  medical   care   or
                   When  used  for  control  purposes,  evaluation  helps   ad-    other  technical  considerations  to  the  development   of   an
               niinistrators  and  executive  conunittees  such  as   RAGs   to    economically  feasible   evaluation   methodology   -   and   to
               be aware of  progress  in  implementation  of  all  the  various    the  supervision  or  performance  of   the   evaluation   plan.


               104







    )rogram  evaluation  is  concerned   with   the   conglomerate          temporal flow and sequence  of  activities  -  that  is,
    activities that have poorly  defined  objectives,  that  often          effects   (,,n   process   and   organization   can   be
    @annot be clarified to the satisfaction of all  the  users  of          observed  in  1-3  years,  but  significant  effects  of
    !valuation   information.   Here   the   evaluator   must   be          transforiiiiii-  the  medical   care   system   on   the
    !xtremely  flexible  and  understanding  in  order   to   deal          process  or  end  results  of  patient  care  may   take
    vith the  complexities  of  the  task.  Rigid  application  of          3-1 0 years.
    raditional  evaluation  approaches,  such  as   may   be   ap-       Stated in a different nianner this process calls for:
    )ropriate  for   discrete   projects,   becomes   increasingly       I .Identification  of  all  activities  of  the  program  -
    rrelevant as  programs  become  larger  or  broader  in  their          past, present and anticipated.
    cope.  Precise  evaluation  of  one  component  of  the   pro-       2. Identification  of  all  possible   effects   of   these
    ?ram usually gives  little  insight  into  the  total  program          activities.
    nd usually  provides  little  assistance  to  those  who  must       3. Developing  methods  for  describing  the  process   and
    dminister orjustify the financing of such programs.                     the effect of all activities, not limiting the scope  to
    Access of Program Evaluation                                            funded operational projects.
                                                                         4. Conducting  the  evaluation  I  in   a   rational   time
      For  the  purposes   of   this   workshop,   program   eval-          frame.
    ation was  defined  as  a  process.  This  process  definition       5. Reportiti(,  the  information  to  decisionmakers  in  a
    ook into account the  very  primitive  state  of  the  art  of          wav  that  helps  them  make  more  rational  decisions.
    @rogram  evaluation.  Although  projects   are   underway   to       Understandin-  program  evaluation  as  a  process   rather
    evelop  the  methodology  of   evaluation   of   broad   range   than  as  a  procedure  is  fundamental  to  evaluators   being
    Dcial  change   organizations   such   as   Regional   Medical   successful in their activities. In  this  context,  success  in
    Irograms, there  are  no  generally  agreed  upon  and  tested   program  evaluation  is  defined  as   a   development   in   a
    Methodologies at present.                                        body  of  information  which  is  perceived  as  being   useful
      This   process   of   program   evaluation    follows    the   by  individual  and   group   decisionmakers   concerned   with
    Dllowing steps:                                                  the  operation  of  the  program  and  that  played  some  part
      I .The   evaluator   shall   develop   a   thorough   under-   in decisions that were made.
         standing  of  the  philosophy,  history,  strategy,   and
         activities of the program. In this  step,  he  may  infer                  EDUCATIONAL PROCESS
         from  his  observations  what  the  objectives   of   the       The  workshop  attempted  to  reproduce   this   evaluation
         program   are   and   how   these   observed   objectives   process.  One   particular   regional   medical   program   was
         relate   to   published    or    reported    goals    and   selected so  that  the  process  and  its  associated  problems
         objectives.   Such   inferences   should   be    verified   could be illustrated.
         when possible.                                                  Following  an  introductory  lecture   on   program   eval-
      2. The  evaluator  shall  determine  who  wants  or   should   uation, a group  of  consultants  met  in  a  panel  discussion
         want   program   evaluation   information.   From    each   with several members of the  staff  of  the  illustrative  pro-
         of these individuals  or  groups,  he  shall  obtain  the   gram. This panel had two major objectives:
         criteria  by  which  they  make   judgments   and   their       1. To  identify  the  philosophy,  history,  strategy,  and
         intended  uses   of   the   information:   justification,          activities of the program.
         control, or learning.                                           2. To  identify  the   question's  that  the  staff   members
      3. Based  on  these  objectives,  criteria  and  uses,   the          felt  needed  to  be   answered   by,   the   evaluation
         evaluator   shall   develop    a    program    evaluation          process.
         methodology.     This     methodology      should      be       The  staff  members  described   their   regional   medical
         comprehensive,  practical,  and  efficient.   Unless   he   program as being directly concerned with tran          sforming
         has  outside  financial  support   for   evaluation   re-   the  medical  care  system  through  influence  and  a  variety
         search,  the  costs  of  carrying  out   the   evaluation   of activities into a system  that  filled  the  gaps  in  care,
         should  probably  be  less  than  10  percent  of   total   made  better  use  of  manpower,  improved  quality  of   care,
         program  funds.   The   scientific   disciplines   incor-   and  controlled  the  costs  of  care.  They  used   the   term
         porated  into  the   methodology   should   reflect   the    opportunistic  intervention"  to  describe   the   fact   that
         needs  of  the  users  of  the  information  rather  than   their  activities  were  guided  more  by  requests   for   as-
         the particular scientific discipline  of  the  evaluator.   sistance    than    by    comprehensive,     objective-oriented
         The   evaluation   should   take   into    account    the   planning.  "TiHing  the  sod"  was  the  term  they   used   to

                                                                                                                                 105




                                               generate  such   requests   in    had no previous.experience in evaluatio    n, participated,,
                describe activities designed to                                    re actively in the discussions. Evaluators,. on the  othe
                geographic areas, and among    groups   not   aware    ofithe    MO
                availability of.assistancp from the  Regional   Medical   Pro-   hand, appeared  very  uncomfortable  with  this  "process",
                gram.  They,  told  multiple  stories  of  how  this  created    approach to evaluation. They tended to react  as  if  their
                change, e.g. a  $1500  contract  was  used  to  assist  seven    previous highly  technical,  project-oriented  approach  to
                community  hospitals  in  developing   home   care   programs    evaluation  was  being  threatened.  As'  a  result   their
                in less than twelve  months;  a  neighborhood  health  center    participation attempted  to  force  the  workshop  from  la
                was opened by  the  city  health  department  in  association    process  orientation  towards  a  structured  approach   in
                with the negotiations  for  a  project  on  the  screeningand    which,  they,  the  evaluators,  defined   the   questions,
                treatment  of  selected  chronic  diseases  that  was   being    criteria, and methodology.
                                                                                  One  example   merits   presentation:   A   staff   member
                submitted from the same geographic area.
                 The  staff  members  developed  a   series   of   evaluative    who was  concerned  with  the  quality  of  the  "linkages"
                questions  that  they  felt  were  important.   Three   major    that she helped establish  was  badgered  by  an  evaluator
                questions evolved:                                               to provide a precise definition of a, ."linkage.'.' Although
                  1. How good are my cooperative arrangements9                   she gave several examples of  what  she  meant,  she  could
                                                                                 not  in  ten  minutes  of  interrogation  give  a   precise
                  2. How balanced are the program activities?
                  3.  What  changes  in  the  health  care  system  have  the    definition. A program  person  pointed  out  that  such  an
                     program activities influenced?                              approach was not  getting  anywhere.  This  led  two  other
                  The  workshop  divided   into   three   small   groups   to    program  persons  to  present  methods  to  describing  new
                discuss  the  priorities   of   these   questions   for   the    contacts  and  working  relationships  between  individuals
                evaluation  effort  and  the  methodologies  that  might   be    that, although not quantitative, did provide  a  basis  foi
                used   to   answer   them.   Each   group   contained   staff    understanding the quality of. these "linkages."
                members   of   the   program   and   consultants.   It    was     The  discussion  groups  tended  to  focus  on  their  own
                intended  that  each  group,  in   addition   to   developing    questions about program effectiveness and  to  discard  the
                priorities and methods,  deal  with  the  practical  problems    questions posed by the staff  members.  As  the  discussion
                of  implementing  the   methodologies   and   reporting   the    groups followed  this  path,  they  became  unable  in  the
                results.                                                         allotted time to  probe  into  the  areas  of  methodology,
                  The  closing   session   of   the   workshop,   a   general    Ibis reflected the difficulty experienced by  many  of  the
                session,  was  designed   to   demonstrate   the   range   of    Participants in  understanding  the  concepts  of  prograrr
                priorities, methods, and solutions  that  were  available  to    evaluation as a process and their  inability  to  play  th(
                solve   common   problems   in   program   evaluation.   This    educational  game  that  had  been  established   for   th(
                depended  on   the   developments   in   each   small   group    workshop.
                discussion and was intended  to  reflect  the  learning  that
                had occurred in this open-ended educational format.                                  IMPLICATIONS
                                                                                   Evaluation of programs  that  have  as  broad  a  mandate
                OBSERVATIONS ON THE EDUCATIONAL  PROCESS                         as  the  Regional  Medical  Programs  is   very   difficult
                                                                                 Proven  methodologies  do  not  exist.  The  effect  of   @
                  The  faculty  anticipated  many  problems  in  this  educa-    social-change  program  is  often  much  greater  than  th,
                tional  endeavor.  Approximately  150  man  hours  of   plan-    sum of the effects  of  the  discrete  operational  project
                ning plus  two  trial  workshops  with  smaller  groups  made    that   they   fund.   Application   of    simple    concept
                the  faculty  aware  that  most  evaluators   would   require    developed  in  project-oriented  evaluation  activities   i
                10-15  hours  of  training  before  they   would   begin   to    often  inappropriate.  Evaluators  who  hold  responsibili@
                understand  the  concepts  of   program   evaluation   as   a    for program  evaluation  are  often  the  victims  of  thei
                process.  In  spite  of  that,  an  attempt   was   made   to    previous training and experience in project evaluation.
                compress  this  learning  experience  into  five  hours.  The       In   order    for   meaningful    program      evaluatio
                planning,  however,  could  not  compensate  for  the   short    methodologies  to  be  developed,   traditional   evaluatio
                time allowed for the workshop. As  a  result  the  objectives    methodologies   must  become  subservient  to   the   broa(
                of the workshop were only partially attained.                    range demands    of these broad-range  programs.  The  fir:
                   Participants   in   the   workshop   immediately    sorted    step  in  this  process  is  developing  evaluation  concep
                themselves  into  two  categories:  evaluation   or   program    that ate similarly broad-tange. It  probably  requires  thi
                                                             mn@t   nf   whom    evaluators no longer sit on the side-lines of  the  prograi





   as judges and that they become  actively  involved  in  the                          SUMMARY
   entire change  process  with  program  responsibilities  in      The  educational  content  and  methods  of  a  workshop
   addition to their evaluation  responsibilities.  Only  when
                                                                 session on evaluation  of  Regional  Medical  Programs  has
   evaluators  have  a   profound   understanding   of   their   been  described.  The  objectives  for  the  workshop  were
   program  will  they  know  which  consultants   and   which   only partially attained.  Observations  on  the  complexity
   methodologies  are  truly  appropriate  to  the   task   of   of the subject, the time limitations of the  workshop,  and
   program evaluation.                                           the previous experiences of the participants  were  related
      Training  and  program   evaluation   begins   with   an   to the partial success of this particular training  method.
   understanding  of  the  program   to   be   evaluated.   It   Further developments in the  field  of  program  evaluation
   proceeds  to  the   development   of   program   evaluation   depend upon  evaluators  actively  participating  in  their
   concepts. Having passed these stages, it can focus  on  the   own program activities  and  in  a  continuing  educational
   application of  proven  methodologies  or  the  development   process.
   of new methodologies. Training in  program  evaluation  is,
   therefore, just as much of a process as  is  program  eval-
   uation itself.

















































                                                                                                                     107





                            WORKSHOP ON RESOURCE ALLOCATION/ECONOMICS

                                                                  Participants
       hn Glasgow, Ph.D. - Nloderator                                          Conrad Seipp, Ph.D.
       ;sociate Coordinator, Research                                          Associate Professor, Health Services
       and Evaluation                                                          Research Center
       )nnecticut Regional Medical Program                                     University of North Carolina
       .A. Florin, M.D.                                                        Robert L. Berg, M.D.
       c)ordinator, New Jersey Regional                                        Professor and Chairman, Department of
       Medical Program                                                         Preventive Medicine and Community Health
                                                                               University of Rochester

       lichael Zubcoff, Ph.D.                                                  Wayne A. Kimmel
       lead, Health Economics Section                                          Department of Economics
       Tennessee Mid-South Regional
                                                                               Public Service Laboratory
       Medical Program                                                         Georgetown University Hospital
       ames R. Jeffers, Ph.D.                                                  Charles W. Caldwell, M.P.A.
       )irector, Medical Economics Research Center                             Associate Coordinator, Iowa Regional
       Jniversity of Iowa                                                      Medical Program
                                                                               Charles L. Joiner, Ph.D.
       ohn E. Wennberg, NI.D.                                                  Director, Bureau of Research and
       coordinator, Northern New England                                       Community Service
       Regional Medical Program                                                Alabama Regional Medical Program

                Cost-Benef it and Cost-Effectiveness                           concerns that the user of  these  techniques  needs  to  keep
                     Analyses in the Health Field                              clearly in mind if he hopes to use them  effectively  and  if
                                                                               he  is  to  understand  what  information  these   techniques
                          JOHN GLASGOW                                         do and do not provide.  The  purpose  is  not  to  present  a
        Rising  levels  of  health  care  expenditures;   the   as-            step-by-step   "how   to   do   it    cost-benefit    manual"
       sociated increases in medical Drices and  alleged  shortages            although  one  might  be  desirable  and  desired.   Examples
       of manpower and facilities; the declaration that  access  to            of  calculations  of  both  a  hypothetical  and  theoretical
       medical care is a right, not a privilege;  and  the  growin-            nature,  in  addition  to  that  provided  by   Crystal   and
       role of the government in the health  care  field  have  led            Brewster,  abound.2-13-  Neither  is  the  purpose  here  one
       to concern with the effectiveness  of  alternative  delivery            of  exploring   new   theoretical   frontiers.   Indeed,   as
       systems  or  resource  allocations.  Concern  with  the  ef-            Klarnian has pointed out "so  much  has  been  written@  .  .
       fectiveness  of  delivery  emphasizes  the   importance   of            about  the  application  of  cost-benefit  analysis  to   the
       using scarce resources (or dollars) in such  a  way.  as  to            health field that almost every  point  that  n-dght  be  made
                                                                                                ,,I 4
       maximize the return per dollar spent.  This,  in  turn,  has            has been made.         Although   perhaps   something   of   an
       led to the search for  planning  and  analytical  techniques            overstatement reminiscent  of  Mill's  premature  claim  that
       WIL-ch might aid in the task of rationalizing  the  resource            everything  that  was  to   be   known   of   economics   was
       aflocative process. Two  such  techniques  are  cost-benefit            known, the observation  has  sufficient  validity  to  narrow
       and cost-effectiveness analyses.                                        the present concern. The  attempt  here  will  be  to  ensure
         The  Crystal-Brewster  paper'  provides  an   introduction            that terms and concepts used in  cost  studies  are  clearly'
       to  cost-effectiveness  and   cost-benefit   analysis.   The            understood   as   to   their   definition,   the   underlying
       ,Present essay attempts  to  build  upon  this  introduction            assumptions,  and  the  result  and  implications   for   the
       and  to  suggest  certain  conceptual   and   methodological            analysis. It should be clear that the objective is not to  be
                                                                                                                                              109




               critical  of  previous  work.  However,  an  understanding   of           principle,  cost-effectiveness  should  do   the   same.
               the  limitations  involved  in  such  studies  both   increases           In  practice,   however,   cost-effectiveness   analyses
               their   value   to   the   decisionmaker    and    provides    a          are often  less  complete  in  listing  the  total  cost
               ren-dnder  of  the  need  for  constant  improvement   of   the           and  benefits.  For  example,   external   effects   are
               analytical  techniques  involved.  A  secondary  goal   is   to           often   ignored   and   certain   desired   results   or
               consolidate into  one  paper  a  number  of  points  which  are           benefits are  specified  with  all  others  regarded  as
               fairly   well-developed   in   the   literature,   but   widely           constants or relatively unimportant.
               scattered and therefore less accessible to  the  less  special-         b. Cost-benefit  analysis   normally   values   the   costs
               ized  reader.  A  final  objective  is  to   provide   to   the           and  benefits   in   monetary   terms.   This   provides
               interested  reader  a  bibliographic   resource   for   further           the   comnion   denominator    necessary    for    corn-
               personal investigation.                                                   parisons  of  alternative  types   of   prograi-ns.   In
                                                                                         cost-effectiveness  analyses   the   measure   of   out-
                          THE  NATURE OF THE BEASTS                                      put often  is  not  in  terms  of  dollars,  but  rather
                Cost-Benefit   and   Cost-Effectiveness   are   terms    often           in some other unit such as man-years saved,
               used interchangeably. In actual  fact,  the  two  are  not  the      2. These   differences   in   comprehensiveness   and   tech-
               same  although  both  concepts  do   derive   from   the   same         niqlic  result  in  cost-effectiveness  being  used   most
               theoretical  fount-capital  budget-@ng  theory.   In   essence,         often   "when   various   benefits   are   difficult    to
               capital  budgeting  theory  is  concerned  with   the   present         measure  or   when   the   several   benefits   that   are
               and future  costs,  and  the  associated  benefits  over  time,         measured    cannot    be     rendered     commensurate."3d
               of  alternative  investment  strategies.   The   goal   is   to      3. Cost-b-nefit    analysis    allows    comparisons    among
               allocate   scarce   resources   to   their   most    productive         several  programs   which   have   different   objectives.
               (profitable)  uses.  Thus,  the  theor,,,  is  concerned   with         Cost  effectiveness   is   used   to   compare   differing
               determining the effects, as well  as  the  costs,  of  specific         ways of obtaining the same objective.
               alternatives available.                                              4. The  objective  of  a  cost-benefit  study  is  to  deter-
                 In  cost-benefit  analysis,  the  monetary  cost  of  a  pro-         mine  if  an  action  or  program  is  worth  undertaking;
               gram,  or   intervention   activity,   is   compared   to   the         the  objective  of  a  cost-effectiveness  study   is   to
               monetary  value  of   the   expected   benefits.   This   cost-         determine  the  best   way   of   achieving   an   already
               benefit ratio (of total costs to  total  benefits)  might  then         determined course of action.
               be  used  to  compare   alternative   programs   to   determine
               which  is  the  best  potential  investment.  For  a   specific       CONCEPTUAL         AND         METHODOLOGICAL         ISSUES
               activity, the comparison of  costs  and  benefits  is  for  the       In  this  section,  differences  between  cost-benefit   and
               purpose  of   answering   the   question:   Do   the   benefits   cost-effectiveness  studies  will  be  ignored  for   the   most
               received justify the expenditure (i.e., is  the  ratio  greater   part. Here  the  concern  will  be  with  the  terms  used,  the
               than I or some other arbitrarily set number)?                     concepts  involved,  and  the  implications  of   the   measure-
                 Cost-effectiveness  analysis,  in   contrast,   attempts   to   ment  techniques  used.  In  general,  the  comments   will   be
               compare   the   cost   of   alternative   approaches   to   the   applicable to both types of studies.
               achievement  of  a  specific  set   of   results.   The   goal,
               therefore, is not to determine  the  feasibility  of  achieving   Ae Measurement of Costs and Benefits
               a goal (theoretically  that  has  already  been  decided),  but       The essence of the  cost-benefit  approach  is  the  assign-
               rather  to  select  from  among  alternative   approaches   the   ment of dollar values to all  resources  so  that  the  benefits
               one approach which will  result  in  a  given  output  for  the   of a  specific  activity  might  be  compared  to  the  cost  of
               least cost or the maximum output for a given cost.                the  intervention   and   to   the   projected   benefits   from
                  Although  somewhat  artificial  in  nature,  the  definition   alternative   investment   opportunities.   Obviously,   it   is
               of the terms does allow  us  to  specify  in  some  detail  the   vital to include  in  the  dollar  valuation  all  the  relevant
               major  characteristics  of,  and  distinctions   between,   the   effects associated with a given action.
               two concepts.                                                         Economic   Costs   of   Disease   Defined.   The   econon-de
                  1.  Cost-benefit  analysis  is  more  comprehensive  in  its   cost  of  disease  or  injury,  as  contrasted  to  expenditures
                     focus than cost-effectiveness analysis.                     for  medical  care,  reflects  both  direct  and  indirect  cost
                     a.  Cost-benefit  includes  a  consideration  of   social   components.  Direct  costs  include  the  actual  medical   care
                         or  external  effects  as  a  part  of  the  complete   expenditures  necessary  for  the  treatment  of   the   disease
                         enumeration   of   the   costs   and   benefits.   In   or   injury.   These    expenditures    would    include    both
                                                                                                                                              110



    )ersonal  (i.e.,  the  cost  of  hospital   care,   nursing   home     technology   when    making    a    long-term    capital    invest-
    @are,  physicians'   service,   drugs,   nursing   services,   and     ment' ' Other examples of  sigiiif@c-an-t-o-mi-ss'i-o-n--s--c--ould
    ;i@lar    type    expenses)    and    non-personal    expenditures     be  provided,  but  the   point   has   been   made.   Many   costs
    'i.e., the cost  of  research,  training,  facilities,  equipment,     and  benefits  are  excluded  because  (1)  there   is   no   known
    ind  a  pro-rated   share   of   the   annual   cost   of   health     way  of  measuring  the  factor,  (2)   because   it   is   assumed
    nsuran ce).  Indirect  costs  are   those   costs   to   the   in-     any  undesirable  side-effects  could  be  corrected   if   desired
    lividual  or  to  society  in  the  form  of   lost   productivity     through  fiscal  tax  and  transfer  measures,   or   (3)   because
    rttributable  to  the  disease  or  injury.   In   essence,   this     the   analyst   considers   them   of   minor   import   for    his
    iinounts  to  imputing  a  dollar  value   to   the   productivity     purposes.  Valid  as  the  reason  for  exclusion   may   be,   the
    ost   through   premature   death   or   disability.    Obviously,     fact  remains  that  the  end  result  is  for  most   studies   to
    -,he  imputation  must  take  into  account   varying   life   ex-     concentrate  on   what   is   easily   measurable.   Unfortunately,
    .)ectancy,  labor  force  participation  and  earning   rates   by     in  many  cases,  the  easily   measurable   are   not   the   most
    Different  sex  and  age  groups;  the  "value"   of   individuals     important   effects   which   should   be    considered.    As    a
    )utside  the   market   pricing   mechanism   (i.e.,   housewives,     result, particularly, in the health field, it  is  vital  to  avoid
    ,Iderly,    children,    unemployed);    and    the    appropriate     undue   stress   on    the    importance    of.economic    measure-
    liscount rate.                                                         ments.   In   general,   this   means   it    is    necessary    to
      It  is  important  to  emphasize   that   the   economic   costs     complement    economic    values    with     other     non-economic
    )f  the  disease  as  defined  above  are  really  the   projected     values in determining the proper resource allocation.
    Denefits  in  any   cost-benefit,   cost-effectiveness   analysis.        The   Quantij7cation   Assumption.   The    most    basic    as-
    rhat is, the benefits  to  be  derived  from  an  action  are  the     sumption  in  any  cost  study  is   that   it   is   possible   to
    -Hminated   losses   in   production   output,   personal    well-
                                                                           quantify  in  monetary   terms   the   benefits   and   costs   as-
    being,   and   resource   utilization   which   result   from    a     sociated with  a  specific  activity.  In  actual  fact,  even  as-
    successful  program.  The   cost   denominator   is   simply   the     suming that  all  benefits  and  costs  will  be  included,  it  is
    projected budget of the program.                                       still  not  possible  to  quantify   'with   precision   even   the
      Enumeration  of  the  types   of   factors   included   in   the     most  relevant  factors  despite   major   advances   in   measure-
    usual cost study makes it clear      that  a   number   of   costs     ment   techniques.4,11   The   reasons   are   easily    explained.
    and  benefits  are  typically  excluded  from   the   calculation.     Th@ implications are somewhatmore subtle.
    In  addition,  a  number  of  assumptions,   both   explicit   and
    implicit,  underlie  the  definition  of   direct   and   indirect        It  was  noted   that   the   benefits   associated   with   the
    costs,  the  valuation  of  specific  components  of   each,   and     success of an  activity  tend  to  result  from  (1)  increases  in
    lime  use   of   the   technique   @f   discounting   which   have     economic   productivity   due   to    decreased    mortality    and
    major  implications  for  the  validity   of   any   cost   study.     morbidity  levels;  (2)  reductions  in  the  need   for   facility
    Again,  the  emphasis  on  the   presence   of   biases   in   the     and  manpower   resources   given   the   eradicated   or   reduced
    technique  and   approach   is   not   designed   to   be   overly     health   problem    and    (3)    the    existence    of    certain
    critical.  Rather,  the  purpose  is   to   explicitly   recognize     intangibles    (consumer    benefits)    associated    with    good
    what  conclusions  these  studies  do  and  do  not  allow  to  be     health  such   as   reduced   anxiety   in   the   individual   and
    drawn from the data presented.                                         society or an increased sense of well-being.
      Despite  the   effort   to   define   the   economic   cost   of        It  should  be  clear  that  (1)  and   (2   above   are   more.
    disease  broadly  to  include  both  direct  and  indirect  costs,     susceptible  to   precise   measurement   than   is   (3).   As   a
    it is obvious  that  not  all  costs  and  benefits  are  included     result,  most  studies  tend  to  ignore  the  latter  effect.  7he
    in  even  the  most  rigorous  analysis.  For   example,   it   is     result  is  therefore  to  often  significantly  understate  the  i
    common   to   ignore   the   so-called.   "spfll-over"    effects.     potential   benefit   of   any   activity   and   to   particularly
    These   are   the    desirable    (or    undesirable)    secondary     underestimate   the   value    of    any    activity    in    which
    impacts   of   a   given   action.   Illustrative   of   such    a     consumer   benefits   constitute   a   major   portion    of    the
    Secondary   impact   would   be   the   effect   on   prices   and     total   benefit.   That   is,   since    the    consumer    benefit
    availability  of  medical  care   for   the   general   population     component  of  total  cost  varies  between   types   of   diseases
    which   resulted   from   the   attempt   to   provide   for   the     or  illnesses,'  6  the  exclusion  of  such  benefits,   or   even
    health   cafe   needs   of   the   aged   through   Medicare   and     their  inadequate  valuation,  wifl.tend  to  result  in   a   mis-
    Medicaid.  Another  cost   often   not   incorporated   into   the     leadingly  low  cost-benefit  ratio  for  those  diseases  with   a
    @Wcuiation  is  the  cost  of  "locking"  oneself  into  a   given     high    consumer    benefit    element     in     comparison     to





       and  a  decision  about  the  desirability  of   past   expend-    there  is  significant   unemployment   among   the   resources
       itures. The second  result  might  be  illustrated  best  by  a    in question, utilization in  this  activity  not  only  entails
       hypothetical  illustration   ftorn   an   article   by   Warren    little or no  cost,  it  may  provide  an  additional  benefit.
       Smith."                                                            That is, the result  may  be  a  pure  benefit  composed  of  a
              assume  -   an   investment   of   $10   in   a   device    net output gain plus reduced welfare costs.
          which  produces  50  units  an  hour  and  we  learn  of  an      Third,  those  who  would  make  use  of   these   techniques
          invention of two improvements [tolincrease - ef-                often  desire  the  specification  of  a  policy  which   would
          ficiency. Item A  -  costinc,  S5  -  [increases  productive    simultaneously provide  the  greatest  benefit  and  the  least
          capacity to]60 units per hour. Item B - costing $7,             cost. While theoretically  possible,  the  attainment  of  this
          can increase the  output  to  65  units  per  hour  -  which    goal is Iii-nited by  at  least  two  factors:  (a)  limits  on
          adapter  would   be   the   best   choice   from   a   cost-    ability  to  spend  and  (b)  requirements   for   expenditures
          effectiveness standpoint?                                       of a given size. To illustrate, it is often possible to  obtain
            The  total  cost,  if  we  buy  item  A,   will   be   the    a  larger  benefit  from   a   larger   expenditure   and   the
          original $10 plus the  added  $5,  or  $15,  and  the  total    increase in benefit  size  need  not  be  proportional  to  the
          resultant output is  60  units  per  hour.  Dividing  output    increase  in  expenditures.  As  a  result,  increased  expend-
          by cost gives us a ratio  of  60  to  15  or  4.  The  total    itures can often result in a  much  higher  cost-benefit  ratio
          cost, using item B, will be the ori-inal $ 1 0 plus  $7,  or    than would be a  lesser  expenditure  for  the  same  activity.
          $17 with a resultant output of 65,  or  a  ratio  of  65  to    But if you do  not  have  more  funds  to  invest,  the  larger
          17 or  3.8.  The  conclusion  using  this  misleading  anal-    ratio is immaterial. In  the  same  way  it  is  possible  that
          ysis is that  item  A  is  preferred  because  it  seems  to    unlimited  funds  properly  allocated  amonc,  a   variety   of
          give the largest ratio of effectiveness to cost.                alternatives  might  provide  a  total  benefit  greater   than
            The  marginal  or  added  cost  for  item  A  is  $5,  and    the same amount invested  in  a  single  project.  Yet  if  the
          the added output is  10  units  per  hour  for  a  ratio  of    required funds are lin-dted, the  use  of  funds  in  one  area
          10 to 5 or 2.0.  The  marginal  cost  using  item  B  is  $7    effectively   precludes   simultaneous   investment   in    the
          or 2.1. Our  conclusion  using  this  correct  procedure  is    alternative. That is, given the  cost  of  doincy  A,  you  may
          that  item  B  is,  preferred   because   of   its   greater    not be able to  do  any  part  of  B  given  its  minimum  cost
          marginal ratio."                                                requirements.  This  suggests  two  factors  of   import.   (1)
          Additional  Issues.  Although  not  technically   cost   and    Cost analysis,  in  the  usual  case,  will  be  able  only  to
       benefit  measurement   issues,   four   other   comments   need    suggest policies which will provide  the  greatest  benefit  at
       to be made concerning this general 'area. First, ma                a given cost or a given benefit for the  least  cost;  and  (2)
                                                                    ny
       studies distinguish between  the  effects  of  an  activity  on    in order to provide even this direction,  there  must  exist  a
       production  (income)  and  the  effect  on   the   distribution    clear-cut  statement  of  the  objectives  desired.  In  short,
       of income  resulting  from  the  fact  that  beneficiaries  are    cost studies are not a  substitute  for  decision  making,  but
       not  necessarily  those  who  pay   for   the   program,   that    rather  a  tool  to  help  rationalize  the   decision   making
       there  can  be  an  impact  on   relative   prices   and   real    process.
       incomes;  that  program  investment  implies  foregone   alter.       Fourth, it is also of  some  value  to  emphasize  that  the
       natives; and similar forces.  Typically,  these  distributional
                                                                          total  dollar  cost  of  a  project  does  not  always  reflect
       effects  are  ignored  in  most  cost  studies  and  for   good
       reasons.  Nevertheless,   it   should   be   noted   that   the    accurately    the   allocation    of   resources    which    it
                                                                          theoretically  summarizes.  That  is,   the   relevant   market
       ignoring of these effects can lead  to  either  an  over  -  or    prices of resources  do  not  necessarily  reflect  their  true
       understatement of  total  benefits  derived.  For  example,  if    value (ie., actual costs)  to  the  system  within  which  they
       an activity not only treats a  disease  but  leads  to  a  more    are being allocated. Some of  the  reasons  why  this  is  true
       equitable tax po   licy,  the  ignoring  of  this  latter  fact    have been previously  alluded  to  (e.g.,  valuation  of  "non-
       seriously understates the value of the program.                    market resources" or  of  human  life  itself'and  the  use  of
          Second,  it  was  previously  noted  that  one  cost   often    previously  unemployed  resources).   Other   reasons   include
       excluded from  most  cost  calculation  was  the  effect  of  a    the fact that prevailing prices reflect  a  given  income  dis-
       program's initation  on  the  price  and  availability  of  re-    tribution. A different  income  distribution  rWght  result  in
       Sources  which  could  have  been  used  in  alternative   ways    a  different  demand  and   price   structure.   Fina   ,   one
       had they not  been  used  in  this  activity.  This  implicitly    might note that only if  the  structure  of  market  prices  is
       'assumed  a  state   of   full   employment.   However,   where    that   which   would   occur    under    perfect    competition





                 would  the  social   opportunity   cost*   equal   the   net   cash       the    productivity    of    an    investment    and    society'
                 payments for the project.' '                                              reluctance  to  sacrifice   current   for   future   consumption
                                                                  2 3
                     Ideally, then, as Wennberg has noted,            the   vigorous       Attempts  to  utilize  a  private   market   rate   of   interes
                 application   of   these   techniques   presupposes   a    detailed       assume  that  the  individual's  time   preference   for   mone,
                 and  accurate   analysis   of   the   system   and   the   economic       coincides  with  the  collective  preference  as   expressed   b
                                                                                                                                              2 4
                 environment  if  the  cost  and   benefit   implications   of   the       the market rate. This is not necessarily ture.          Indeed
                 proposed project are to be fully understood.                              it is argued that the individual's discount rate        for  th(
                                                                                           distant  future  will  tend  always   to   exceed   socie ty,S.21
                 The Discounting Procedure                                                 Fourth,  the  time  preference  for  money   is   not   constant
                    Previous mention was         made   of   the   desirability   of       with age.  That  is,  it  tends  to  vary  inversely  with  lif(
                 expressing  future  benefits   and   cost   in   terms   of   their       expectancy.  Finally,  for  any  discount  rate  chosen,  it  i@
                 present  equivalent  value  (i.e.,   to   determine   the   present       usually   assumed   that   the   general   price    level    and
                 value  of  future   dollars).   The   present   value   of   future       productivity will remain constant  over  time.  This  is  not  a
                 expenditures  is  the  sum  of  money  that  would   have   to   be       valid  assumption,  but  an   understandable   one   given   the
                 set  aside   at   present   and   cumulated   at   some   rate   of       measurement     problem     involved.     However, Klarman     has
                 interest  in  order  to  equal  the  monetary  cost  of   the   ex-       suggested  the  desirability  of   developing   an effective   net
                 penditure  at  the  time  it  will  be  incurred.   Reversing   the       discount   rate    by    combining    price    and productivity
                 idea,  one  n-Light  discount  a  sum  of  future  money   by   the       changes   that   are   simultaneously    operative into  a  single
                 interest rate chosen to get its present equivalent.                       rate.3b   For   example,   one   might   divide    the    chosen
                    Obviously,  the  choice  of   the   discount   (interest)   rate       discount  rate  by  average  price  change  (in  percent).  This
                 used  in   the   calculation   is   of   vital   importance.   Some       ratio  divided into  the  sum  of  the  present  value  of  output
                 argue  that  the  proper  interest  rate  to  use   is   the   pre-       in dollars   terms  multiplied  by        the    increase      in
                 vailing  market  rate.  Others  argue   that   this   is   inappro-       productivity expected  would  give  an  effective  net  rate  of
                 priate  for  a  number  of  reasons.  No  attempt  will   be   made       discount.
                 to  examine   the   controversy   surrounding   the   proper   rate       It  is   clear from  the  above  summary  that   the   choice   of
                 of discount to use since this entails a  field  in  itself.  It  is       the   discount rate  to  be  used,  no  matter   how   universally
                 of   value,   however,   to   briefly   summarize   some   of   the       accepted,   is an   exercise   in   value   judgment   and   quite
                 major  issues  involved  in  the  controversy  leaving   to   those       arbitrary.  Under  these   circumstances,   one   might   wonder
                 interested  the  task  of   reading   the   references   previously       why  the  discounting   exercise   is   performed.   Blum,   for
                                                                                                                                            16
                 cited.                                                                    example,   suggests   abandoning   the   practice.      However,
                   First, even a  desire  to  use  a  market  rate  of  interest  is       it  seems  clear  that  there  is  no  other  effective  way  to
                 hampered  by  the  fact  that  there  is  no  single  market  rate.       reduce   continuous    and    unequal    dollar    streams    to
                 Rather the  rate  varies  with  the  type  of  loan  or  obligation       comparable  values.  Consequently,  accepting   the   need   for
                 involved,   the   borrower,   and   time   period,   among    other       and  value   of   discounting,   the   concern   is   with   the
                 things.  Second,  in  the  choice  of   a   proper   discount   for       implications of the  process  for  the  results  of  the  study.
                 social  benefits  and  costs   associated   with   public   invest-       7he  most  obvious  implication   is   that   relatively   small
                 ments,  the  choice  is  complicated  by   the   existence   of   a       variations   in   the   discount   rate   chosen   can   produce
                 close   relationship   between   investment   decisions   and   the       relatively large differences  in  the  cost-benefit  ratio.  And
                 social   discount   rate   used   in   investment   planning    and       the  greater  the   time   span   involved   the   greater   the
                 between   investment,   the   method   of   financing   used,   and       variance.  A  second  implication  is  that   the   higher   the
                 fiscal policy.  Third,  a  discount  rate  is  intended  to  equate       discount  rate  chosen,  the  less  likely  programs  with  long
                                                                                           delayed returns  are  to  be  given  high  benefit-cost  ratios.
                   *Social  Opportunity  Cost  is  the  reduction  in   consumption        A third implication,  which  flows  from  the  second,  is  that
                 and investment which occurs due to  the  transfer  of  funds  from        service  programs  will  be  favored  over   research   programs
                 the private to the public sector. It is the sum of (1)  the  amount       in the usual case. As  a  result  it  often  is  suggested  that
                 of foregone direct consumption in the private sector  and  (2)  the       studies should  provide  multiple       rate     analysis     to
                 discounted value over time  of  the  decrease  in  future  consump-       demonstrate  the  range  of  priority   rankin   which   results
                 tion which  would  otherwise  have  resulted  from  the  investment                                                   9
                 of the portion of after-tax  income  not  presently  consumed.  For       from different rates.
                 an excellent  review  of  the  concept  and  its  development,  the       Miscellaneous Yr6blems
                 @iterested reader might consult the references to Feldstein in  the
                 bibliography.  Further  and  more  recent  works   are   those   by       In  addition  to  the   biases   and   weaknesses   imposed   by
                 Baumol, ARTOW, and Pauly also listed in the bibliography.                 measurement  techniques  or   the   discounting   process,   two

                 114




                                                                                                                l,nust  ,yield  unambiguous
                                                                                     useful  these  techniques                    12 3 it is true that use
                       in    areas    might    be     nen,ioned.     First,          riteria on the project over another",                     ves to be
             proble    ,  valuation  of  human  life   Or   some   of   the          c                  techniques    does     'Ice pKogram object.!
             ,pared to th ts,        the        calculation         of         amounts of             these cified.               , difficulties involved in
             indirect   COS                       s)          is          conceptually unambigously     spe arPue        that        th
                       edical care (direct cost                                      Finally,   One   can   -                     lue in itself. Certainly,
                                                                                 'cult                  te study        ha,         va
             ended for in speaking, however, it may be as diff                       doing   an   adp-qua                                      aker  to
             practically                                      ts       given       the these   problems   should    force    the    decision    ni
                       urate    estimates    of    these    COS ty        of         ap-                r the technique should even be applied
             develop acc le,   accurate   data,   the    difficul                    question      whethe                         That is, in many cases
             k of availab                  efits when multiple                    morbid to certain  Problems or           decisions.          analYlis
             rtionment of total         cost-ben and, the existence          of          free           .red, and  the  sophistication  of
             riditions  exist  in   concert, o    the     degree     this     difficulty the    time    reqtl be  greater than equired or affordable,.
             rvices Or  payment  in  kind.'r                  ther             over-or involved,        may tudy      itself       will involve the use of resource
                       ticular  case,  beiefits   may   be   ei                      After  all,  the   I   prof
             jsts in a par     icant amount .                                        ,vhich  might  be   more    Itably employed elsewhere.
             riderstated by a signif                          ost        of        his
                       ny    cost    studies    subtract    thec
             second,  ma                                      alculating           the                      Bibliography
             lailitenance from    future    'earnings    ill     cit       should       be                                             ost Benefit and
                                man."'       if        deducted                                         oyal A. and   A.   W.   Brewster,   "C   tll Field:  An
                       alue of a.          an    result    in    a    value    measure l.               crystal, P, ti,eness    Analyses     in the       Heal
             @conomic  v                                                                                Cost Effec
             -ealized that the calculation c                  t     is,     it     may                  "Inquiry   3(4)   1),c,,ber.            Program
                       which    might    be    negative.    Tha                                         introduction,                          I
             af output loss                eakinz    the     best     course     would 2.               Department of Health, Education aid Welfare
                                                                                                                  ted Disease control Programs
             appear   that   econ0mically'sP                  This    is     generally                  AnalY sis Gro up on selec              otor Vehicle
                       ff, the population at risk.                                                                                t-cost Analysis to M
             be to "kill 0                                    .    In    any     case,                  a. Application  of  Benefi   .,itant Secretary for Program
                       red    a    practical     recommendation                                         Accidnt,,   office   of   A
             not conside   -               ias      program      el,,tion       toward                  coordination, August,  1966            Division  of
             the practice  Will  tend  to  b                  ,      or       younger,                  public Health  Programs  fr puthrit's
             those  activities  aimed  at  the  'chigh   income                                         b. nic       Diseases,       U.S.  Public  Health  Services,
                                                                                                        Chro
                       ctive   worker                                                                   ember, 1966
             more  produ                                                                                Sept                      .ng, ilrogallning, and Budgeting
                                                                                                        C. An  Analysis  of  Planni  -troi Branch, Division of
                                                                                                                                  c
                                CONCLUSION                                                              Cancer Control,  Can  er   Con         ices, 1966
                                                              re    not     infallible                  in ic Diseases, U,  S,  Public Health    Serv
             It  should  be   clear   that   cost   studies   a                                         Chron
                                                              fact,            applied                  . Herbert F-                           niversity  Press,
                                    urce     allocation.     In                      3.          Klarman, O,,i,,      of       Health (Columbia    U
             guides  to  proper  reso                         fit     ratios     would                  a. Econ
                        comparison      of      cost       bene                                         1965) pp. 162-11                       g Benefits of
             vigorously a                                     rograms          serving                  vt)hillis control Programs, in  Measurin fnan,  ed.
             tend  to  result   in   a   prepondenance   of   p                                         b.IIs. .ment     Expenditure,,     Robert     Dor
                       adult, white, college niale,                                                     Govern                    ion, 1965) pp. 361-4"
             the   young ally  clear  that   p                ent   state    of    the                  rookings           institut pact of Heart Disease," in Second
                                           ,iven    the    pres                                         (B
             it  is  equ                   -e   to   include    all    the    relevant                  C. "Socioeconomic         In'- Cardio-vascular Disease, The
                       St  study   can   hop                  those           included                  National   conference    on            ashington, D.C.:
             art, no  co            I    0     measure     even                                         Heart  and  Circulation,   vol.   2   (W or Experimental
             costs   and   benefits   0    tf    precision.    Indeed,    the     whole                                           erican societies f
             with      areal    degree,     0                                        e-                 Federation      of      Ain
                       ny                  ization    of    objectives    to    measut                  Biology,  1965)-   pp.   69.3-707.     osenthal, "Cost
                       Om         conceptual                  rcise     ill      value                      j.o.ls  Francis,  and   G.   S-   P,  of Chronic
             process  fr                   .continuous       exe                                        d.  ness   Applied   to   the   Treatment
             ment  Of  benefits    is      a                  with     events     that                  Effective                 I Care 6(l) January-February,
             judgements  compounded      by      a      concern                                         Renal    Disease,"    Medka
                       rtain    and     often     unneasurable.                      he                 1968,      pp.       48-54,
             are    unce at case,          .er with   such   studies    at    all?    T                 4. Rice, Dorothy                                      d
                              why       both                                                                                      of Cardiovascular Diseases an
             In      th-        .          e. if   one   keeps   in   mind   that   these               a. ,Economics            costs         ies No. 5 (U. S.
             reason   is quite          Simple                                       of
                                           an       unwarranted       a       Dearance                                                         on, D.C.)
             techniques      often      give                     P.                                                                                  'Economic
                                                              ot     a      substitute for                                                     Health
             objectivity  and           that they      are      nques      can-be      of real          b.                                         11 Amen  .can
                       making,    then    these    techni     -                                                                                Life,
             decision                      maker.   They   can   be   of   valbe    by                                                         mbe,,  1967)  pp.
             value    to the          decision                itly list  the  expected                  C.  al                    lic Health (No,
             forcing   the    decision    make,    to    explic ctivity   and    thereby                1954-1963.                             -Benefit Analysis-. A
             bene   ts   and   costs   Of    a    proposed    aclaims.  It   highlights                 5. Prest,  A.  R.  and  R.  TurveY,   "Cost  1965)  PP-
                       ri                  ation    of    these                                                                      (December,
             allow     critical     exainill                                                            survey,"  Economic  Journal
                                           alue      judgements,assumptions          and                35.
             the    pre    n    e    of    v                  a   method   for    Sys-                  683-7                     on chronic Kidney Dieas@s, (U-
                       se C.        .      . it i' I in  short,                                         6. Report  of  the   Committee         ington, D -C.).1967
                       ry         valuations                       pilatio;,,      and                                            tinF
             arbitra                         s                corn.                                                               office,. Wasli
                                           development)                                                 S.     Government      Prin
                       ic        information                                         be
             -@temat                                t   is   not    true    that    to                                                                        115
             utilization.          Moreover, while i





                    7.  WiIdavsky,  A.,  "fhe   Political   Economy   of   Efficiency:     24. Huffschmidt, Maynard M., "Standards and          Criteria  fo
                        Cost-Benefit   Analysis,   Systems   Analysis,   and   Program         Formulating   and   Evaluating   Federal    Water    Resource
                        Budgeting."   Public    Administration    Review    (December,         Developments,"  (U.   S.   Bureau   of   the   Budget,   1961
                        1966) pp. 292-310                                                      especially p. 11
                    8.  Smith,  Warren  F.,   "Cost-Effectiveness   and   Cost-Benefit     25. Baumol,  William  J.  Welfare  Economics  and  the  Theory,o@
                        Analyses  for  Public   Health   Programs,"   Public   Healtli         the State (Harvard University Press, 1952) pp. 91-92
                        Reports (November, 1968) pp. 899-906        '                      26. Blum,  Henrik  L.  and  Associates,  Health   Planning   1969
                    9.  Hallan, J.  B.  and  B.  S.  Harris,  "The  Economic  Cost  of         (American  Public   Health   Association,   Western   Regiona
                        End-Stage Uremia," Inquiry (December, 1968) pp'. 20-25                 Office, 1969) P. 8-21
                    10. Levin, A.  L.,  "Cost  Effectiveness  in  Maternal  and  Child     27. See,  for  example,  D.  J.  Reynolds,  "The  Cost  of   Roa(
                        Health." New England  Journal  of  Medicine,  278(19)  May  9,         Accidents" Journal of the Royal Statistical Society,  1  19(4
                        1968, pp. 1011-1047                                                    September, 1956, pp. 393-408
                    11. Muskin, Selma J.
                        a.    "Health  as  an  Investment,"   Journal   of   Political
                              Ecotiomy,   70(2)   Supplement,   October,   1962,   pp.              Role of Social and Behavioral Scientists
                              129-57                                                                           in RMP Evaluation*
                        b.           and Francis d'A.  Collings,  "Economic  Costs  of
                              Disease and Injury," Public  Health  Reports,  vol.  74,                        MICHAEL ZUBKOFF
                              September, 1959, pp. 795-809                                    First  le.t me  preface my remarks by stating that
                    12. Fein, Rashi,  Economics  of  Me?ital  Illness,  (Basic  Books,
                        1958)                                                              believe  the  social  and  behavioral  scientists'   key   contri
                    13. Wiseman;  Jack,  "Cost-Benefit  Analysis  and  Health  Service     butions  to  RMPs  are  in  the  areas  other   than   evaluation
                        Policy," Scottish  Journal  of  Political  Economy,  vol.  10,
                                                                                           such as being an  initiator  of  change  in  the  region  as  wel
                        February, 1963, pp. 128-405.
                    14. Klar7nan, H. E., "Present Status of Cost-Benefit  Analysis  in     as  aiding  in   the   development   of   program   strategy   fo
                                                                                           achieving  RMP  specific  goals  of   increased   regionalizatioi
                        the  Health  Field,"  American  Journal   of   Public   Health
                        57(l 1) November, 1967, p. 1948.                                   and more equitable distribution of health services.
                    15. Thedi,  Jacques  and   Claude   Abraham,   "Economic   Aspects        Before turning  to  a  "definition"  of  the  role  for  socia
                        of Road Accidents,"  Traffic  Engineering  and  Control,  vol.     and  behavioral  scientists  in  RMP  evaluation,  it  is   neces
                        2, February, 1961, pp. 589-95.                                     s
                    16. Weisbod,  Burton  A.,   7he   Economics   of   Public   Health     ary  to  spend  a   few   moments   revienving:   1)   the,variou
                        (University of Pennsylvania  Press:  Philadelphia,  1961)  pp.     levels  of  evaluation  that  exist  and  2)  possible  strategic
                        95-98.                                                             of evaluation within RMP.
                    17. YJarman,  H.  E.,  "Some  Technical   Problems   in   Areawide
                        Planning  for  Hospital  Care,"  Journal  of  Chronic  Disease                           Levels of Evaluation
                        17(9) September, 1964, pp. 735-747
                    18. See,  Eckstein,  O.,  "A  Survey  of  the  Theory  of   Public        Basically there are three levels of evaluation:
                        Expenditure  Criteria,"  in  James  Buchanan   (ed.),   Public        1 .Monitoring of specific proi    ects.
                        Finances (Princeton, 1961)                                            2. Medical  evaluation  of  specific  projects  in   terms   o
                    19. Feldstein, Martin S.
                        a.    "Opportunity   Cost   Calculations    in    Cost-Benefit           quality of care.
                              Analysis," Public Finance, 1964, No. 2. pp. 11 7-39.            3 .Social,    behavioral    and    economic    evaluation    o
                        b.    "Net Social Benefit Calculation and the  Public  Invest-           RMP  specific   goals   of   increased   coordination   ani
                              ment  Decision,"  Oxford   Ecoizo?nic   Papers,   March,           more equitable distribution of health services.
                              1964
                        C.    "The  Social  Time  Preference  Discount  Rate  in  Cost
                              Benefit Analysis," Economic Journal, June, 1964                                  Strategy For Evaluation
                    20. Pauly, Mark V.,  "Risk  and  the  Social  Rate  of  Discount,"        The following     breakdown  is  suggested   as   a   possible
                        American   Economic   Review   60   (1)   March,   1970,   pp.     strategy:
                        195-198.
                    21. Arrow,  Kenneth  J.   "Discounting   and   Public   Investment     Role of RAGS
                        Criteria," in A. V.  Kreese  and  S.  C.  Smith  (eds.)  Water        1.  The  setting  of   priorities   between   categories   an(
                    I   Research (Baltimore, 1966)
                    22. Baumol, W. J., "On the  Social  Rate  of  Discount,"  American           within   categories,   and   the   SUPPORT   thereof,    fo
                        Economic  Revie@v,  58  (3)  September,  1968,  pp.   788-802.           complete  end  results  medical  evaluation   of   specific
                    23. NVennberg,   John   E.,   "Cost-Benefit   Analyses-Limitations           projects   throughout   the   nation   which   PMS    feel,
                        and Uses," pp. 110-1 1  3  in  Proceedings:  Conference  Work-
                        shop on  Regional  Medical  Progra?ns,  January  17-19,  1968,         'ne  author  wishes  to  express  appreciation   to   Wflliat
                        Washington,   D.C.   Volume   11,   Public   Health   Services     Rushing, Dan Davis and Robert Metcalfe for  aiding  in  the  deve
                        Publication No. 1774 (USGPO: Washington, D.C., 1968)               opment of these comments.

                    116




           may   be   worthy    of    possible    replication    (i.e.,      social   and   economic   components)   aimed   toward    as-
           coronary care units,etc.).                                        sessing  RMPs'  specific  goals  of   more   equitable   dis-
                                                                             tribution of  health  services  and  better  coordination  of
           Role of Local RMPs                                                services. Here projects can often  be  evaluated  on  an  in-
           1. Monitoring of all projects.                                    dividual  basis  although  it  is  in  terms  of  the   TOTAL
           2. Assessment of project and  the   program's   ability   to      program's efforts (results of all projects)  that  this  type
           effect  RMP  specific  goals   of   increased   regionaliza-      evaluation is most  relevant.  Such  program  evaluation  can
           tion   and   more   equitable   distribution    of    health      only be done in the true sense in the long.run.
           services throughout the nation.                                   The social scientist's  tools  for  analysis  of  changes  in
           3. Aiding   in   those   "medical"   evaluations   that    RMPS   the distribution  of  health  services,  regionalization  and
           designates as needing such in-depth evaluation.                   cooperation  must  be  at  the   heart   of   ANY   and   ALL
           The   reasoning   behind   this   type   of   breakdown   is      attempts to evaluate local RMP programs.
           basically  that  RMPs  should  "practice  what  we  preach".      The  methods  of  measuring  RMPs   ability   to   meet   its
           In other  words,  we  preach  reduction  of  duplication  of      goals  will  be  many.  One   may   study   RMPO   -role   in
           efforts  within  our  region,  while  at   the   same   time      bringing about:
           fostering continual duplication of efforts with  respect  to      I  .Changes in functions of individual providers.
           evaluation  of  projects.  Without  having  access  to  RMPS      2. Changes in organization of providers.
           records, it is impossible to tell the extent of this duplica-     3. Changes in the accessibility of care.
           tion; however, as  one  meets  evaluators  from  around  the      4. Changes in patterns of financing.  .
           nation, it is quite discouraging to discover that  the  same      5. Changes   in   behavior   following   continuing    educa-
           type of  project  is  concurrently  being  evaluated,  often         tion courses.
           without adequate  support,  in  numerous  regions.  This  is      In addition  to  evaluation  efforts  aimed  at  judging  the
           using up substantial portions  of  RMPs  limited  resources.      program's  (and/or  project's)  achievement  of  its   goals,
           Thus,  it  would  seem  wise  for  RMPS  to  set  priorities      there is in evaluation efforts another area in  which  social
           where  in-depth  medical  evaluation  should  be  undertaken      and behavioral analyses should pay off.
           to determine whether or riot  specific  projects  should  be      That  is,  trying  to  assess  WHY  a  program  (or  project)
           replicated  throughout  the  country  with   RMPS   support-      fafls or succeeds (i.e., what are  the.  behavioral,  social,
           ing said evaluation in terms of dollars and manpower.             cultural  and  economic  forces  that  make  for  success  or
           With respect  to  evaluation,  a  paradox  seems  to  exist.      failure). There are a number of  advantages  to  this  focus.
           RMPs  are  charged  with  trying  to  act  as  catalysts  to      Foremost  among  them  is  the  ability  to  anticipate   the
           initiate change with respect to  increasing  regionalization      outcome of Project  A  (or  Program  Strategy  #I),  that  is
                                                                             in many respects quite different  from  Project  B  (or  Pro-
           and increasing  a  more  equitable  distribution  of  health
           services, which as  a  process  is  definitely  a  long-term      gram Strategy #2) 'which has  received  evaluation  -  (e.g.,
                                                                             if there are social and econon-dc forces that are related  to
           phenomenon, while at  the  same  time.  the  criteria  being      the failure/success. of a physician's assistant project,  the
           imposed by Washington for evaluation is short run.         f      same forces may be related  to  success/failure  of  projects
           It is important  to  understand  that  the  effectiveness  o      to  recruit  physicians,  or  even  the  success/faflure   of
           RMPs  must  be  measured  as  a  long-term  phenomenon   and
           in fact I would  suggest  that  if  RMPs  do  their  job  as      coronary care units).
           catalysts well,   while  documentation       of       change
           coincident to RMPs' entrance into  a  situation  or  setting
           will be possible, credit for their role  will  probably  not             The Application of Economic Analysis
           ever be acknowledged.  This  can'in  part  be  explained  by                  to Regional Medical Programs
           the difficulty and  perhaps  impossibility  of  sorting  out                       JAMES   JEFFERS
           the  changes  that  have   resulted   from   the   program's
           activities and those changes  which  have  come  from  other         INTRODUCTION
           community activities.                                                Economics  is  the  study  of  the  allocation  of  scarce
                                                                             resources  among  competing  needs  for  them.   It   is   an
                    Social Scientist's Role in Evaluation                    economic fact  of  life  that  even  our  rich  nation's  re-
           The  role  of  social  .  and  behavioral  scientists   must      sources are not sufficient  to  produce  all  the  goods  and
                                                                             services that we as  consumers  -want.  Therefore  priorities
           primarily be related to those evaluations  (the  behavioral,

           117






            have  to  be  established,  and  choices  involving   how   much because   they   are   plausible.   Many    propositions    are
            of  our  limited  resources  are  to  be  devoted  to  producing plausible, but not all are  true.  Such  statements  rightfully
            particular goods and services must be made.                      should  be  regarded   as   conjectures   or   hypotheses   and
               The real  cost  of  producing  a  quantity  of  a  particular should  not   be   regarded   as   scientifically.   meaningful
            good  or  service  is  the  value  in   consumption   of   those unless they relate specifically to  a  body  of  data  that  in
            goods  and  services  not  produced  which   could   have   been principle  could  be   examined   by   some   means   for   the
            produced   had   resources   been   used   to    produce    them purpose  of  adding  support  or  rejecting  the  existence  of
            instead of other  things.  Thus  economics  is  the  science  of the relationships proposed by the theory.
            determining:  (1)  what  needs  exist,  (2)  how  resources  can    This  is  the  point  at  which  inductive  reasonings   take
            be used most efficiently in the production of    '    goodsand   over.  The  statements  produced  by   theory   are   deductive
            services,  and  (3)   how   rational   choices   can   be   made generalizations  set  somewhat   unfirmly   on   a   foundation
            among consumption and production alternatives.                   of  assumption   and   on   some   not   so   certain   "knowl-
                                                                             edge."  The  truth  of  the  theoretical  conjectures  may   be
                      METHODOLOGY OF ECONOMICS                               presumed to  bear  no  more  closeness  to  reality  than  that
               The methodology of economics consists         larReIv   of    of the truth  of  the  assumptions  and  "facts"  on  and  from
            abstraction,  deduction,  and  induction.   By   abstraction   I which  they  are  drawn.  Thus  these  theoretical  conjectures
            mean   the   formulation   of   models.   Models   are   logical must  be   tested   against   data   purporting   to   describe
            devices  erected  on  a   foundation   of   certain   assumption reality.
                                                                        s
            and  empirical  knowledge  of  behavior,  custom,   and   insti-    In  economics  such  tests  are  usually  conducted   statis-
            tutions   and   are   welded   together   by   deductive   logic tically. While it is not usually possible  to  effect  environ-
            resulting   in   one   or   more   statements   or    hypotheses mental   control   in   sufficient   measure   to   make   data
            capable of empirical confirmation or refutation.                 conform  to  the  degree  of  abstraction   required   of   the
               The trick in  model  building  is  to  abstract  sufficiently theory,  advances  in  the  theory  of  statistical   inference
            from   reality   in   order   to   avoid    the    overwheln-dng and  econometrics  permit  a  degree  of   standardization   of
            Complexity  posed  by  the  real  world.  At  th@   same   time, variables  permitting  the  testing  of  many,  but  not   all,
            sufficient  specificity  with  respect  to  key  elements   must theoretical  conjectures.    Multivariate    analysis,    as
            be  retained  in  order  to  provide   reliable   and   relevant e.xemplified  by  analysis  of  variance  and  multiple   regres-
            deductions  as  to  how  key  variables   are   likely   to   be sion- techniques, pern-dts  the  estimation  of  the  relation-
            related and how  they  interact  in  real  world  processes.  In ship  existing   between   economic   variables   of   greatest
            a  certain  sense,  abstraction   plays   the   same   role   as interest  while  at  the  same  time  neutralizing  the  impact
            Itcontrol"  in  the  research  methodology   characteristic   of of other variables on these relationships.
            the natural and biological  sciences.  Since  social  scientists    Thus  the  final  "proof  of   the   pudding"   in   economic
            in general, and economists  in  particular,  seldom  have  an    analysis  lies   in   answering   the   question:   Do   the
            opportunity  to  "standardize"   populations   or   otherwise    hypotheses advanced on  the  basis  of  theory  square  with
            manipulate  social   conditions   with   the   exactness   of    the facts as exhibited by real world data? If the answer  is
            environmental  control  provided  by   modern   laboratories,    no and if the assumption that  statistical  design  used  in
            theoretical abstraction permits,  at  least,  clear  thinking    testing is appropriate,  the  hypothesis  must  be  rejected
            concerning   a   few   highly   important   elements   of   a    and the  theory  discarded  as  not  being  useful.  If  the
            complex system or process                                        answer to the question is yes, the  theoretical  conjectures
               The  resultant--of  the  construction  of  a   theoretical    should  remain  in  the  list  of   plausible   explanations
                                                                             suggesting  "how  things  work"  in  the  real  world  until
             iodel is the clear statement of behavior or  of  a  relation
            @,l@p  that  logically  exists  given  the  assumptions   and    such  a  time  as  subsequent  empirical  investigation  may
            empirical  knowledge  on  which  the  model  is   based.   As    refute the theory. Meanwhile the  estimated  values  of  the
            such,,  these  statements  purport  to  say  something  about    parameters of  the  relationships  identified  may  be  used
            reality and may be useful in  the  sense  that  they  provide    for policy purposes.
            a logical explanation  of  how  certain  things  of  interest       Thus  is  the  methodology  of  economics.  It  is   sum-
            work.  Very  often  they  are  convenient  ways  of  "looking    marized in Figure 1.
            at things"  and  are  suggestive  of  new  relationships  and       The  emphasis  in  economics   is   on   explaining   the
            new "ways of looking at things" as well.                         behavior of the economic aspects of  a  social  system,  and
               For the  scientific  researcher,  however,  things  cannot    therefore a premium is paid for  a  theoretical  explanation
            terminate    wit.h   accepting   such    propositions    simply  that is  consistent  with  reality  as  opposed  to  a  mere

                                                                                                                                      118




       description of reality. Pure induction involving  statistical      APPLICATION     OF     ECONOMICS      TO      PROG,RAN
       analysis,.Iet us say correlation -techniques, may  provide  a              PLANNING AND EVALUATION
       good  description,of  what  is  "going  on"   in   a   social      The   process   of   planning   involves    a    continuous
       context.  But  statistics   by   themselves   never   provide   conscious effort involving the following elements:
       answers  concerning  "why"  things  go  on   the   way   they      1.  The  specification  of  objectives  of  the  course  of
       ,'do. Data do  not  interpret  themselves,  but  rather  they
                                                                       action being considered.
       must be interpreted @vithin,a  context  of  logic  involving.      2. The    specification   of,   alternatives    by    which
       cause  and  effect  relationships.  Thus  interpretation   of
                                                                       objectives may be obtained.
       economic  phenomena  is  facilitated  by  a  clear  statement      3. The  collection  and  interpretation  of  relevant  data
       of.  the  theoretical  relationships  that.logically  may  be   and information.
       expected  to  exist.  This  logic  is  incorporated  in  what      4.  The  specification   of   the   potential   costs   and
       economists  mean  by  a  model  which,  as  explained  above,   benefits of each alternative means of reaching          each
       is  merely  'an  abstract  prototype  of  how  key  variables   objective.
       may be expected to be related in the real world.                   S.  The  development  of  a  model   that   abstracts   the
                                                                       relevant features of the situation being considered.
         In   my    experiences    many    medical    administrators      6. The specification of a  decision-rule  or  criterion  by
       underempha-size  the  importance  of  a   clearly   specified   which it is possible to rank alternative  ways  of  Attai
                                                                                                                               @ing
       theoretical  model  prior  to  the  collection  of  data.  In   objectives in order of their desirability.
       many cases g reat haste is made to  collect  data  without  a      Effective evaluation  is  also  a  continuous  process  and
       clear conception as to  how  the  data  may  be  analyzed  or   differs from planning in the following- respects:
       interpreted to provide  answers  to  questions  essential  to      1.  Alternatives  are  not  considered  in  the  course  of
       program planning and evaluation.                                evaluation since a course of action for  attaining  a  desired

                                                 FIGURF, l.-Economics Methodology



                                                                                 7. If Negative
                                                                                    Reject Theory





                                                                                                         5. Statistical
               1. Real
                  World                                              6: Data                                Tests   and    Estimation
                                                                                                            of Parameters





                                                                               7. If Positive
                                                                                  Accept Theory                4.   Formulation    of
                                                                                  and Use Parameters              Statistical
             2.  Economic                                                                                         Tests
                                                                                  Estimate
                 Model

                 Abstraction
                 Assumptions
                 Empirical                                         3.    Hypotheses
                 Knowledge                  Deductions                 Concerning
                 Logic                                                 How Real
                                                                       World "Works"



                                                                                                                                    119





                 objective -has already     been selected.     (However,     after       interval must be reasonable and initially      can only       be
                 evaluation   has   been   performed,,it   may   be   decided   to       determined or estimated  on  the  basis   of   the   experience
                 terminate    a  particular  program       in   favor   of    some       of other programs  conducted  elsewhere.  or  on   the   basis.
                 alternative);                                                           of expert opinion
                    2. Costs   and benefits are measured       in.  actual  rather       Economists  can   be   of   some   assistance   in   developing
                 than in potential terms;                                                statements   of   objectives.   Economists   can   point    out
                    3.  The  model  abstracting  the  relevant  features  of   the       objectives  that  are  conflicting  and  can  assist   in   the
                 situation may be     'modified  in  light   of   experience,   the      development   of   quantitative   statements.   However,    the
                 accumulation of data, or refinement in its design; and                  ultimate  responsibility  for  doing  so  lies  with  regional'
                   4.   The   decision   rule   or   criterion   adopted    should       advisory groups.
                 apply   consistently   for   all   implemented    programs    for       Given    an     appropriate     statement     ofI   objectives,
                 purposes  of  assessing  their  relative  contributions  to   the       economists  can  m@e  a  very   significant   contribution   to
                 overall objectives of the program.                                      the  evaluation  process  in  the  areas  of   modeling,   data
                   One  of  the  prime  requisites  of  effective  evaluation   is       collection, and the analysis -of data.
                 the  statement  of   the   objectives   of   a   given   program.
                                                                                         COST-BENEFIT               AND               COST-EFFECTIVENESS
                 Statements  of   objectives'@hould   not   be   too   broad   and
                                                                                         MODELS
                 imprecise,   should   not   be   conflicting,   and   should   be
                 stated  in  quantitative  terms  whenever  possible  so   as   to       There' are many     specific  models      which      economists
                 facilitate both planning and evaluation.                                have developed  over  the  years  that  would  be  us eful     to
                   A  much  too  broad-  statement   of   an   objective   for   a       program  evaluation.  They  are  too   numerous   to   describe
                 regional  medical  program  would  be:  to   reduce   the   pain,       in the space allotted.  Therefore  at  the  risk  of  dmittiiig
                 suffering,  and  mortality  of  heart  patients   living   within       many models that  may  be  of  interest  for  the  purposes  at
                 the  boundaries  of  the  region  of  consideration.'The   state-       hand, I will briefly describe the  one  that,  in  my  opinion,
                 ment  is  much  too  broad   since   any   coronary   care   pro.       is particularly useful. This.  is  the  cost-benefit  or  cost-
                 gram,  be  it  one  of  continuing  education  or  one  involving       effectiveness  model.  It  is  particularly  useful  for  eval-
                 the  use  of  a  mobile  coronary  intensive  care  unit,   would       uation   purposes   since   in   principle   it   permits   the
                 conform.to  the  objective,  and  it  would  be   impossible   to       simultaneous  evaluation  of   the   performance   of   several
                 judge the relative efficacy of these two programs.                      different   operational    programs.    Economists    generally
                                                                                         regard  cost-benefit-  analysis  as  an  offspring  of  welfare
                  An  example   of   a   conflicting   statement   of   objectives       economics   and   public   finance,   although    the    first.
                 might  be:   to   reduce   the   morbidity   and   mortality   of       practical applications of the technique   -were-    made     by
                 coronary  disease  in  a  given   region.   This   statement   of
                                                                                         engineers in this country  around  the  turn  of  the  century.
                 objectives   is   conflicting   because    the    reduction    of
                 coronary  mortality.may  well  raise  the   average   number   of       ii.owever,  economists   had   initially   developed   the   tech-
                 heart  attacki  experienced  by  many  patients,  thus   raising'       ruque   in   the   middle   1850's'   and   had   refined   the
                                                                                         principles of the methodology by  the  early  1950's.
                 morbidity  in   statistical   terms.   Clearly,   reductions   in       In essence,  cost-benefit  analysis  is  a  way  of  evaluating
                 morbidity  and  mortality  are  desirable,  but  it   should   be       the desirability of a project or of  a  set  of  projects  when
                 recognized  that   these   objectives   are   conflicting.   They       it is important to view project activities  over  a  long  time
                 should  be  stated  separately,  and  decision  makers  must   be       span where there are likely to  be  many  spfll-over  or  side-
                 prepared  to  compromise   between   the   attainment   of   both       effects  on  people,  other  programs,  and  other  activities.
                 'objectives since they are in conflict.
                                                                                         In  simplest  terms  the   method   consists   of   a   careful
                  Nei@e@    of    the    statement    above    are    sufficiently       enumeration of  all  direct  and  indirect  elements  of  costs
                 quantitative in that they  fail  to  clearly  relate  to  a  body       and benefits.  It  should  be  noted  that  when  benefits  and
                 of data  that  may  be  examined  in  the  interest  of  planning       costs  are   not   measured   in   comparable   units   (e.g.,.
                 and   Fvaluation.   A   better   statement   would   include    a       dollars), 'the technique is usually, but  not  always,  labeled
                 specification  of  the  extent  to  which  improvement   in   the       cost-effectiveness   analysis-thus    explaining    the    dif-
                 condition  of   pa'tients   is   expected.   An   example   of    a     -ference in. the terms  used  to  title  this  section  of  the
                 better'statement  would  be:   to   reduce   the   morbidity   of       paper.  The  phrase  "cost-benefit  analysis"  will   be   used
                 coronary.  heart  disease  by  "X"  percent  over   a   specified       throughout the remainder of this paper
                 time  interval.  Of  course,  the  specification  of  the   exact       Cost-benefit  analysis  involves  a   comparison   of      costs
                 percent  of  reduction  of   moribidity   or   the   exact   time       and benefits associated with a program- or set     of      pro-

                 120





                                                                                               an  consider  the  case   for   a   set   of   projects
                                                        clusions         con-     Now we c
       their relative  significance,  and draw con                                                                  . :"  11  .
                           of the patient's illness.                             which'can be  ' rt        e         varying   levels   rather    than
                                                                                               pa   icipat   din   at
       cerning the nature
       -grams, where  the  latter are  viewed as       alternatives        or    in an absolute fashion. Attempt, is made to achieve:
       competitors  for  overall   program   funds.   The   cost   side   of.
       the   equation   consists   of   estimated   or    realized    program
       expenditures   as   itemized   in    program    budgets    with    due                                       where  subscripts  1-n  represent_
       allowance   for   the   real   costs    of    resources    voluntarily                                       different projects.
       contributed  to  the  project  effort.  In   general,   benefits   are    mb,    nib2              nibn      Mc is the  marginal  cost  of  the
       viewed   as   future   losses   that   will   be   avoided   by    the    -      -
       success   of   -programs.   The   major   purpose   of   health   pro-    MCI    mc2               Men       ith cure.
       grams  is  to  save  lives  and  reduce  fllness'  There   are   three                                       mbi is the marginal be     nefit)of
       general  categories  of  benefits:  (1)   gains   in   economic   out-                                       the ith cure.
       put (usually measured in terms o       f   income),   (2)    satisfac-
                                                                                   It   is   profitable   to   participate   in   a   program    until
       tions  from  improved  health,  and  (3)  savings  in   the   use   of    mbn > I + i; that is, as long as benefits achieved           .are    i
       health resources.                                                         m@n -
          Before  going  further  it   should   be   noted   that   some   of    (where  i  represents  the  discount   rate)   times   greater   than
                                                                                 the  cost  of   producing   the   benefit.   With   this   considera-
       the   differences   among   authors   as   to   how    they    measure    tion  in  mind,   the   optimally   sized   regional   medical   pro-
       benefits are due  to  differences  in   the   availability   of   data    gram  budget  is  one  which  allows  that  all  projects  a   reiibn
       and   do   not   reflect   philosophical   differences   as   to   the
       appropriate   use    of     the     methodology.     However,     some    wishes  to  undertake  are  participated  in  to   the   level   that
                                                                                 the  return-from  each  project   is
                    of   philosophy   do   exist    even    if    the    same
       differences
       authors   had   access   to   identical   data.   Mention   will    be
       made of tms-later in the paper.                                                     mb 1  =  mb                 mbn          +
          Once   having   enumerated    all    types    of    benefits    and                            2
       costs,   usually   some   sort   of   discounting    technique    must               MCI      mc2               Men
       be  discounted   by   an   appropriate   interest   rate   to   adjust
       comparable.   This   is   because   benefits   are   likely   to    be                                       L DIFFICULTIES
                                                                                         SONIE              CONCEPTUA
       realized  over  an  appreciable  period  of   time   and   costs   are
       usually incurred in the present.                                            The   first   conceptual   problem   that   one    encounters    is
          For   example,   the   benefits   of   a   program   designed    to    in   developing   appropriate   measures   of   benefits.   One    is
       save  lives  may  be  measured  by.  the   earnings   of   individuals    tempted   to   measure   what   appears   to   be    objective    and
       whose  lives  are  saved  over   the   period   during   which   their    reproducible   at   the   expense   of   other   benefits   not    so
       lives  have  been   extended.   Since   such   earnings   extend   for    easily  measured.   The   economic   gains   of   saving   lives   is
       a  significant  tiyne  in  the   future   the   income   stream   must    usually   measured   by   taking    account    of    the    increased
                                                                                 income   stream   forthcoming   to   the   individual   whose    life
       be  discounted   by   an   appropriate   interest   rate   to   adjust
       future   earnings    downward    rendering    them    comparable    to    was  saved.  This  is  tantamount  to  saying  that  the   value   of
       costs-th@t  are  incurred  in  the  present.   The   choice   of   the    a   man   is   what   he   earns   and   neglects    the    affection
       appropriate    interest    rate    is    as    yet    an    unresolved    accorded  to  the   aged   who   have   lived   a   productive   life
       theoretical   issue   and   thus   in   most   applications    several    and  who  are  retired  and  who   are   no   longer   employed.   As
       interest  rates  are   used   resulting   in   alternative   estimates    yet   a   satisfactory   measure   of   the   loss   of    a    "non-
                                                                                 productive"   member   of   society    has    not    been    devised.
       of the discounted benefits of each program.                               Similarly no indices of the welfare       .gains    stemming     from
          Once   having   measured    costs    and,benefits    for    several
       different   projects   we    can    make    a    comparison    between    reduced  pain  and   suffering   exist.
                                                                                    Even   if   income   or    earnings    are    adopted    as    the
       them.  If  we  are  faced  with   selecting   one   project   to   the    appropriate   measure   of   benefits,    questions    remain    con-
       elimination  of  all  others,  -the  analysis  is  simply   a   matter
                                                                                 cerning   whether   income   net    of    consumption    should    be
       of  determining   which   project   has   the   largest   benefit   to
       cost   ratio   and   implementing   that   project.   Note   that   no    the measure or whether gross income should be used.
       provision  is  made  for  the  project  with   a   benefit   to   cost
       ratio'   of   less   than   one.   Such   a   project   would   not    be--                       CONCLUSIONS
       undertaken  since  the  returns   to   such   a   project   would   be       Mention   of   these   problems   serves   to    underscore    the
       exceeded by the costs of the project.                                     fact   that   economic   models   in   general    and    cost-benefit
                                                                                                                                                     121





                  analysis   in   particular   cannot   provide   easy    objective    odology   for   evaluation,    we    command    some,potentiall,
                  answers  to  all  questions  involved  in   program   evaluation.    powerful   techniques   for   this   purpose,   but    we    ham
                  However  one  of  the  major   contributions   of   an   economic    harnessed very little  of  their  pron-dse  in  a  systematic  o
                  model  is  to  systematically   categorize   the   key   economic    organized way..
                  issues,  variables,  and   relationships   that   are   involved.       Like heart disease,  cancer,  and  stroke  we  seem  to  lacl
                  Once  these  have  been  set  out,   analysis   using   objective    the ability to relate the various pieces of th      e   technica
                  data  provides  guides  as  to  appropriate  decisions.  Even  if    competence   we   command    to    pursue    evaluation    irit(
                  complete  answers  cannot   be   provided   on   the   basis   of    meaningful    total    arrangements.    The    involvement     o
                  objective  data  and  analysis,  a  systematic  specification  of    social  scientists  in  the  evaluation   of   regional   medica
                  the   evaluation   problem   coupled    with    what    objective    programs is likely to  prove  productive  only  to  the  exte  @
                                                                                                                                                      n
                  evidence  is  available  facilitate  the  consistent  application    that  -there   is   widespread   understanding   and   conceptua
                  of  judgment   and   expert   opinion   so   vital   to   correct    clarity  on  the  part  of  program  administrators  about   th(
                  decisions.                                                           evaluative process.  Social  scientists  on  the  basis  of  th(
                                                                                       particular  skills  they  possess  are  in  a  position  to  con
                                      Summary     of     Remarks                       tribute  to  the  evaluation   of   on-going.programs.   However
                                 JOHN E. WENNBERG, M.D.                                their relevant role  is  restricted  and  confined  to  certair,
                                                                                       discrete levels of the process of evaluation.  !7urtiler,  theii
                     A  successful  health  planning  and                management    entry   into   the   process   most   often   presupposes    the
                  capability  requires  the  development  of   an   adequate   data    exercise of a great deal of prior normative judgment.
                  base.  This   should   be   approached   through   the   use   of       In order to use  social'scientists  in  appropriate  ways  in
                  multiple  disciplines   in   both   the   design   and   analytic    the evaluation  of  social  programs,  it  is  necessary  to  be
                  phases.   Relevant    disciplines    include    biostatisticians,    clear about the  different  levels  of  the  evaluative  process
                  epidemiologists,    economists,    sociologists    and    systems    and  about  the  underlying  values  which   assert   themselves
                  analysts.                                                            in  any  particular  program  under  review.  We  must  be  able
                     The   NNE/RMP   has   developed   a   planning    and    eval-    to  specify  the  purposes  to  be  served  by  evaluation   and
                  uation  base  by  assembling  existing,  data  sources   into   a    the  criteria  of judgment  that  are  reflected        in   the
                  compatible,    computer-based    system.     T    ata  ase  .  as    formulation of those purposes.
                  been   supplemented   by   ad   hoc   field   studies   involving      Prograi-n  evaluation  is  predicated  on  various   essential
                  retrospectively  collected  utilization   data   and   facilities    assumptions,  however  obvious   these   may   appear   to   be.
                  inventory.   In   addition,   a   complementary   field    social    It is necessary, for  example,  to  accept  the  belief  that  a
                  -survey capability has been organized.                               program   embraces    purposive    activity,    that    socially
                     Details  concerning  the   data   system   are   reported   at    valued  resources   are   deployed   with   inte'fit in  order  to
                  another  conference  session.  Here  I  would  like   to   report    accomplish  something.  Programs  must  have   goals   if   they
                  by   way   of   example   how   socioeconomic   analysis,   using    are to be evaluated. We are also  sensitive  to  the  fact  that
                  information  in  the  data  base,  can  help  clarify,   if   not    resources are limited.  At  a  time  when  the  availability  of
                  answer, certain questions of concern to planners.                    resources  appears  to  be  beconu'ng   progressively   tighter,
                  i   The   questions   chosen   for    example    include    those    this is another  premise  which  is  easy  to  accept.  Programs
                  related  to  the  cost  of  care  and  consumer  preferences  and    accordingly  reflect  the  exercise  of  some  form  of  ration-
                  opinions.  The  importance  of  these  questions  to  the   plan-    ing. The first essential task that we face  in  the  process  of
                  n-ing process vAll be emphasized.                                    evaluation is therefore to ascertain t   he  extent   to   which
                                                                                       our  programs  are  accomplishing  the  goals  which   we   have
                                   Social  Scientists  and   the                       set  for  them.  Programs  consist  of  a  bundle  of  more   or
                                     Process of Evaluation                             less discrete  projects.  If  the  planning  of  a  program  has
                                    CONRAD     SEIPP,      Ph.D.                       reached  an  acceptable  degree  of  precision,  each   of   its
                                                                                       constituent  projects  possesses  a  clearly  defined   set   of
                     The  field  of  evaluation  is  like  the   field   of   heart    targets. A target is a statement  of  the  end   results   which
                  disease, cancer, and stroke. In both there is  a  serious  gap  -    are sought through    the activity  that  is  called  for  in  a
                  between what  we  are  able  to  do  and  what  we  are  in  fact    project. It identifies the amount oC@-co-mp-lis-h--m-e-nts-,  if-
                  today   doing.   We   know   a   good   deal   more   about   the    possible  in  quantitative  terms.  to  be  achieved  within   a
                  process of evaluation  than  current  practice  suggests,  it  is    specified period of time. A number of - proj         ects   'are
                  my contention. There      is  a   substantial   body   of   meth.    collectively  the  means  for  achieving  the  objectives  which

                  122--





                                                                                                            efficiency lies in            reducing
                                                  the first level   of           its efficiency. The quest for  ,     zin  the           resources
    ,are set for the total program. Thus,                                                 it of output, of mini.mi       9
                  is   in   principle   at    least    relatively    simple    and inpifts per un                            t   or   a   set   of
           evaluation                                                            which must be expended to obtain a: targe
           clear.   It   is   to-   measure   the   extent   to   which    varions
                                                                                 objectives. In the  case  of  effe  tiveness,  we  want  to  know
           projects are meeting their targets.                                                             'c
                                                                                 how  much  we  are  getting  as  return  on  the   resources   we
           It is an integral   part'   of   the   responsibility    of    the    ad- are expending. These must.be seen as    separate     ptoblei-ns_
           ministration  of  a  program   to   ascertain   the   extent   of   the
                                                                                 to be  dealt  with  in  the  process  of  evaluation.  The  eval-
           progress  that  is  made  in  fulfilling  project  targets.  There   is uation  of  a  program  entails  analysis   along   both   lines.
           no  point  in  becoming   involved   in   other   levels   of   evalua- How  the  resulting  intelligence  is  to  be  Assembled  into  a
           tion  unless  this  kind  of   intelligence   is   at   hand.   Perhaps comprehensive   assessment   of   a   project   or   a    program
           the  particular  tasks   and   routines   upon   which   reliance   has
                                                                                 depends  upon  the  assumptions   and   the   suppositions,   lye
           been placed in  the  planning  of  some  project  came  to.be         bias,  if  you  like,  which  is  incorporated  in  it.   Program
           viewed   as   inadequate   on   their   definition,   their   organiza-
           tion  or  their  implementation,  but  it   makes   little   sense   to evaluation  should  conform  to  the  nonns  and   t@e   crite@ia
           evaluate   the   adequacy   of   these   unless    there    is    firm- of  judgment   which   are   manifest,   however   co@d@tly,   in
           knowledge   of   where   a    project    stands    in    meeting    the the  planning  and  the  design  of  a   program,   even   though
           targets  whiclihave  been  set  for  it.  Similarly,   it   is   point- the  evaluation  that  is  done  of  a  particular   program   by
                                                                                 others    may    be    predicated    on    different    ndKmative
           less   to   attempt   to   assess    total    program    accomplishment grounds.
           until  the   extent   to   which   the   targets   of   the   component
                                                                                 I  The  thrust  of  these   comments   is   to   underscore   the
           projects are being met has been acertained.
                                                                                 importance of  clarity  about  the  values  that  inspire  evaiu-
           Buried  in  the  targets  of   a   project,   however,   are   a   host
           of   value   judgments   which   need   to   be   made   explicit    if ative-  effort.  Evaluation,   involving   th@   measurement   or
                                                                                 assessment   of   program   accomplishment,   proceeds   on   the
                 n   is    to    be    pursued    at    a    higher    and    more      ertain   standards   of   comparison    and    particular,
           evaluatio                                                             basis of -c
           inclusive  level  of  concern.  Program  people   in   the   field   of normative  criteria  of  judgment  which   are   current   in   a
           health   and   medical   care   still   speak    of    securing    -the program   and    these    must    be    understood    and    made
           greatest   possible   return,   however   this   niay   be    measured, explicit. The social scientist who  is  involved  in  the  evalu-
           for   the   least   expenditure   of   socially    valued    resources.                                           portant       contri-
           One   knows   what   they    mean    when    they    say    that    @e@ ation of programmatic endeavor has an im
           want  to  get  the  most  for  the  least.  However,  this   kin(i   or bution   to   make   in   exposing   and    laying    bare    the
                                                                                 construct  of  values  which  are  reflected  in  a.   particular
           formulation   of   the   economizing   intent   of   a    program    is
                                                                                 program.  The  need  to  insure  a  continuing   explication   of
           inimicable to evaluation.  It   must   be    challenged    if    evalua value  pren-dses  is  not  only  a   requisite   for   meaningful
           tion  is  to  proceed.  For  the   most   is   in   theory   infinitely                                                               f
                                                                                 of evaluation;  it  must  also  be  made  an   inherent   attributes
           great   and   the   least   is   zero,   and.   this   makes   nonsense program  planning.  This  is  the  point   at   which   planning,
           their.concern.     A.     program     administrator     is     motivated evaluation   and   research,   meaning    evaluative    research,
           either  to   maximize   output   the   desired   end   results   of   a
                                                                                 operations  research,  administrative  research,  can   it   what
           course   of   action,   with   a   'given   input   or   he'wants    to
                                                                                 vou  will,  emerge  most  explicitly  as  aspects  of  a   single
           secure  some   specified      accomplishment      with      the       -
                                                                                 function.
           minimum  expenditure   of   socially   valued   resources.             The'  ways  in   which   the   social   scientist   is,currently
             Those responsible     for a  program     are     most'o     n,     in involved  in  this  aspect  of  program  evaluation  is  at  best
                                   o   accomplish   as   Much    as    they    can shadowy  and  uncertain.  The  relevance  of  Ws  skill  at  this
           fact, motivated both   t
           with the resources at   their   disposal   and   at   the   same   time level  of  concern  needs  to  be  more.fidly   appreciated   and
           to   reduce   what   is   required   to    achieve    the    objectives the  role  which  he   potentially   can   play   requires   more
           which    they    entertain.    The    evaluator,    however,     cannot -definitive  delineation.  The  credentials  which   the   social
           simultaneously    oursue    both    concerns,    for    they    consti- scientist  commands   to   enhance   the   sensitivity   of   the
           tute   separate   and   discrete   analytical   tracks.    Each    must staff  of  a  program  to  the  value   implications   of   their
           be  independently  assessed  as  part   of   the   process   of   eval- actions  are  none  too  solid   or   convincing.   His   contri'
           uation.  Further,   the   evaluator   must   ascertain   the   relative bution  in   this   regard   is   surrounded   with   difficulty.
                                                                    a         pro- Further,  the  more  penetrating  and   critical   he   is,   and
           importance   to   be   attached   to   each    in    a    particul    r
                                                         udgmen't            which thereby the  more  useful  the  less  appreciated  he  is  likely
           gram.  This  rests upon a normative j
                                                    el    evaluation.            to be.
           constitutes a given at this second lev
                                                      the          differentiation  The task which  I  am  suggesting  for  the  social  scientist
              Correlative to this distinction is
           between,  the  effectiveness  of  a   program   or.   a   project   and at this level of concern   is  to  ask  those  administering  a
                                                                                                                                                 123





                program  why  they  are  doing  what  they  are   doing,   what    the assistance of a social  scientist.  There  do  not  appear
                evidence  they  possess  to  validate  the  assumed  worth   of    to be any  compelling  reasons  to  suppose  that  the  social
                those  actions,  and  how  they  see   the   consunnnation   of    scientist has a unique contribution  to  make  in  such  tasks
                particular tasks and  activities  as  related  to  the  attain-    as  ascertaining  whether  the  development  of   a   coronary
                ment  of  the  broader   objectives   of   their   program.-The    care  unft  in  a  hospital  is  on  schedule  or  determining
                social  scientist   is   hopefully   equipped   somewhat   more    where things  stand  in  instituting  a  tumor  registry.  The
                adequately  than  others  to  reco  nize  the  'ways  in  which    same applies if  the  evaluative  concern  in  regard  to  the
                                                  9
                diverse values assert themselves  in  a  program  and'  to  ap-    tumor  registry  is  less  proximate  and  centers   upon   an
                preciate  the   various   social   roots   of   the   normative    assessment of its consequences  or  impact.  If  the  planning
                judgments that are  reflected  in  the  activity  he  observes.    of  the  program  has  been  adequate,  the  problem   is   to
                His presence first of all  may  help  to  make  this  dimension    determine the extent to which the     project did in fact  fit
                of  a  program's  endeavor  more  explicit.  He  is   able   to    into  the  larger  scheme  of   the   regional   endeavor   as
                assist   others   in   identifying   and   acknowledging    the    intended. Very possibly the  talents  of  a  social  scientist
                normative   premises   upon   which   action   is   based,   in    might usefully  be  drawn  upon  if  the  issue  that  emerges
                recognizing  the  existence  of  forces  which   Militate   for    in  the  course  of  the  process  of  evaluation   comes   to
                alternative  standards  of  judgment,   and   in   @   exposing    center upon the efficacy or  the  validity  of  the  technical
                inconsistencies between  the  value  base  of  different  parts    prescriptions  that  a  program  has   made   to   achieve   a
                of a program. In  this  respect  the  social  scientist's  role    particular end result. Y@t  this  type  of  concern,  I  would
                within  a  program  is  essentially  one   of   education;   it    argue, should  not  be  included  as  a  primary  function  of
                involves increasing the self-consciousness of the  staff  of  a    program  evaluation.  Rather,  it  should  be  considered   as
                program  about   the   social   forces   which   impinge   upon    an assignment for evaluative  research  which  relies  upon  a
                them and of which they are a part.                                 different institutional base and set  of  resources.  Regional
                                                                                   medical programs will inevitably      become    involved    in
                  The  social  scientist  can  obviously  make   no   exclusive
                                                                                   such activities but not, as I see it, as the agents  who  have
                claims to such a role. Yet  he  is  in  a  position  to  deploy    a  primary  responsibility  for  undertaking  such   analysis.
                the  special  competence  he   is   assumed   to   command   in    Rather, they should be  a  part  of  a  larger  consortium  of
                clarifying  the  normative  bias  of  a  program,  particularly    concern that is involved in the  pursuit  of  such  questions.
                as  it  is   expressed   in   the   functional   linkages   and    'De  social  scientist  does  appropriately  come  back   into
                relationships  which  the  program  generates.   In   this   he    the process of program evaluation  in  the  appraisal  of  the
                helps to  expedite  the  process  of  evaluation  at  the  same    broader and less  specific  objectives  of  a  program.  Here,
                time that  he  contributes  to  the  course  of  planning.  His    for example, one  encounters  the  need  to  evaluate  success
                contribution, if he functions with  effect,  is  to  facilitate    in   promoting   the   legally    mandated    obligation    to
                the  formulation  and   appreciation   of   a   clearer,   more    promote cooperative arrangements  as  an  end  in  itself  but
                meaningful  design  of  the  interrelations  between  ends  and    I
                means.  Each  project,  I  have  suggested,  should   have   an    aiso  at  the  same  time  to  see   those   arrangements   as
                explicit  target,  an  end  result  which   has   been   opera-    instrumental to  improvements  in  the  health  care  delivery
                tionalized   as   a   'i-neasurable   accomplishment   to    be    system.  However,  at  'this  level  of   concern   the   per-
                ackieved  within  a  specified   period   of   time.   However,    formaiice  of  those   social   scientists   who   have   been
                each project must also  be  seen  as  the  means  for  attainin    involved in the  evaluation  of  programs  is  far  too  often
                                                                              9    disappointing.  There  is  a  gross  disparity  between   per-
                the  objectives  of  a  program.  Further,  the  place  of  the    formance  and  promise.   Social   scientists   tend   to   be
                program as a part of the endeavor to realize  the  'aims  of  a
                     '   inclusive   health    plan    must    be    adequately    mesmerized  by  a  conviction  in   experimental   design   as
                more                                                               the  only  road  to  salvation  and  they  are  reluctant   to
                visualized. This is the essential  conceptual  matrix  for  the    abandon  the  rigor  and  the  apparent  certainty  that  such
                conduct of effective evaluation.
                1.                                                                 procedures  imply.  Only  slowly  and  with  great  pain   are
                 oGiven   an   adequate   spell-out   of   this   kind   of   a    they learning of  the  tremendous  practical  difficulties  of
                hierarchy  of  goals  and   of   the   interrelations   between    hiiposing  experimental  designs  upon  on-going  social  pro-
                them,   the   problem   of   program    evaluation    is    not    grams. Yet there are  also  theoretical  grounds-for  suggest-
                especially  complex,  it  seems  to  me.  Assessment   of   the    ing  that  excessive  emphasis  has  been  placed   upon   the
                extent  to  which  targets  are  achieved,  even  the  measure-    controlled  experiment  as  i-nethodologically  essential   in
                mciit  of  the  accomplishment  of  program   objectives,   can    the  evaluation  of  programs   with   broad   and   ambitious
                and should  proceed  without  any  particular  need  to  enlist    aims.  These  are  as   compelling   to   me   as   the   many

                124



         obstacles  which   are   encountered   in   practice   in   trying   to of    evaluation    some    methodological    alternatives     for
         execute an experimental design.                                       analysis at this level of the process.       s     fully      Upon
         Let   me   with   desperate   brevity   identify    some    of    those If  the  social  scientist  can't  rely  a
         problems.   It   is   extremely   difficult   for   those   responsible experimental  design  as  he  would   like,   what   methods   and
                                                                               procedures   can   he   appropriately   pursue   in    evaluatiti-
         for   a   social   program   to   establish   and   maintain    control                                                                C,
         groups,    since    this    entails     withholding     the     assumed social   progranis9   Admittedly,   he   does   not   have    much
         benefits    forthcoming    from    some    intervention    for     some that  is  firm  to  fall  back  upon,  but  there  may   be   more
         individuals  or  groups  at   the   expense   of   others.   Not   only than  is  currently  acknowledged.  I  would   identify   a   body
         is  such  practice   often   difficult   to   justify,   but   problems of  effort  that  has  come   to   be   identified   as   "process
         are   also   frequently   incurred   in   trying   to   enforce    such analysis." It is also possible to  point  to  the  successful  use
         discrimination.   Many   courses   of    action    are    simply    not that  has  been  made  of  the  critical  incident  technique,  an
         divisible  so  that  the  possibility   of   controls   is   precluded. ad   hoc   kind   of   appraisal   of   developments    as    they
         Another  issue   is   the   growing   resistance   on   the   part   of actually  happen  to  unfold.   Much   more   can   be   made   of
         various   segments   of   the   consumer   public   to    serving    as investigations   of   the    communicative    network    to    en-
         subjects  for   research.   Many   are   not   impressed,   to   under- compass   a   systematic   appraisal   of    the    volume,    the
         state  the  point,  with  what   they   deem   to   be   the   academic quality  and  the  pathways   of   exchanges   and   transactions.
         games  which  the   research   worker   engages   in   and   they   are At  this  stage  of  things  we  must  resign  ourselves  to   the
         unwilling  voluntarily  to  offer  themselves   as   a   statistic   in necessity  of   being   selective   and   of   concentrating   our
         the   promotion   of   a   professional   career.   Yet    even    more attention   on   certain   indicator   states   and    conditions,
         important   is   the   friction    which    the    conduct    of    ex- rather  than  the  entire  range  of  potential  consequences   of
         perimentally   designed   evaluation   studies    tends    to    create a  program.  One  important  area  to  look  for  help  from   the
         between   those   responsible   for    the    administration    of    asocial  scientist  in  developing  such  indices  relates  to  the
         program  and  the  social   scientist   who   is   involved   in   such assessment   of   satisfactions   and   the   crossmatch   between
         investigation.                                                        measures   of    provider    and    consumer    perceptions    and
          From   the   beginning    of    a    program,    administrators    are aspirations.  These  are  examples  of  some  of  the   kinds   of
         likely  to  seek  advice  and  assistance   from   the   evaluator   in analytical   concerns   and   techniques    which    the    social
         refining   the   desicn   of   the   program   and   in    implementing scientist  can  bring  to  the  evaluation  of  the  broadest  and
         the   plan   as   it   is   formulated.   However,   this    inevitably most open-ended objectives of your programs.
         commands   second   priority   to   the   evaluator   who    is    com- The   hope   of   enlisting    social    scientists    in    such
         mitted   to   a   protocol   which   involves   experimental    design. endeavors  depends  to  an   important   extent   upon   the   ef-
         Once  he  has  established  his   controls,   it   is   necessary   for fectiveness  with  which  they   are   involved   in   the   other
         him  to  try  to  hold  a  program  frozen   in   order   to   preserve levels  of  evaluative  effort  which  I  have  tried  to  charac-
         the   continuation   of   the   experiment    he    has    constructed. terize.  They  must  be  given  the  opportunity  to   learn   for
         Good   planning   from   the    viewpoint    of    the    adiwnistrator themselves  that  the   broader   the   aims   of   the   programs
         may   involve   maintaining   the   greatest    possible    flexibility they  engage  in  evaluating,  the  more   inadequate   is   their
         in   order    to    maximize    opportunities    to    capitalize    on penchant  for  experimental  design  likely  to   be.   The   more
         unforseen   options.   But   good   evaluation   in   the    eyes    of rapidly  they   are   able   to   recognize   the   necessity   of
         the  research  workers  seeking  scientific   rigor   is   certain   to relying  upon  less   rigid   evaluative'   methods   and   proce-
         n-fflitate   against   departures   from   a    predetermined    course dures,  the  sooner  they  will  see   their   way   to   relevant
         of   intervention.   The   exclusiveness   of   these   concerns    all endeavors  which  challenge   them   not   as   mere   technicians
         too  often  eventuates  in   mutual   recrin-dnation,   if   not   out- but  as  seekers  of  new  knowledge.  Two   things   are   neces-
         right disaster.                                                       sary  if  this  is  to   come   about.   The   administrators   of
           Where   the   context   and   the   environment    of    a    program programs   must   become   more   certain   about    the    impor-
         are   highly   fluid   and   dynamic,   where   objectives   are   far- tance  of   evaluation   and   they   must   acquire   a   greater
         reacfiing   in   their   consequences   and   are   hard   to    opera- confidence  about  their  role  in  that   process.   The   social
         tionalize   or   specify   with   precision,   where   the   unintended scientist,  in  turn,  must   become   more   modest   about   the
         results  of  interventions  are   felt   of   major   importance,   the potential  of  his  claims  in   regard   to   evaluative   under-
         program   administrator   would   do   well   to   be    critical    of takings   and   seek   to   develop   methods    and    procedures
         what   the    social    scientist    can    accomplish    within    the which  can  supplement,  at  some  critical   points,   substitute
         framework   of   experimental    design.    The    social    scientist, for  the  design  of   experiments   upon   which   he   has   too
         in  turn,  should  try  more  consciously  to   bring   to   the   task often and too unrealistically come to depend.
                                                                                                                                                125





                               Cost Benefit-Cost Effectiveness                      ing ship, or on a desert expedition)  or  two  65  year  old
                               Studies, and Their Application to                    Other things being equal,  the  "life-years  saved"  princip
                                    Allocation of Resources                         should lead you to opt  for  the  alternative  involving  t]
                                   ROBERT L. BERG, M.D.                             most life-years saved. In this case  (using  1967  data)  ti
                                                                                    maximal  number  of  life-years  saved  would   result   fro
                     Making  choices  among  alternate  proposals  in  the   RMP    saving the 40 year  old  (34.3  fife-years  versus  2  x  14
                  characterizes   something   rather   new   and   uncommon   in    years = 29.6 life-years for the 65 year olds). This leads  i
                  the health field: the acceptance  that  there  are  limits  to    an  action  favoring  one   life   saved   over   two   liv,
                  what can be spent in  money  or  in  specific  resources  such    saved,-troubling to all of  us  committed  to  doing  all  v
                  as hospital  days,  or  doctors'  or  nurses'  time.  In  many    can  for  our  patients,   but   probably   comparable   wil
                  respects,  health  care  is  dispensed  as  if  there-were  no    public policy in the health field.
                  limits.  Except  in  certain  prepaid  group   health   plans,       If you have  survived  this  painful  moral  dilemma,  I(
                  there has been little  budgeting  in  ambulatory  or  hospital    us turn  to  another  perplexing  issue:  that  medical  cai
                  care.  Physicians  behave  as  if  they  were  guided   by   a    may  not  save  many  lives  but  is  mostly  concerned  wit
                  fantasy that they always  do  everything  possible  for  their    improving the condition of  life.  Instead  of  simple  lif(
                  patients. But  none  of  us  has  ever  done  everything  pos-    years  saved,  our  more  usual  achievement  may  be   som(
                  sible for his patients.  We  have  always  made  choices:  how    thing  Eke  years  of  improved  f@nction  saved.   Here   w
                  much  time  we  spend  with  a  patient,  how  long  we   keep    come  against  the  major  unresolved  issue  in  the  benef
                  him in the hospital, how many tests we order.                     field (for we are clearly dealing at this point  solely  wit
                    The following  proposal  is  an  attempt  to  make  explicit    the  benefit  half  of  the  cost-benefit  problem):  namel)
                  the grounds  on  which  medical  care  choices  are  made.  No    how   do   we   weigh   years   of   improved   function   a
                  new  value  judgment  is  suggested.  Rather  an   effort   is    compared with life-years saved.  We  intuitively  make  suc"
                  made   to   understand   what   values   guide   health   care    judgments in medical care  situations.  We  work  harder  t,
                  decisions,  and  in  understanding  them   to   make   medical    save  a  life  than  to  improve  function,  but  how   mucl
                  judgments   more   consistent   with   themselves   and   with    harder and under what circumstances?
                  these principles. Furthermore,  if  we  can  make  the  values       If we  can  explicitly  articulate  the  basis  for  the&
                  sufficiently  explicit,  a  whole  new  world   of   decision-    judgi-nents,  we  can   be   more   consistent   in   futuii
                  making  is  opened   through   powerful   econometric   models    judgments.  Take  the  example  of  the  burning  house.   I
                  which permit rational  allocations  of  resources  in  circum-    contains tluee  40  year  old  persons.  Two  single  quadra
                  stances   where   the   complexity   of   data   and    inter-    plegics (totally bed-ridden) are on the  bottom  floor,  an(
                  relationships   exceed   the   capacity   of   common    sense    a  single  working  person  on  the  top  floor.  Assume  al
                  decision-making.                                                  three  have  the  same  life   expectancy.   In   your   bes
                    Let us set a  goal  for  any  allocations  problem:  to  get    judgment, you can  either  save  the  two  quadraplegics  oi
                  the  most  for  our  money.  But  the  problem  is  to  define    the single working man-what would you choose?
                  ,'most." To most  patients  and  doctors,  the  "most"  is  to      If you save the working man,  then  you  are  assigning  2
                  save a patient's life. But one patient's life i-nay  not  have    value of something less than  one  half  the  value  to  the
                  the  same  social  significance  of  another,  an   evaluation    life-year of a bed-ridden patient as a working man.  Or,  it
                  mostly related to the  age  of  the  respective  patients.  If    you save the  two  quadraplegics,  you  are  assigning  them
                  there must be a choice,  the  40  year  old  will  usually  be    something more than  one  half  the  value  of  the  working
                  preferred to the 90 year old. This  has  led  to  the  use  of    man.
                  "life-years saved"  rather  than  lives  saved  (cf.  Michael,      This comprises the basis for  a  proposed  benefits  scale
                  Spatafore, et  al).  Public  policy  adheres  more  nearly  to    as below. The assigned values  are  arbitrary  but  not  out
                  this notion of life-years  saved  than  the  old  favorite  of    of keeping with public opinion.
                  economists, "life earnings saved."                                     1.0 good health, working
                   What  is  the  practical  significance  of  life-years  saved          .7 not working, at home
                  when resources  are  limited?  On  the  basis  only  of  life-          .4 not working, institution
                  years saved, would you give the next  spot  on  an  artificial          0  dead
                  kidney to a 90 year  old  or  a  40  year  old?  Most  of  you     If policy makers were able to  decide  on  such  a  benefit
                  would  select  the  40  year  old.  But  what  would  be   the    scale, and with whatever data is available  on  the  results
                  choice if there were only time.or  resources  to  save  either    of medical programs, rational explicit  decisions  could  be
                  one 40 year old man  from  a  burning  building  (or  a  sink-    made with such a scale as a basis.

                  126



        For  example,  relative   benefits   of   competing   programs      The   example   in   Appendix   C    indicates    that    the
                                    rms  of  life-years   saved.   But      optimal mix would be:
        could  be  calculated  in  te               re   than   saving         750  stroke  patients  rehabilitated
        saving the life of a  working'man counts mo                           6992 persons screened in EDDU
        the  life  of  a  bed-ridden  patient:  more  than  twice   as
                                                                                  0 patients cared for in ICU
        much.                                                                  430 patients entered in a cancer registry
        Relative   benefits   could   -also    be    calculated    for      Some  KMP's  may  operate  as  if  there  is  only  a   money
        improvement  of   function.   For   example,   returning   two      constraint  and  no  limit  to  the   number   of   available
        and  a  half,  not-working   institutionalized   patients   to
                                                                            doctors, nurses, or hospital  beds.  But  increasingly  plans
        work would be equivalent to  saving  the  life  of  a  working
        man.                                                                must  determine  the  most  efficient  way  to   deploy   our
         These  rough  and   ready   calculations   have   not   taken      limited resources.  Doctors  and  nurses  will  be  available
        into account the number of  years  a  patient  would  gain  in      for  a  new  RMP  program  only  if  they  are  lifted   from
                                                                            other pro,,rams.
        each   functional    category,and   a    proper    calculation      It  must  be  pointed  out  that  the  data   are   imaginary
        includ@s  these  adjustments.
         A   theoretical    example    with    hypothetical    numbers      and indeed identify a major need  for  any  allocation  tech-
        illustrates how these calculations could  assist  in  choosing      nique: better data  on  the  effectiveness  of  medical  pro-
                                                                            grams.  The  lack  of  such  data  is  no  comfort  for   the
        among  four  proposed  projects   to   be   funded   from   an      intuitive planner as compared to  the  explicit  cost-benefit
        RMP.                                                                planner. Both are at a  great  disadvantage.  But  the  tech-
          The  calculations  (Appendix  A  and  B)  indicate  how   to      nique forces any planner  to  specify  what  expectations  he
        get the most for the  money:  put  it  all  into  the  program      @  -for  his   proposed   programs   an.d   what   values    he
        where  you  get   the   most   for   the   money:   the   EDDU      attaches to the results.
        (Early  Disease  Detection  Unit)  at  a  cost  of  $2351  per      In   summary,   cost   benefit   analyses    encourage    the
        benefit unit.                                                       planner  to  specify  his  value  system  and   behave   con-
          But  this  solution  does  not   take   into   account   the      sistently Nvith it and to be  explicit  as  to  the  benefits
        number of   stroke  patients  to  be  rehabilitated  nor   the      he expects  from  given  programs.  It  then  allows  sophis-
        number of doctors, nurses,  or  hospital  beds  available.  It      ticated solutions for  problems  too  complex  to  be  solved
        is this  quandary  that  the  multiple  equation  linear  pro-      intuitively.
        gramming  model  helps  to  resolve.  Given  20,000   doctors'
        hours  (on  an   annual   basis),   100,000   nurses'   hours,                            REFERENCE
                                                                             l,fichael, Jerrold M., Spatafore, George, and Williams, Edward
        15,000 bed  days  and  100,000  office  visits,  what  is  the      R., "A Basi, information System for Health Planning,"  PUBLIC
        optimal  mix  of  programs  (not  the  single  best  program)?      HEALTH REPORTS, Vol. 33, No. 1, January, 1968.








                                                                                                                                      127











                                                                                       APPENDIX A

                                                                                Calculation of Benefits
                                      1.0                            SI  = Working                                  Fl Working Year Saved
                                      0.7                            S2  = Not working - at home                    I (Patient Otherwise Would Diel
                                      0.4                            S3  = Not working - institution                L= I Benefit Unit
                                      0                              S4  = Dead

                                                                                        Example

                     Coronary  Care   Unit

                           Assume benefits of  program  equal  any  improvement  in  functional  state  due  to  program.
                           Assume I patient in 20 survives who would otherwise die, and assume his functional status for balance of his life is as follows:

                                                                                                                        IS4  -  S3]
                                     S3 (Acute Hospital)                                     0.1 year x  0.4  x  0.05  .002
                                                                                                                        fS4  -  S2]
                                     S2 (Home - Not Working)                                 0.3 years x 0.7 x.  0.05  .0105
                                                                                                                        IS4 - SI  I
                                     SI (Working)                                            5.0 years x 1.0  x  0.05  .2S
                                                                                                                        IS4  7  S2]
                                     S2 (Retired-home)                                       5.0 years x 0.7  x  0.05  .17S
                                                                                                                       .4375    Benefit Units
                                                                                                                                Per Patient


                    By Similar Calculations:

                                     Stroke Rehabilitation                                    .9400  Benefit  Units  Per  Patient
                                     Early Disease Detection Unit                             .0265  Benefit  Units  Per  Patient
                                     Cancer Registry                                          .1 107 Benefit  Units  Per  Patient





















                  128








                                                                 APPE@X B
                                                       comparative Costs Per Benefit Unit
                                                       For Each Patient in Program
                 Assume                 MD    Hours              $  15 (not counting overherd in institution or office)
                                        RN    Hours              $    4
                                        Hospital Days            $100
                                        Office Visits            $  5 (not counting MD income)

                                        Stroke                      EDDI                     U c  onay unit
     MD hours                  20x$     15=  2            0.1 x  $15  =  1                      15 =      300                        0
     RN hours                  80x$      4=               0.2x$      4=                          4=       600                        0
     Hospital days             20 x $100  =    2000                0                    10 x $100 =      1000                    0
     Office Visits             10x$      5=      50           12x$ 5=  60                              0            20x$ 5=        100
     Total cost per patient                   $2670                    $62.30                           $1900                     $300
     Cost unit benefit         2670= $2840                    2@30 $2351                1900= $4343                 130-= $2981
                               -                             .0265                     .4375
                               .9400













































                                                                                                                                              129









                                                                                        APPENDIX C




                                                                                7he Most Efficient Program

                    3                                          Maximize      0.94X,   +   0.0265X2   +   0.437SX3    +    0.1107X4
                                                 Stroke                  EDDU                   ICU                      Ca. Reg.
                    MD hours                     2OX,                    O-lX2                  2OX3                        1OX4          20,000 (10 doctors)
                    RN hours                     8OX,                    0.2X2                  15OX3                       2OX4          100,000 (50 nurse
                                                                                                                                                              S)
                    lnst. days                   2OX,                    0  X2                  1OX3                        OX4            15,000 (45 beds)
                    Office visits                lox,                    12 X2                  OX3                         2OX4         ioo,ooo
                    Solution:                    750                     6992                   0                           430
                      Total Patients


                    MD hours                     7@O                     6992                   0                           430       Total          Left Over
                                                 x2O                     x 0.1                  -                           X10
                                                 15,000                  699                                                 4,300    20,000             0
                    RN hours                     750                     6992                   0                           430
                                                 x8O                     x 0.2                  -                           x2O
                                                 6.0,000                 1398                                                8,600    70,000       30,000
                    Inst. days                   750                     6992                   0                           430
                                                 x2O                     X0                                                      X0
                                                 15,000                  0                                                       0    15,000             0
                    Office visits                750                     6992                   0                           430
                                                 X10                     xl2                                                x2O
                                                 7,500                  83,904                                               8,600      100,000          0


                               Accountability       and        Decision-Making                  Our   system   cannot   be    compared    with    PPBS    structures    in
                              in the Iowa Regional Medical Program                              large   bureaucratic   agencies,   but   it   illustrates   how    certain
                                                                                                PPBS concepts can be applied at any functional level.
                                     CHARLES          W.          CALDWELL
                                                                                                    I   can   offer   no   at   formula   for   evaluating   a   Program's
                                                                                                                       p
                      My    charge    is    to    describe    how    Planning-Programn-dng-     overall   impact   on   a    Region,    for    establishing    priorities,
                    Budgeting     concepts     are     being     implemented     in      an     even   for   determining   broad   program   direction.    But    I    can
                    ,accounting/decision-making    system    in    the    Iowa     Regional     tell  you  of   a   system   that   does   permit   the   core   structure
                    Medical  Program.   I   will   note   some   of   the   advantages   of
                                                                                                to     provide     certain     objective      information      to      the
                    the   system   over   the   more   traditional    accounting    systems     decision-making           process.
                    and  relate   some   of   the   problems   which   we   face   in   our
                    constant   effort   to   remain   true   to   the   concepts   we   are         It   should   be    emphasized    that    the    system    does    not
                    incorporating.                                                              make    decisions.    It    merely    provides    objective    information
                      The   Iowa   Regional   Medical   Program    is    a    small    pro-     which,   in   actuality,   may    be    completely    ignored    by    the
                    gram,  funded   at   a   level   of   slightly   over   $700,000.   Our     decision-makers   in    favor    of    information    that    is    purely
                    core  structure   consists   of   ten   professional   staff   members.     subjective in nature.

                    130




       Because  of  our  organization's  size   and   due   to   the     Alternative Ways to do a Given Job
       numberless intangibles  which  confront  all  of  us  in  the        Rearran-enient  of  input  to  an   already-existin-   pro-
       health field,  we  turned  to  the  PPBS  approach  described              0                                          t)
       by  Samuel  Greenhouse  in  an  article  that   appeared   in     grain in order to improve output.
       Public  Administration  Review,  to  guide  us  in   devising     Systems Analysis
       our system.' His approach is  simple  and  clear.  He  listed
       the major  structural  members  of  PPBS  as:  (Slide  Number        Application of cost studies.
           Objectives                                                    Objectives
           Programs                                                         The  success   of   our   system   stems   largely   ftoi-n
           Program Alternatives                                          accurately defining  this  term.  Without  doubt,  it  is  the
           Outputs                                                       "apex  term"   in   the   PPBS   idea-structure.   These   are
           Progress Measurements                                         criteria for judging  the  validity  of  an  objective  within
           Input                                                         our system:
           Alternative Ways To Do A Job                                     I .It Must be directly related to  the  overall  mission  of
           Systems Analysis                                                   the IRMP.
          So  that  we  do  not  become  confused  by  semantics,  1        2. It must  contain  a  description  of  an  important  end-
       would like to offer a precise definition for  each  of  these          service.
       terms:                                                               3. It MList-at  least  to  the  fullest  extent  possible-be
                                                                              amenable   to   quantitative   measurement.    What    is
                 SLIDE I-DEFINITION OF TERMS                                  not  quantifiable  has  no  valid  usefulness  within,the
       Objectives must                                                        PPBS context.
          1. be directly related to overall mission;                        4. It  must  be  honest.   In',other   words,   the   stated
                                                                              objective  must  be  identical  to  the  true   or   real
          2. describe an important end service;                               objective.
          3. be amendable to quantitative measurement;                      5. When  appropriate,  it   must   be   broken   down   into
          4. be honest;                                                       immediate and long-range expectations.
          5.  be  broken  down   into   immediate   and   long-range
       expectations.
                                                                         Programs
                                                                            A program is a package that encompasses each            and
       Program                                                           every  one  of  an  RMP's  efforts  to  achieve  a  particular
          A  package  which  encompasses  each  and  every  one   of     objective  or  set  of  allied  objectives.  A  program  could
       an RMP's efforts to achieve a  particular  objective  or  set     consist of  a  single  comprehensive  project  or  of  several
       of objectives.                                                    projects which have allied objectives. It  is  confusing  that
                                                                         in RMP jargon, the overall effort within a  region  is  called
       Program Alternative                                               a  "program."  But  for  the   purpo@s:of   our   system   the
          Other  possible    programs  besides     those     already     term will be used as just defined,    - ,
                                                                            The whole PPBS  idea  is  to  facilitate  the  coordination
       decided upon.
                                                                         of all our efforts to meet  a  particular  objective,  so  the
                                                                         validity of each  program  may  be  judged  in  terms  of  its
       Output
                                                                         overall strategy, dimension and  costs.  This  permits  it  to
          Tangible outgrowth of a particular program.                    be  compared  with  other  programs,  potential  or  existing.
                                                                            In our system no  objectives  are  acceptable  unless  they
       Progress Measurement
                                                                         suggest a program specifically,.  designed  to  fulfill  them;
          Answers the question: How closely does the progress            and no entity can be described  as  a  program  unless  it  is
       planned for match the progress actually realized?                 designed to accomplish explicit objectives.

       Input
                                                                         Program Alternatives
          Total quantity of manpower, facilities, equipment                 Program alternatives are  programs  to  the  sarhe  general
       and materials applied to a program.                               end  other  than   those   alreadk   decided   upon.   Program
                                                                                                                                    131




                   alternatives  suggest  a  choice  between   two   or   more   pro-                 THE IOWA SYSTEM
                   grams designed to advance the same overall mission.                Six major steps  have  been  identified  by  the  IRMP  a
                   Output                                                           essential  to  significant  progress  toward   its   overa.
                                                                                    mission. In  aspect,  each  step  is  continuous  and  oper
                    An  output  is  a  product  or  a   service.   As   produced   by ended, and its  influence  changes  as  now  information  i
                   the  RMP,  it  is   a   tangible   outgrowth   of   a   particular gathered and updated. (Slide Number 2)
                   program. It must  be  a  kind  of  service  that  can  be  singled The first of these six  steps  involves  the  gathering  o
                   out  as  an  indicator  of  program  results.  It   must   be   an morbidity  and  mortality  data  and  related  infon-natiot
                   important   end-service   and   must    satisfy    an    important that pern-dt  us  to  evaluate  the  effectiveness  of  th(
                   objective.                                                       existing health care system in Iowa.
                                                                                     The  second  step  is  the  assessment  of  all   existini
                   Progress Measureiiient                                           health resources within the region  that  fall  within  the
                    If   output   means    only    those    pragmatic    end-services parameters of RMP legislation.
                   that  satisfy  explicit  RMP   objectives,   then   program   ful- The third step is the identification  of  needs.  From  the
                   flument   demands   an   output   that   was   planned   and   has information  provided  by  steps  one  and  two,  an   Iowa
                   been    produced.    Therefore,    progress    measurement    must Regional Health Profile is being developed.  It  should  be
                   satisfy  one  question:   Does   the   progress   achieved   match ei-nphasized again that this profile is open-ended and will
                   the progress anticipated?                                        continually   change    as    new    information    becomes
                                                                                    available.  On  the  basis  of  the  existing  profile,  we
                   Input                                                            endeavor to identify where existing  services  need  to  be
                   Input  is   the   total   quantity   of   manponver,   facilities, expanded, coordinated  or  reinforced  to  meet  the  needs
                   equipment   and   materials   applied   to   the   program.   Like identified. We determine where  new  services  need  to  be
                   most, we summarize this input in units of dollars.               initiated and supported.
                                                                                    The  fourth  step  is  the  establishment  of   priorities.
                   Alternative TVays To Do A Given Job                              Conventionally, the criteria considered in  the  establish-
                                                                                    ment of  priorities  should  include  evaluation  of  need,
                   This   concerns   the   rearrangement   of   input   invested   in scientific feasibility, practicality, effectiveness, timing,
                   an   already-existing   program   to   expedite   production    or amount   of   resources   available   and   community   ac-
                   upgrade  services.   In   other   words,   one   would   rearrange ceptance.
                   the  manpower,   facilities,   equipment   and   materials   going The fifth  step  is  the  planning  and  implementation  of
                   into a  program  in  order  to  improve  the  quality  of  service programs to meet these priorities.
                   or  arrive  at  the  stated  objective  in  a  shorter  period  of The sixth  step  is  the  continuous  evaluation  of  those
                   time.  Do  not  confuse  "alternative   ways   to   do   a   given programs  accompanied  by  modification  based   upon   how
                   job"   with   "program   alternatives."    Program    alternatives
                   are  output  oriented.  Utilization  of  a   program   alternative Nvell they meet their planned objectives and-insofar as  it
                                                                                    can  be  determined-the   ii-npact   that   meeting   these
                   changes the output,  because  it  is  a  substitute  for  a  whole
                   program  and  has  different  specific   objectives.   Alternative oojectives  is  having  an  achievement  of   the   overall
                   ways  to  do  a  given  job  are  input  oriented  and  deal  with n-dssion of the organization.
                   the  best  way  to   achieve   an   already   chosen   output   or Our   accountabflity/decision-niaking    system    involves
                   objective.                                                       only steps five and six. Not until we reach step  five  can
                                                                                    we measure precisely how well an  objective  is  being  met
                   Systems Analysis                                                 and  consider-if  indicated-an   alternative   program   to
                                                                                    meet those objectives.
                   Systems   analysis   within   the   IRMP   system   is   primarily Other than  intuitively,  we  have  no  way  of  evaluating
                   the  application  of  cost   studies.   These   studies   are   of the overall impact of the IRMP on the  health  systei-n  in
                   special  usefulness  in  tavo  areas  of  the  system:   (1)   the Iowa. A principal reason for  this  is  that  presently  we
                   determination  and  evaluation  of  alternatives   and   (2)   the have no  way  of  obtaining  accurate  morbidity  data.  We
                   measurement   of   costs   versus   'progress   within   a   given hope to solve this problem soon.
                   program.                                                         Good  data  on  morbidity  will   certainly   aid   us   in
                   These   might    be    called    "pure"    definitions.    As    Iselecting priorities. Such data will not,  however,  better
                   proceed, you   will   see   how    we    bend    and    abuse    these enable us to evaluate  the  overall  impact  of  the  IRMP-
                   definitions within our system.                                   because we will still have no basis for relating changes in
                   132








                                                                  SLIDE 2



                                                                 STEP     1

                                                              dity  and   Mort
                                                              tistics  and  Ot
                                                           statistics  on  the
                                                           Order to Determine the Effective-
                                                           ness of lowals Existing Health Ca
                                                           System.


                                                                                                     STEP 2
                            STEP 6                                                               The Assessment of
                        Program Evaluation                                                         all Health Resources







                                                                                                       STEP 3
                           STEP 5                                                                  Identification of
                 The Design and implementa-                                                           Needs
                    tion of a Program to Meet
                    Those Needs




                                                        Establishment   of   P

                                                                  STEP    4
      morbidity  to   the   existence   of   our   organization.   Too    hierarchy.  Actually,  the  system  can  be   applied   at   any
                   ptions  would  have  to  be  made,   due   to   un-    level-so  long  as  we  remember  that,  viewed  fron,  the  top
      many assum                                                          of the hierarchy, all these levels  are  means  to  anetid.  and
      controllable variables.                          can   be   used    no ends in themselves.
        Now let us examine how the Iowa system                               In Iowa, all staff members contribute  to  our  syst@th-
      in  the  development  and  selection  of  programs  to  meet                                                      king       program
                                                                          particularly in gathering information, ma
      priority  objectives  by  looking  at  the  decision-rnaking                                    ing  cost  studies.   They   also
      .process  from  another  perspective.   (Slide   Number   3)        evaluations and undertak
      Visualize  a  hierarchy  of  objectives   that   relate   to        dissen-dnate .the  resulting  information  to  the  decision-
      different levels of this process.                                   makers.
             1. At the top is the organization's overall mission.            The  nuts  and   bolts   of   the   system   can   be   best
             2. At  the  second   level   we   put   program   objectives recognized  by  breaking  it  down  into  four  broad  areas  of
             designed to meet priority needs.                             activity:
             3. At the third  level  are  groups  of  project  objectives    1. Establishing the costs of program alternatives.
             Ithat make-up a program package.                                2. Establishing  the  costs  of  Alternative  ways  to  do  a
             4. At  the  fourth  level  we   find   the   objectives   of       given job.
             specific activities within a project.                           3. Accounting  and  costing  of  existing  programs   on   a
             5. The  final  level   is   occupied   by   the   day-to-day        monthly    basis.
                                                                             4.  Accounting  and  evaluation  of  core  activities,oii   a
             objectives  that  are  to  be  met   within   a   project
             activity.                                                           monthly    basis.
             Our,  system  is  applicable  at  the  second  level  and       First,  cost  estimates  are  made  on  all  program  alter-
       downward, since it  supplies  data  that  grow  in  objectivity    natives. Most  of  our  program  alternatives  come  to  us  in
       and  preciseness  as  we  travel  toward  the  bottom  of   the     the  form  of  new  project  proposals.  The,  cost   estimates
                                                                                                                                      1:33










                                                                                                                w
                                                                                        o-C    c)-   rri                                         CL 4A            n
                                                                                                                                                 CD C
    4  co                                                                                                o                                       3 :to@  c-
                                    O'
       0                                 -  ',  0                                                 3                                                 3    M   G)   M
               <                    '0-0     =,                                         0 -..R  CL       M                                       0
       m       M                                                                                                ID                               5@3 -a  o   ;u
       rn
                                    0-        -C                                                  mm
                                                        O@      'D
                                                                                                  CL@ 0
                                                        cn                                        T:o    0                                       =3,D    m
                                                                                                                                                 0       cn
                                    =,a.      0         c:                                        0 1 0  m                                       :1 B 2;
                                            OM          co                                        , m r  L-                                      0
                                    Iq     M"Y.                                         3         O'     rn
                                    o                                                                    0
                                                        0
                                                                                                                                                 n o
                                                                                                                                                 m wLI1
                                                                                                  o-     :i
                                    011'        -                                                        <                                       CD a,
                                    , - 0- @. ---       ror,                                             m                                       cu
                                    =a)  Flo                                                             cn                                      0
                                    0                                                                                                            cx
                                                                                                                                                 rz
                                    c
                                            o-
                                              :,o,
                                            4x                                          c = 5-0
                                            mm                                                                                                   ID
                                              0                                                                                                  O'
                                                                                        Lc       MO                                              < 7;. cw
                                                                                                  R s-@                                          @. C-1 =
                                            0)  Er                                                ID au 0)                                       goZ
                                                                                                  -3                                             cr CL 0
                                                                                                  0 U,
                                                                                                  c-
               r;
                                                                                                  05                                             m
                                                                                                  0                                              0CX
                                                                                                  m 0                                            w 0
                                                                 rr
                                                                                                                                                 Z-.o
                                                                                                                                                 a:E                      0.  o      (A
                                                                                                  3                                              m                        Di, o      o
                                                                                                  CL                                             0                        M            0

                                                                                                                                                 0                        SD  0      M
                                                                 co                               m                                              0
                                                                                                  0                                                                           I
                                                                                                                                                                          0
                                                                                                                                                                          0)  M      r 0

                                                                                                                                                 CL                                                  cn

                                                                                                                                                 MO
                                                                                                                                                                          q   o      =r              t4

                                                                                                                                                                          m
                                                                                                                                                 0
                                                                                                                                                                                     M5
                                                                                                                                                                                       g-.  o<
                                                                                                                                                                                       17   m
                                                                                                                                                                                     0
                                                                                                                                                                                            >
                                                                                                                                                                                            rr--
                                                                                                                                                                                       !SF

                                                                                                                                                                          0)  -o       o
                                                                                                                                                                          U)  (D       -    0
                                                                                                                                                                          CD  0
                                                                                                                                                                              Ci)      CD
                                                                                                                                                                                     =r 0)
                                                                                                                                                                              m
                                                                                                                                                                                     0
                                                                                                                                                                              =1       =r

                                                                                                                                                                              =r       0)
                                                                                                                                                                              0)     m
                                                                                                                                                                                       m


                                                                                                                                                                              0
                                                                                                                                                                                     o
                                                                                                                                                                              =r
                                                                                                                                                                              M
                                                                                                                                                                              0)     OQ (D








                                                                 cr

                                                                                                                  2
               1                                                 2
               >                                                 0
               a
               2'
               -Z
                                                                 i                                                                                                                             I     .


       I       1;                                                                                                                                         .  ..
               4
               0                                                                                                                                                                       -    .
                                                                                                                                                                   C)
    I  I





    e  broken  down  to  determine  what  it  will   cost   to      Our  monthly  accounting  of  programs  by  objectives  is
    :hieve each precise objective of  a  program  alternative.   based primarily on time  studies  that  are  completed  daily
     Each  program  alternative  must   compete   with   other   by project staff. In each project,  these  time  studies  are
    -ogram alternatives and with  all  existing  programs  for   broken  down  by  the  project's  tangible   output.   (Slide
    Le limited  resources  available.  In  summation,  we  are   Number  4)  Each  output   can   be   easily   measured   and
    king two actions with program  alternatives:  (1)  we  are   relates  to  a  precisely  stated  sub-objective.  The   per-
    .termining  cost  factors  and  (2)  we  are  providing  a   centages of time are converted  to  dollars.  Since  salaries
    amework for comparisons by the decision-makers,              usually  make  up  more  than  75  percent  of  a   project's
     Where  arriving  at  the  costs   of   alternatives   are   budget,   the   unassignable   remainder   of   the    budget
    )ncerned, we have  a  lot  to  learn-not  only  about  the   expenditures  are  arbitrarily  broken  down   according   to
    chniques  involved  but  in  making  the  figures   under-   personnel expenditure percentages. Large equi          pment,
    andable.  We  need  to  improve  in  the  presentation  of   consultation  or  travel  expenditures  that  can  be  easily
    .formation to our decision-makers so  that  they  will  be   assigned to a given output are assigned separately.
    )le to use it more readily  to  make  informed  decisions.
     Alternative ways to do a given job  are  usually  in  the      The  monthly   report   is   similar'to  this  one.  (Slide
    )rm of new, single project proposals that  fit  within  an   Number  5)  Monthly  expenditures  for   programs,   projects
    ready-existing   program   package.    They    may    also   and project activities are  reported  by  traditional  budget
    -nanate from an existing project as a request to alter  or   categories. As you can see,  projects  are  grouped  together
    place a certain project activity.  The  latter  source  is   in program packages  when  that  is  appropriate.  This  way,
    ;ually staff-generated. Except that they fall at  a  lower   we  are  able  to   know   more   precisely   what   we   are
    vel of the  hierarchy  of  objectives,  they  are  treated   achieving   within   a   program   area   and    what    that
    luch the same as program alternatives.                       achievement is costing us.
                                                                                                         Name:
                   SLIDE 4-COMPREHENSIVE STROKE MANAGEMENT PROJECT                                       Position:
                                            Time Study by Tangible Output                                Month:

    Oate      Nursing      Nursing      Home        Stroke       Public     Physician      Total       Notes
             Workshops   Education      Service       Unit     Education    Education























                                                                                                                           135






                                      SLIDE 5-MONTHLY AccoUNTING BY PROGRAM,PROJECT AND OBJECTIVE



                                           Personnel     Consultants       Equipment    Supplies   Travel  Publications      Other    Total  Direct  Cost

                                         *
                  Program   A            *
                    Project  AI          *
                       Objective   Ala   *
                       Objective   Alb   *
                                         *
                    Project  A2          *
                       Objective   A2a   *
                       ,Objective  A2b   *
                       Objective   A2c   *
                       Objeptive   A2d   *
                                         *
                    Project  A3          *
                       Objective   A3a   *
                                         *
                  Program   B            *
                    Project  BI          *
                       Objective   Bla   *
                                         *
                  Program   C            *
                    Project  Cl          *
                       Objective   Cla   *
                       Objective   Clb   *
                       Objective   Cld   *
                                         *
                    Project  C2          *
                       Objective   C2a   *
                                         *


                    TOTAL





                       At the same time costs are  recorded,  accurate  records            One important  evaluation  factor  that  isn't  portraye(
                  of  tangible  output  are  maintained,  which   makes   cost          here is the quality of the output. Output  is  evaluated  fo
                  analysis an easy task at any time it  is  needed.  Here  are          quality  in  much  the  same  manner  that  all  RMPs   earn
                  examples  of  how  these  outputs   are   reported.   (Slide          out  evaluation,  which  includes   pre-testing   and   post
                  Number  6)  (Slide   Number   7)   There   is   no   uniform          testing, attitudinal  questionnaries  and  other  techniques
                  method  of  reporting  and  these  outputs  are  reduced  to          Like all RMPS, we, are  constantly  endeavoring  to  improvl
                  different types of  units  for  costing.  We  probably  need          our evaluation methodology.
                  more tiniforn-dty,  but  due  to  the  constant  changes  in             Of course, it is easy to       see that  this  system  isn'
                  many  of  our  programs,  any   standard   form   would   be          comprehensive.   Many   intangible   benefits    are    unac
                  obsolete  before  it  was  off  the  press.  Each  of  these          counted for. In the  presentation  of  our  objective  infor
                  reports usually involves several telephone calls to  clarify          niation we attempt to. qualify  the  information,  carefull)
                  information.                                                          spelling  out  those  probable  benefits   which   are   nol
                       This is an example of the type of  cost-analysis  report         reflected  by  tangible  output.  We  cannot   ignore   thal
                  that can be    made at  any  interval  and  presented  to  the        benefits,  whether  tangible  or  intangible,  form  an  im-
                  decision-makers.  (Slide  Number   8)   (Slide   Number   9)          portant part of the analysis.
                  This particular example  includes  costs  other  than  those             In the third  broad  area  of  activity,  we  are  trull
                  being met  by  the  IRMP  and  therefore  required  informa-          bending-if  not  breaking-the  conceptual  rules   of   PPBS
                  tion not available on a month-to-month basis.                         because we are accounting for core activities that              it

                  136




      most cases cannot be related to an end-service. Here  is                                           SLIDE 6-NURSING WORKSHOPS
      how it works: (Slide Number 10)                                                                     (July, 1, 1 969 - January 31, 19 70)
          Daily    time    studies    are    made    based    on    a    breakdown    of                                    Capital      Division
      core    activities    into    the    functional     activities     shown.     Each
      of     these     activities     produces     a     measureable     output     that               Type of                   Number        of
      does    not    relate    to    any    precise    objective    in    many    cases.               ivorkshop                        workshops         Attendance
      The    time    contributed     to     operational     projects     as     depicted
      in   the   first   nine   columns   can   be    related    to    project    output           I.............                       15                       295
                                                                                                   if ............                      15                       301
      and figured into the costs of operating the projects.                                        III ............                     11                       214
          The column entitled "other"  is for those core                          staff            IV ............                         9                     161
      functions    that    can    be     related     to     an     end-service     other           TOTAL      ......                    50                       971
      than    that    of     an     operational     project.     For     example,     we                                North Central Division
      have    a    central    medical    library     network     that     receives     a           .............                        16                       371
      limited    amount    of    attention     from     core     staff     members.                ............                         14                       395
          The      last      four      columns,'entitled       "Project       Planning,"           III . . . . .......  .               10                       199
      "Data       Collection,"       "Public       Information'.       and        "Staff           IV ............                         9                     151
      Education,"    are    strictly    functions    and    do     not     relate     to           TOTAL      ......                    49                       1,116
      an     end-service.     However,     we      have      arbitrarily      identified                                  NorthwestDivision
      tangible       out'put   as   a   gauge   to   evaluate    our    core    activity..
          For     example,     we     can     compare     the     amount     of     time           I.............                       33                       810
      invested    in     new     project     development,     -,vhich     would     fall           11 ............                      26                       617
      under     "Project     Planning,"      with'      the      number      of      new           III ............                     35                       725
      proposals      submitted      to      our      decision-makers.       We       can           IV ............                      35                       643
                                                                                                   TOTAL      ......                     129                     T7 9-5
      compare   the    amount    of    time    we    are    spending    on    a    given
      operational     project     with     that     project's     output.                          Each workshop is three hours in duration.

                                                   SLIDE 7 -NURSING EDUCATION CONFERENCES
                                                                (July 1, 1969 - January 31, 1970)

                     Location                                                     Number of Days                                                A    ttendance
                     Des Moines                        ..................                  2                                                        110
                     Mason City                         .................                                                                            25
                            TOTAL                                                                                                                   135

                                                                      HOME SERVICE CONSULTATION
                                                                        (July   1,   1969    -January    31,    19    70)
                     July -August, 1969                           .................        139 visits                                           49    patients
                     September                              .......................        42  visits                                           23    patients
                     October                            ...........................        28  visist                                           25    patients
                     November                            ..........................        23  visits                                           19    patients
                     December                            ..........................        41  visits                                           35    patients
                     January, 1970                          .......................        58  visits                                           37    patients
                             TOTAL                          .......................        331 visits                                           188   patients

                     Average patient load:        72
                                                                                                   STROKE UNIT
                                                                                  July 1, 1969 -January 31, 19 70)

                     Total number of patients admitted:                                    81
                     Average patient stay in stroke unit:                                  12.5  days

                                                                                                                                                                 137





                                                                             SLIDE 7-(Continued)


                                                                              PUBLIC EDUCATION


                                                                                     Conferences                                Attendance


                                      Northwest Division             .............            7                                        122
                                      Capital Division             ...............         I  0                                        347
                                      North Central Division         .............           26                                        576
                                             TOTAL                                           T4                                      1,045

                                      Conferences averaged one hour each.



                                                                 SLIDE 8-CoST ANALYSIS FINDINGS


                                                                                 Nursit?g IVorkshops
                            1. The cost to the RMP for each nurse who was a student in the workshop was $1.88 per hour (Student Hours).
                            2. The combined cost to the RMP and the Heart Association (I 6 cents per student hour) was $2.04.
                            3. The total cost to the RMP, the Heart Association and Heart volunteers (an added 39 cents per hour) was $2.43.
                            4. The cost of instruction to the RMP for the workshops was $40.31 per hour (Instructor Hours).
                            S. The cost to the RMP and the Heart Association per instructor-hour was $43.60.
                            6. The cost to the RMP, the Heart Association and Heart volunteers per instructor-hour was $52.07.

                                                                          Nursing Education Conferences
                            I .The cost to the RMP  per  student-hour  was  $3.77
                            2. The cost to the RMP and the Heart Association per student-hour was $4.08
                            3. The cost to the RMP, the Heart Association and Heart volunteers was $6.15
                            4. The cost to the RMP per instructor-hour was $307.70
                            S. The cost to the RMP and the Heart Association per instructor-hour was $ 3 33.30
                            6. The cost to the RMP, the Heart Association and Heart volunteers was $502.66


                                                                             Home Service Consultation
                            1 .RM-P cost per visit made to a patient  was  $20.86
                            2. RMP and Heart Association cost per  visit  was  $22.56
                            3. RMII, Heart Association and Volunteer cost per visit was $24.1 5
                            4. RMP cost per patient in  the  program  was  $83.44
                            S. RM? and Heart Association cost  per  patient  was  $90.34
                            6. RM?, Heart Association and Volunteer cost per patient was $96.60

                                                                                    SLIDE     9


                                                                                    Stroke Unit

                            1. RMP cost per patient admitted to the stroke unit was $197.24
                            2. RMP and Heart Association cost per patient was $213.07
                            3. RMY, Heart Association and Volunteer cost per patient was $306.04
                            4. RNIP cost per patient day in the stroke unit  was  $15.77
                            5. RMI? and Heart Association cost per patient day was $16.24
                            6. RNIP, Heart Association and Volunteer cost per patient day %vas $24.48


                                                                                 Public Education

                            1. RMP cost for each individual attending conferences was $3.71 per hour (Student hour).
                            2. RMP and Heart Association cost per student-hour was $4.06
                            3. RMP, Heart Association and Volunteer cost per student-hour was $4.90
                            4. RMP cost per instructor was $91.64
                            5. RMP and Heart Association cost per instructor-hout was $98.79
                            6. RMP, Heart Association and Volunteer cost per instructor-hour was $119.1 1.


                            138




     There has been no attempt to make a judgment as to whether or not these costs are reasonable. 'Me value of this type  of  analysis  is
     -atly enhanced when the unit costs can be compared to the sanw unit costs in similar  projects.  This  information  should  be  shared
                                                                                           ilar projects in the hope that they, in turn, will
     th other Regional Medical Programs and Heart Associations that are conducting sim
     are similar information with us.
                                                                          the sub-regions of the Stroke Management Proje t          Nursing
     The one classification that can be compared internally between                                                          c  is
     orkshops, -since all three sub-regions of the project have identical programs. The results of these internal comparisons, based on RMP
     ,sts only, are depicted below.
                                            Northwest Division             Capital Division               North Central Division
     ,-r student-Hour                          $  1.18                        $ 2.70                             $ 2.24
     ,r Instructor-Hour                        $25.49                         $52.40                             $50.97


























                                                                                                                                        139


























                           ts

                  z




                          13
                           1-3





                           bo







                        - - - - - - - - - - - - - - - - - - - -
                           00















                           tn



                             - - - - - - - - - - - - -




                             ---- -----------------
                            en



                             - - - - - - - - - - - - - - -






                         tn











             140




       This  information  places  us   in   a   better   position   to     Our  cost  studies  on   projects   would   be   more   valuable
       letermine  how  and  in  what  areas  we  should  be   spending     if  we  had  cost  studies  from  other  regions  with  which  to
       )ur time. Except as it relates to  operational  projects,  this     compare  them.  Because  not  everyone   is   willing   to   play
       reformation  is  not  reported  regularly  to   our   decision-     under  our  rules,  we  sometimes  feel  -like  the  only  honest
       liakers. It is  presented  at  regular  staff  meetings,  where     guy in a crooked crap game.     .
       ve  jointly  evaluate  how  usefully  our  time  is  spent  and     We  need  a  national  review   and   evaluation   system   that
       @stablish work priorities.                                          is  more  consistent  in   both   scheduling   and   methodology.
       Following  a  true   PPBS   structure,   the   entire   expense     For example, we  have  had  four  fiscal  years  assigned  to  us
       )f the core activity  would  be  assigned  to  project  output.     in three years' time.
       [n truth, within the. core  structure  we  are  not  evaluating
                                                                            We  need   to   be   permitted   to set   our   own   priorities.
       )ur  success  on  the  basis  of  end-product.  We  are  evalu-     Presently,  while  we  are   setting our   own    priorities    we
       ating  means,  not  ends.  However,   we   are   supported   by     must  try   to   second-guess   what is   currently   popular    in
       )nc  school   of   thought   which   believes   that   indirect     Washington.
       activities  should  be  allocated  to  a  program   only   when      Finally, in  my  opinion  PPBS  is  not  a  set  of  techniques
       such an allocation  would  contribute  to  a  better  decision.     so much as it is  a  set  of  attitudes.  Unless  one  is  really
       In summation, the system permits us to,                             interested in getting  the  most  for  the.tax  dollar,  it  will
       1.  Undertake    better    cost-accounting    for     individual    not work. Old  concepts  such  as  the  "budget  is  a  political
           projects.                                                       tool,"  "the  harboring  of  privileged  information,"   or   the
       2.  Obtain   more   efficient   use    of    scarce    manpower,    "measure  of  an  organization's  success  by  the  size  of  its
           including staff time.                                           budget"  are  concepts   which   are   not   corn   atible   with
       3.  Provide   more    accurate    cost    estimates    to    our                                                    p
           decision-makers.                                                PPBS concepts.
       What  I  have  described  here  is  only   a   start   on   the       The   purpose   of   PPBS   is   to    bring    together    the
       construction   of   a   system   designed   to   support    our     budgeting   process    with    the    decision-making    process,
                                                                           evaluating  both  processes  on  the  basis  of   tangible   out-
       decision-making   process   with   objective   information.   I     put.  Its  intent  is  to  make  and  keep  us  mission  oriented
       believe  the  system  has   influenced   decisions   as   those     since we will be ultimately judged      on  how   well   we   ac-
       decisions  are  concerned  with  alternative  ways  to,  do   a     complish our mission.
       Oven job. In all honesty,  I  can  see  very  little  influence
       on decisions that  relate  to  program  alternatives,  possibly                             FOOTNOTES
       because  we  haven't  considered  that  many   program   alter-       1.  Samuel  N.   Greenhouse,   "Ile   Planning-Programming-Bud-
       natives  since  the  IRMP  became  operational.  I   think   it     geting  System:  Rationale,  Language,  and  Idea-Relationships,"
       may  have  more  influence  at   the   end   of   the   current     Public Administration  Review,  XXVI,  No.  4  (December,  1966),
       three-year  funding  period  when  political   influences   and     p. 273.
       obligations will be greatly lessened.
         The  system  is   faced   with   many   problems.   We   need         Resource   Allocation   and   th.e.Evaluation Process.
       in-depth cost-benefit studies which  will  carry  all  the  way                       CHARLES L. JOINER
       down  to  the  consumer  and  will  take   into   account   the
                                                     alth   care.   We     ECONOMICS,         SOCIAL          PRODUCTION          FUNCTIONS,
       many econorrdc variables that affect he
       need  to  develop  better  ways  to  present  our   information                AND RESOURCE ALLOCATION
       to the volunteer decision-makers.                                   Economics
         Presently,  we   have   neither   the   resources   nor   the       Economics  is  the science of allocating          scarce    re-
       expertise   to   deal   with   social   costs.    Comprehensive     sources  among  alternative  uses  so  as  to       attain    the
       costing  should  also  include  estimates  of  cost  that   are     greatest  or   maximum   fulfillment   of   society's   unlimited
       related to changes in  other  human  systems  as  a  result  of     wants, i.e., "doing the best with what we have."
       decisions we make.
                                                                           Optimum Allocation ofresourees
         We  must   continue   to   search   for   better   and   more
       comprehensive  ways  to  quantify  services.  It   is   to   be        Classical  econorrgcs  assumes. the       "rati,onal.     man"
       remembered,  however,  that  we   are   primarily   a   service     . concept.  Therefore,  if  the  decision  maker  then  wishes'to
       organization  and  therefore  must  be  conscious  that   there     combine  resources  to  n-dnimize  the,co@t@   of   producing   a
       is a point of diminishing returns.                                  given  level  of  output;  if  heknowsthe.   resources   (inputs)
                                                                                                                                         141





                 that can be used in producing the  output,  and  if  he  also             this paper now  turns  to  the  question  of  social  produc.
                 knows  the  prices  for  increasing  each  input   (and   the             tion  functions  in  relation  to  the   political   decision
                 increase  in  output  that  will  result  from   each   input             process and such problem-solving approaches as PPBS.
                 entry),  then  the  way  to  achieve  minimum  costs  is   as               Social  Production  Functions  and  the   Decision   Process
                 follows:  the  decision  maker  should  use  those  resources                Before one is  completely  enthralled  with  the  idea  of
                 in such  a  combination  that  the  additional  increment  in             the determination of  social  production  functions  and  the
                 output per dollar spent on each input is equal.                           role of benefit-cost analysis in  the  allocation  of  scarce
                    The allocation  of  resources  under  the  assumptions  of             resources,  some  reflections  on  the  realistic   political
                 classical  economics  is  assumed  to  be  optimized  because             decision  process  are  necessary.  Charles   Lindblom'   has
                 of the competitive nature of the system  itself.  Unlike  the             quite  adequately  described  the  real  political   decision
                 classical  model,  many  social  action  programs,  including             process  which  in  some  ways  appears  to   be   distinctly
                 health, involve the allocation of  relatively  scarce  public             different  from  the  problem  solving   approach   of   PPB.
                 resources.  In  addition,  there  is  the  need  of  properly             Lindblom states as a first rule of the  successful  political
                 meshing  these  public  funds  with  private-  resources  for             process, "don't force a  specification  of  goals  or  ends."
                 maximum   effectiveness   for   improving   or    maintaining             The reasoning here is that not only is the  specification  of
                 health. Needless  to  say,  any  model  constructed  for  the             objectives intellectually difficult, but  also  pragmatically
                 allocation of resources  for  better  health  will  have  its             harmful.  In  fact,  it  could  mean  that  agreement   among
                 shortcomings, e.g., the allocation of  resources  for  health             diverse interests on  specific  measures  may  be  completely
                 means fewer  resources  available  for  non-health  purposes.             blocked.
                    If one considers the health sector as a system of  itself,                For example, the Elementary and Secondary                    uca-
                 optimum resource  allocation  requires  that  the  additional             tion Act of  1965,  which  is  considered  a  landmark  piece
                 benefit  rising  from  the  allocation   of   an   additional             of legislation in terms of federal            aid   to   education,
                 expenditure (cost)  for  a  particular  health  problem  must             needed the support  of  at  least  three  divergent  interest
                 be equal to ratios of benefits  to  costs  for  other  health             aroups. The pa     rochial schools saw it as  a  step  in  pro-
                 problems.   For   a   theoretical   explanation,   additional             viding financial assistance for  parochial  school  children.
                 benefits  and  costs  may  be   referred   to   as   marginal             A second group saw  it  as  an  anti-poverty  measure,  since
                 benefits and costs.  Therefore,  the  optimum  allocation  of             the distribution of funds for Title I of the bill  was  based
                 resources toward the  solution  of  various  health  problems             on the number of  poor  children  in  each  school  district.
                 is accomplished when:                                                     A third group  saw  it  as  a  broad  beginning  of  a  large
                                                                                           program of federal aid to  public  education.  It  does  seem
                                    .MB                                                      uite possible that the bill would  have  been  defeated  had
                                       a @ MBb @ MB... - MBn                               q
                                    mca  MCb  MC,           Mcn                            any  attempt  been  made  to  secure  strong   agreement   on
                                                                                           long-run objectives.
                 where:  MB  equals  marginal  benefits  accruing  from  the   iinple-        A  second  major  feature  of   a   desirable   decision   process
                 mentation of a particular  technique  or  approach  for  solving  the
                 health problem within a series of health problems, a, b, c,    n.         as seen   by   Lindblom   is    its    incremental    characteristic.
                                                                                           The   process   toward   objective    attainment    should    proceed
                 MC equals  the  marginal  costs  resulting  from  the  implementation
                 of a particular technique or approach for solving  the  health  prob-     in  very  small  steps  because  of  our  inability  to  foresee  the
                 lem within a series of health problems, a, b, c,   n.                     full   social   consequencies   of   any   program   and   the   fact
                                                                                           that   political   decision   costs   tend   to   increase   as   the
                     This    marginal    benefit-cost     approach     for     optimum     decisions conflict with values held by interest groups.
                 allocation  of  resources  for  the  solution   of   various   health        The  third   major   element   in   the   Lindblom   approach   is
                 problems   may   also   be   applied    to    the    allocation    of     referred   to   as    the    "advocacy'.    process    of    reaching
                 resources   among   alternative   strategies   or   approaches    for     decisions.  To  the  extent   that   advocates   of   every   related
                 the  solution  of  any  given   health   problem.   In   fact,   this     interest  have  a  voice  in  policy  making,   the   self   interest
                 benefit-cbst  approach  should   be   an   inherent   part   of   any     motivation   will   insure   that    each    advocate    takes    the
                 normative    decision    making    process.    However,    the    ap-     responsibility   for   researching   the    consequences    of    any
                 pfication    of    such    a     theoretical     approach     becomes     action  for  the   value   he   represents.   Obviously,   this   ap-
                 extremely   difficult   when   the   decision    maker    does    not     proach  is  not  idealistic.  Instead,  it  is  pragmatic,   stresses
                 know   or   can   not   determine   precisely   the    benefits    or
                 outputs   of   a   particular   technique   or   approach   to    the       Charles  Lindblom,  "The  Science  of  Muddling  Through,"   Public
                 solution  of  a  health  problem.  It  is  for   this   reason   that     Administration Review, Vol. 19,  No.  2  (spring  1959),  pp.  79-88.

                 142




      process  rather   than   substantive   criteria.   Therefore,   by     instances,  evaluation  must  involve   in-depth   studies   using
      definition,   a   "good"   decision   is   one    which    obtains     sophisticated     statistical     techniques-particularly     when
      consensus  rather   than   one   which   meets   the   requirement     the 'impact  of  one  program  is   only   a   part   of   a   much
      of efficiency or effectiveness.                                        larger  program.  Feedback  of   results   from   operating   pro-
        In   order   to   properly   relate    political    values    to     grams  is  an  absolute  essential  to   program   planning,   and
      analytical  program  decisions   involving   the   allocation   of     systematic  analysis   provides   the   necessary   feedback   for
      resources,  the  decision  process  must   include   some   deter-     decision making and planning.
      rnination   of    the    social    production    functions    that
      translate   program   specifications    (input)    into    program     INTRODUCTION OF CONCEPTUAL AND ACTUAL
      consequences   (output).   An   analogy   may   be   drawn    here              PROJECT EVALUATION PROCESS
      to   consumer   preference   theory.    Econon-dc    factors    of                     IN RELATION TO PPBS
      production-land,    labor,     capital,     and     management-are        It is commonplace   to   wade    through    an    article    on
      not  directly  evaluated   in   terms   of   consumer   preference     evaluation and find  it  is  like  the  last  ten  you  read.  The
      functions,  but  only   through   a   process   which   translates     mass  of  articles   on   evaluation   emphasize   the   necessity
      these  inputs  into  outputs.  It  is   the   output,   or   final     for  evaluation  and  they  generally  state  that  a   conceptual
      product,  that  enters   directly   into   consumer   preferences.     evaluation   model   should   be   designed.   These    evaluation
      The  process  of  translating  inputs  into  outputs,  of  course,     articles  stop  at  this  point.  I  plan  to  go   beyond   where
      assumes   knowledge    of    the    production    functions    in-     others  stop  and   speak   to   you   on   a   conceptual   model
      volved.                                                                designed and tested at ARMP.
        If  the  analogy  is   applicable,   we   need   to   know   the        In  June,   ARMP   instituted   a   systematic   and   in-depth
      social  production  functions  of  health  programs.  It   is   at     evaluation of  all  approved  projects.  This  was  a  first  step
      this  point  that  the  task  of  the  social  scientist   becomes     in   total   program   evaluation   and   an    experiment    with
      more  difficult  because  many   of   the   social   action   pro-     PPBS.
      grams  of  the  federal  government   do   not   deal   with   the        The  majority  of   core   staff   at   ARMP   were   skeptical
      simple  translation   of   factors   of   production   into   com-     about PPBS. It  was  decided  that  the  first  two  aims  of  the
      modities,   but   the   production   functions   are    determined     PPB  system  should   be   used   to   evaluate   ARMP   projects.
                                                                       2
      largely   by   institutional   or   behavioral    characteristics.     These aims are:
      Determination   of   social    production    functions    involves        l.'the   careful   evaluation   and   examination   of    goals
      complicated   systems   in    which    institutional,    technical          and  objectives  in  each  major  area   of   activity,   and
      and  economic   factors   interact   with   each   other.   There-          then to
      fore,  we  cannot  expect  the  technical  expert  to  define  all        2. analyze  the  output  of  a  given   program   (project)   in
      of  the  input-output  relationships,  i.e.,  relying  totally  on          terms of its objectives.
      physicians  to  evaluate  all  health  programs  or  engineers  to        An  evaluation  model  was  designed   and   used   with   four
      implement the design of pollution control systems.                     projects  this  past  summer.  The   model   was   found   to   be
        It  seems   imperative   that   the   analysis   of   production     adequate   for   use   on   these   projects.    The    experience
      functions    in    most    public    programs    must    take    a     acquired  by   developing   an   evaluation   model   along   with
      systematic  approach  rather  than   being   confined   to   tech-     the  actual  evaluation  process  has  led  to   a   more   knowl-
      nical  considerations.  Many  times  it  is  extremely   difficult     edgeable    understanding    of    problems    associated     with
      to  predict  with  any  real  degree  of  certainty  the  speci  c     analytic  investigation  as  well  as  giving  an  indication   of
      performance   of   new   or   proposed   social   progrwns.   Some     problems linked with PPBS.
      of  this   uncertainty   concerning   the   relationship   between
      inputs  and  outputs   can   be   reduced   via   either   ex-post
      evaluation   of   operating   programs   or   the   implementation     Divisions of the Model and Experiences Gained
      and   evaluation   of   demonstration   projects.   Although   the     Project Development
      process   of   decision   making   described   by   Lindblom    of        Assumptions:   1.   When   projects    are    developed,    the
      incremental  changes  has  been   recognized   as   an   effective     alternatives, if known, are brought out and discussed.
      means   of   proceeding   under   uncertainty,   this   does   not
      reduce   the   need   for    systematic    analysis.    In    some        2.   The   project   goals   and   objectives   meet    program
                                                                             goals and objectives.
        2 Cbarles L. Schultze, The  Politics  and  Economics  of  Public        Step  1.  Determination  of  the   project   goals-This   first
      Spending,  (Washington,  D.C.:  'Me  Brookings  Institution),  pp.     step  consists  of  determining  in   rather   broad   and   long-
      55-76.                                                                 range  terms  what  is  to  be  achieved   by   the   project.   A

                                                                                                                                            143





                 statement   of   project   goals   is.   necessarily   broad   and     Actual Evaluation Begins
                 frequently  long-range,  and,  for  these  reasons,  a   project's
                                                                                            Step   S.   Collection   of    performance    data-once    the
                 goals  may  not  be  capable  of   direct   measurement   in   the     desired level of  action  is  decided,  the  relevant  data  which
                 sliort-run.   One   problem   encountered   in   the    evaluation
                 process   was   that   seIveral  of  the  projects  did   not   have   will  permit  the  determination  of  the  actual  level  of  per.
                                                                                        formance   must   be   collected.   Collection    of    evaluation
                 realistic goals.                                                       data  should  be  an  integral  part  of  the   on-going   project
                     Step   2.   Determination    and    statement    of    project     implementation.  If  steps  I  through   4   are   complied   with
                 objectives-Project  objectives,  as  used  in   this   evaluation,     as  described  above,  then  actual  evaluation  can   easily   be
                 are   narrow   and   short-range   statements    of    what    the     accoi-nplished.  It  is  a  matter  of  inserting  data  into  the
                 project  is  to  accomplish.  Project   objectives   are   derived     proper  place.  Output  studies  are  important   and   the   type
                 from  and   must   be   compatible   and   consistent   with   the     study  (e.g.,  cost-benefit   analysis)   should   be   determined
                 project  goals.  The  difficulty   encountered   here   was   that     when   the   project   begins   so   that   adequate   data    are
                 often  the  project  objectives   were   vague   (e.g.,   increase     available.
                 patient  care)   and   had   to   be   rewritten   in   measurable         Step   6.    Comparison    of    actual    performance    with
                 terms.                                                                 standards  previously   set-This   is   considered   the   program
                     Comment:   These   problem   areas   have   been    corrected.     (project)  effectiveness  step.  Programs  may  differ  in   their
                 Realistic goals and measurable  objectives  are'  a  part  of  all     effectiveness   depending   on   the   extent   to   which    pre-
                 new   projects.   The   evaluation   process    actually    begins     established objectives are  attained  as  a  result  of  activity.
                 during  this  stage  of  project  development.   All   goals   and     Based   upon   a   comparison   of   actual    performance    with
                 objectives  are   being   challenged   by   the   evaluation   co-     the   standards,   the   performance   will   be   concluded    to
                 ordinator  to  make  sure  they  are   feasible   and   applicable     have  been  satisfactory  or  unsatisfactory.   After   a   deter-
                 to total program goals and objectives.                                 mination   of   satisfactory   or    unsatisfactory    performance
                                                                                        has  been  made,  the   project   administrator   has   a   number
                 Pre-Evaluation Process                                                 of  alternatives  available  to  him.  If   the   performance   is
                                                                                        concluded   to   be   satisfactory,   the   pr   'ect    may    be
                                                                                                                                        Oi
                     Step   3.   Determination    of    measures    of    objective     continued  unaltered,  or,  if  the  goals  and  objectives   have
                 attainment-these   measures   would    include,    for    example,     been  met,  the  project  can  be  satisfactorily  concluded.   If
                 such things  as:  days,  hours,  dollars,  ratings,  ratios,  per-     the  performance  is  determined   to   be   unsatisfactory,   the
                 centages,    attitude    changes,    and     patient     behavior.     administrator   may   modify   his   project   objectives   and/or
                 Repeatedly,   it   was   found   that   project    directors    of     standards   (objectives    or    standards    are    unrealistic),
                 funded  projects  did   not   know   what   data   to   keep   and     attempt  to   improve   efficiency   (inefficient   use   of   re-
                 how to record  collected  data  so  as  to  justify  the  project.     sources), or recommend discontinuance.
                 There  were  several  reasons   for   this,   one   being   poorly         Comment:  It  is  felt  that  a  seventh  step   is   required
                 written project objectives.                                            between   step   6   and   the   final    recomniendation.    This
                     Step   4.   Establishment   of    standards-sta-@idards,    as     would  be  a  step   for   feedback   between   the   evaluator(s)
                 used in this  evaluation,  refer  to  desired  levels  of  attain-     and   members   of   the   program    (project).    Honest    com-
                 inent. Only  through  the  use  of  implicit  or  explicit  state-     munications   should   take   place   between   the   evaluator(s)
                 inents   of   acceptable   and/or   unacceptable   standards   can     and  the  project  staff  so  that   apparent   results   can   be
                 the  administrator  decide  whether  to   continue,   adjust,   or     discussed.  If   discrepancies   are   discovered   during   these
                 discontinue   a   particular   project.    Standards    frequently     discussions,  further  study  can  be   made.   The   evaluator(s)
                 were not written into  the  projects.  This  has  led  to  a  poor     and   project   members   should    agree    on    the    results,
                 percentage   of   approved    projects    for    ARMP    at    the     whether satisfactory or unsatisfactory.
                 national  level  (27%).  The  lack  of  standards  has  also  made
                 projects difficult to evaluate.                                        Sunitizary a)zd Coyiclusions
                     Comment:  The  problems  in  steps   3   and   4   are   being         Economics  is  the   science   of   a   ocating   scarce   re-
                 corrected  by  a  pre-evaluation  process.  Before   any   project     sou rces  among   alternative   uses   so   as   to   attain   the
                 is written,  measures  for  objectives  are  agreed  upon  by  all     greatest   or   maximum   fulfillment   of   society's   unlimited
                 people   concerned.    During    the    pre-evaluation    process,     wants, i.e., "doing the best with what we have."
                 standards  are  established.  Alternatives  to  the  project   are         If one considers the health sector  as  a  system  of  itself,
                 further discussed.                                                     optimum  resource  allocation   requires   that   the   additional


                 144




     benefit  rising  from  the  allocation  of  an  additional  ex-       Step 4. Establishment of standards
     penditure (cost)  for  a  particular  health  problem  must  be
     equal to ratios of benefits to costs  for  other  health  prob-     Beginning of Actual Evaluation
     lems.                                                                 Step 5. Collection of performance data.
         This   marginal   benefit-cost   approach    for    optimum       Step   6.   Comparison   of    actual    performance    with
     allocation of resources for  the  solution  of  various  health     standards previously set.
     problems  may  also   be   applied   to   the   allocation   of
     resources  among  alternative  strategies  or  approaches   for
     the  solution  of  any  given  health  problem.  However,   the       After a small-scale testing of     the  first  two  aims  of
     application   of   such   a   theoretical   approach    becomes     PPBS, ARMP reported the following benefits:
     extremely  difficult  when  the   decision   maker   does   not       I . Improved project development.
     know or can  not  determine  precisely  the  benefits  or  out-       2.  Increased control of funded projects.
     puts  of  a   particular   technique   or   approach   to   the       3.  A  better  appreciation   and   understanding   of   the
     solution of a health problem.                                             value of evaluation.
         In  order  to   properly   relate   political   values   to       4.  An acceptance  by  the  staff  that  the  total  program
     analytical  program  decisions  involving  the  allocation   of           should   be   evaluated,   probably   using   the    PPBS
     resources,  the  decision  process  must  include  some  deter-           method
     rnination   of   the   social   production    functions    that       5.  Development   of   a   more   sophisticated    decision-
     translate   program   specifications   (input)   into   program           making mechanism.
     consequences (output).                                                In   November,   ARMP   will    continue    to    experiment
         Determination  of  social  production  functions   involves     with PPBS and will  further  evaluate  its  effectiveness.  At
     complicated   systems   in   which   institutional,   technical     the  present  time,  however,  ARMP  is   working   on   other
     and economic factors interact with each other.                      priorities-some  of  which  were  determined  by   the   eval-
         The second part  of  this  paper  speaks  to  a  conceptual     uation process described in this paper.
     model   designed   and   tested   at   the   Alabama   Regional     I EDITORS   NOTE.-   Tv,,o   Appendices   to    Dr.    Joiner@
     Medical  Program.  The  model  was   foLmd   to   be   adequate     paper are not repiin ted in the Proceedings. ney are:
     after  it  was  used  to  evaluate  four  projects  during  the
     summer of 1970.                                                       1.    Medical    Information    System     via     Telephone
                                                                               (M.L S. T) Evaluation Report.
         Divisions of the model are:                                       2.  Reality   Orientation   Technique   Evaluation   Report.
     Project Development
         Step 1. Deterryu'nation of the project goals.                     Both   were    prepared    for    the    Alabanm    Regional
         Step  2.   Deterrfflnation   and   statement   of   project     Medical  Program  by  Edlwrd   M.   Smith,   Ph.D.,   Research
     objectives.                                                         Associate,  Bureau  of  Research   and   Conununity   Service,
                                                                         School  of  Health  Services  Administration,  University   of
     Pre-Evaluation Process                                              Alabama   in   Binningham   and   Douglas   Patterson,    MHA,
         Step  3.  Determinatio  of  measures   of   objective   at-     Evaluation    Coordinator,    Alabama     Regional     Medical
                                n
     ta'mment.                                                           Progrant













                                                                                                                                    145







                                               SPECIAL -INTEREST MEETINGS
                             STATISTICAL NIODELS AND OPERATIONS RESEARCH


                                                               Participants

       Francis C. Ichniowski-Moderator                                     Vernon E. Weckwerth, Ph.D.
       Acting Chief, Systems Management                                    Director, Systems Development Project
       Regional Medical Programs Service                                   University of Minnesota
                                                                           Member, Regional Advisory Group
       David H. Gustafson, Ph.D.                                           Northlands Regional Medical Program
       Assistant Professor
       Industrial Engineering Division
       University of Wisconsin


                 A "Weighted Aggregate" Approach                           strategies.  A   parallel   strategy   involves   simultaneously
                      To R&D Project Selection                             taking  two   or   more   approaches   to   solving   the   same
           DAVID H. GUSTAFSON, GOPINATH K. PAI,                            problem.  In  a  sequential  strategy  the  best   approach   is
                                                                           pursued;  other  possibilities  being  considered  only  if  the
                         GARYC.KRAMER                                      first  approach  proves  unsuccesful.  The  authors   have   in-
                                                                           corporated  the  incremental  cost  of,  adopting   a   parallel
                              Introduction                                 strategy, the  probability  of  success  of  each  strategy  and
         There  appear  to  be  few  formal  decision   theory   proce-    the  cost  of  failure  in  a   normative   mathematical   model
                                                                           which selects which stratep-v to use.
       dures   for   optimally   allocating   funds   among   potential
       projects.  One  reason  for  this  is  the  lack  of   effective      This  paper  will  (1)  describe  a  general  project  evalua-
       methods  for  assigning  a  value  to  each  alternate  project.    tion   model,   (2)   discuss   problems   with   current    ap-
       With a fe@ notable exception       @, 3 @  previous      project    proaches   to    implementing    the    model,    (3)    propose
       evaluation  systems  have  been   either   theoretical   efforts    methodologies  to  solve  these  problems,  (4)  report  on  the
       requiring  many  modifications   before   being   practical   or    evaluation   of   some   of   these   methodologies,   and   (5)
       methodologies   lacking   the   scientific   rigor   to   assure    suggest areas for further research.
       reliability or validity.
         Two  excellent  articles5,  6  have  reviewed   the   research    7he General Model
       up to 1967  so  their  efforts  Will  not  be  duplicated  here.
                                3                                            Complex   evaluation   problems   generally    possess    five
       Since  then,  J.R.  Miller  has   suggested   some   interesting
       ..but  relatively  untested  procedures  for  evaluating  alter-    characteristics. First, there are  several  criteria  which  are
       native projecti usin      additive  model  where  the   criteria    important  in   evaluating   the   merits   of   the   projects.
       are weighted according to importance.                               Second,  the  relative  importances  of  these   criteria   vary
                                                                           from  one  judge  to  another.  Third,  the  extents  to   which
         L.P.   Hellman'   has   evaluated   a   value   measure    for    these  criteria  are   satisfied   are   not   always   directly
       selecting   proposals   for   research   grant   support.    The    measurable  on  an  interval  scale.  Fifth,  the  criteria  are
       model  he  used  is   based   on   the   Churchman-A@koff8   ap-    ometimes interdependent.
       proximate  measure   of   value,   modified   to   satisfy   the    s
       needs of the  National  Institutes  of  Health.  The  evaluation       Recognizing  that  the  overall  evaluation   is   some   ag-
       of each proposal  was  based  on  the  relative  values  of  the    gregate of the  valuations  of  individual  criteria,  we  write
       objectives  of  the  funding  agencies,  the  relevance  of  the             n              n+m
       proposat's  objectives  and  the  probability  of   success   of       E        Wig (@.) + 2       WjRj
       the  proposal's  objectives.   Proposals   with   high   overall                             j=n+l
       expected  values  were  selected   for   fundiig;   this   model
       appeared  to   be   superior   to   the   previous   method   of
       proposal selection.                                                 The i subscripts  are  associated  with  quantitative  variables
          Abernathy   and   Rosenbloom9   have   discussed   the   pros    and  the  i  subscripts  are  with  qualitative  variables.   Wi
       and  cons  of  parallel  and  sequential   pr   'ect   selection    represents the relative weight of the ith criterion and
                                                     Oi

                                                                                                                                        147




                    (X              he   utility   function   associated   with   the  Second,  the   regression   approach   will   not   improve   corn-
                    itd)crrietperrieosenntsxit represents the extent to which the ith  n-dttee  decisions,  only  predict  them,   because   this   method
                    variable is  present  and  Ri  represents  the  extent  to  which  is  based  on  decisions  that  were   made   by   the   committee,
                    the  ith  criterion  is  satisfied.  All  criteria  as  well   as  rather  than  decisions  that  should  have  been  made.   Man   is
                    projects are assumed to be independent.                            progressively    less    accurate     in     evaluating     complex
                    In   order   to   implement   such   a   model   we   must    (1)  problems   as   the   number   of   criteria    influencing    I-ds
                    select   project   evaluation   criteria,   (2)   assure    inde-  decision increases' 1 " 2. Hence, the regression app         roach,
                    pendence,  (3)  establish  the   relative   importance   of   the  as  a  normative  model,   breaks   down   when   the   number   of
                    criteria,  (4)  develop  scales  with  which   to   quantify   or  criteria  are   large.   The   decision   makers   become   "cogni-
                    categorize  the  variable,   (5)   determine   for   quantitative  tively  overloaded"   and   the   decisions   made   may   not   be
                    variables   the   utility   function   associated    with    each  the ones they would like to make.
                    criterion, and (6)  aggregate  the  evaluations  of  all  judges.       We  evaluated  a  third  set  of  criteria  weighting  methods
                    Such a  model  has  two  uses.  First,  it  can  be  used  as  an  where   weights   are   estimated   by   the   committee   members.
                    aid  in  the  proposal  evaluation   process.   Technicians   can  'Mere is evidence' 1   l 2, 1 3to  indicate  that   under   certain
                    use  the  model  to  estimate  the   relative   value   of   each  conditions,   men    do    this    quite    effectively.    Miller3
                    proposal  and  report  the  results  to  the  committee  as   ad-  suggests a hierarchical approach to criteria weighting.
                    ditional  information  for   their   decision   making   process.
                    Second, it could  be  used  as  a  guide  to  proposal  modifica-                             Example
                    tion.  The  model  could  predict   what   decisions   would   be
                    made   by   the    committee.    The    proposer    could    then         Assume  that  a  list  of  criteria  have   been   developed
                    improve  the   proposal   where   necessary.   By   knowing   Wi,       in a  hierarchical  form  (Figure  1).   AR   criteria   in'one
                    0   (Xi   R,,   and    the    cost    of    increasingXiorRibyone
                    unit, )@e could  select  the  criteria  to   give   the   'greatest     column that  are  connected  by  lines  are  related  in  that
                    increase in value for the least cost.                                   they  are  components   of   one   larger   criterion   in'the
                                                                                            left, adjacent  column.  We  will  refer  to  each  column  as
                                                                                            a  "level".  Decision  makers  are  asked  to:  (1)  rank,  in
                    Oiteria Weighting                                                       order of importance, the related criterion in a       I   given
                     A   criterion's   relative   importance   (weight)   should   be       level, (2) assign  a  value  of  100  to  the  most  important
                    directly   proportional   to   its   impact   on   the   decision       and  values  between  0  and  100  to  the  others  so  as  to
                    making   process.   Because   weights    define    organizational       reflect  relative  criteria  importance.  These  weights   are
                    needs,  a  set  of  concisely  defined  and   properly   weighted       normalized    and    then    successively    multiplied     by
                    criteria  can  guide  proposers  to  develop  programs  to   meet       weights  of  related  criteria  at  each  higher   level.   In
                    those  needs.  Those  who  lack   this   guidance   may   propose       Figure   2,   vertical   lines    represent    criteria    and
                    programs  of  little  interest,  become  discouraged   with   the       horizontal  lines  connect  related  criteria.   Suppose   the
                    process, and be lost as a resource to the organization.                 first  level  criteria  were  -tanked  II,  III,  and  I   and
                      From  the  proposal  evaluator's  point   of   view,   criteria       weights  of   100,   60,   and   40   were   assigned..Weights
                    weights  permit  him  to   more   accurately   and   consistently       assigned   within    criteria    sets    (BI,92B3B4),    (CD),
                    model   the   committee's    project    evaluation    philosophy.       (D,,'D2)  are  shown  in  Figure  2a.   Next,   weights   were
                    Proposals  are  frequently  too  detailed  or  numerous   to   be       normalized  by  dividing  each  weight  by  the  sum  of   all
                    evaluated   by   the   whole   decision   making   committee   so       weights  within  a  set.  The  final  weight  of  each  lowest
                    they  are  normally  reviewed  by  a  subset   of   members   and       level   criterion   is   the   product   of   the   normalized
                    staff.  Unless  each  evaluator   knows   the   relative   impor-       weights  of  itself  and  the  connected  criterion  at   each
                    tance  of   each   criterion,   their   evaluations   will   lack       of the higher levels.  Thus,  the  final  weight  of  criteria
                                                                                            DI' is the  product  of  weights  assigned  to  criteria  DI",
                    consistency.                                                                $
                      Some   project   selection   techniques   assume    that    all       Do , and C:', in Figure 2b.
                    criteria   have   equal   weight   in   the    decision    making       While  this  approach   reduces   the   number   of   criteria
                    process.  The  success  of  this  approach  is  directly  propor-    being considered  at  once,  it  replaces  one  bias  (assessment
                    tional  to  the  degree  to  which  this  assumption   is   true.    error  due  to   cognitive   limitations)   with   another   (ag,
                    Other   models   estimate   weights   by   using   an   empirical    gregation  error  due  to  multiplication  of  errors   occurring
                    technique  such  as  multiple  regression.'  0.   The   committee    at each  level  of  the  hierarchy).  As  the  number  of  levels
                    rates  hypothetical  projects   described   in   terms   of   !h@    increases,  the  second  type   of   error   becomes   important.
                    criteria.  Coefficients  are  estimated,  using  the  method   or
                    least  squares,  so  as  to  best  predict  committee  decisions.       We  compared  this   approach   with   a   modification   (the
                    There  are  two  problems  with  this  approach.  First,  it   is    "ratio  method")  that  appears  to   reduce   both   aggregation
                    difficult  to  obtain  enough  data  (and  therefore  degrees  of    and assessment errors:
                    freedom)  to  yield  valid,   reliable   coefficient   estimates.       1. Rank the criteria in order of importance.

                    148




                            FIGURE i.-Project Evaluation Criteria Dispia)?ed in a Hierarchical Fashion.


                                                                                  Incidence
                                                                                  Mortality
                                                                                  Duration
                                                   Impact on Health,4 - Distress
                                                                                --Quality of
                                                                                     Whole Life

                                                                                .Different Allocation
                                                                                     of   Resources
                   Benefit                         Potential Benefit
                                                     Income From                  Using Another
                                                                                     Projects Resources


                                                                                  Improves
                                                                                     Subregional Groups

                                                   Ancillary                      Strengthens
                                                                                     Other Programs
                                                                                  Other
                                                                                --Within                         Now
                                                                                     the Region                  After Demonstration
                    Regional                       Cooperation                    outside    Region                 is Complete
                       Considerations            -Geographic Needs

                                                   meets Standards of
                                                     Medical Profession
     PROJECT                                                                      Use of Personnel
      VALUE                                                                       Data
                                                                                  Data   Collection
                                                                                     Method
                     Methodology s                 Project Design               -Steps in Process
                                                                                -Time      Required
                                                   Chance of                         Per Step
                                                     Accomplishing                Method of
                                                     Objectives                      Starting Project
                                                   Time Before Benefits
                                                      Reach Population
                                                   Measure of
                                                      Effectiveness             -Types
                                                                                -Uses of


                                                                                -Technical
                                                   Proposer                     -Administrative
                     Capability                    Personnel                    -Needs > Capability
                                                                                -Capability > Needs
                                                   Total Cost
                                                   -
                     Cost
                                                   - Percent to
                                                      Be Assumed










                                                                                                                                     149






                                                   FIGURE 2.-Demonstration of HierarchicalMethodfor Diteria Weighting

                                                                           2a-Criteria pyramid including criteria weights.


                                                      A!40                    B;: 1 00

                                                                                                                                            5 0
                                                           B      00  B@50          B,-25        B,    =25                  D@E100                  D, 1 0 0


                                                                          2b-Criteria weights normalized within subsets.


                                                      A'1.2                    B'::.5

                                                                                                                C,
                                                                                                                                         lu
                                                                                                                     .667                        ".333
                                                              B, 1.5       B, 1.25            125  B4='125                   D, E75              D'    1.5
                                                                                                                                                     2

                                                                            2c-Criteria weights for lowest level criteria.


                                               Criteria                                            Products                                              Weight

                                                A                                                  (.2)                                                  .2000
                                                B                                                  (.5) (-5)                                             .2500
                                                B'2   . . . . . . .       . . . . . . . . . . . . .(.5)   (.25)                                          .1250
                                                B                                                  (.5) (.125)                                           .0625
                                                                                                   (.5) (.125)                                           .0625
                                                B 4   . . . . . . . . . . . . . . . . . . . .
                                                C'    .........................                    (.3) (.667)                                           .2000
                                                D  ..........................                      (.5) (.333) (.3)                                      .0500
                                                D'2   .........................                    (.5) (.333) (.3)                                      .0500
                                                                                                                                                 Total     1.0000



                       2.  Compare the most important criteria with  every                                    Two factors may  have  caused  the  superior  per-
                           other related criteria. Estimate how many times                               formance     of     the     ratio     method.     First,     the     hierarchical
                           more    important    the     top     ranked     criterion     is     than     method    may    yield    higher    errors     because     the     errors     are
                           each of the other criterion.                                                  multiplied    rather     than     added.     Second,     the     ratio     method
                       3 . Repeat   steps   I   and    2    for    a    new    set    of    criteria     uses     an     odds     estimation     methodology     while     the     hierar-
                           composed     of      the      most      important      criteria      from     chical   method   uses   ratings   on   a   0   to   I   00    scale.    Previous
                           each set.                                                                     research'    '    indicates    that    odds    estimation    leads    to     more
                       4.  Multiply   the   weights    assigned    to    criteria    in    step    2     accurate      estimates      of      subjective      probabilities.      Possibly
                           by   those    assigned,    in    step    3,    to    the    top    ranked     the    results    extend    to    criteria    weighting.
                           criteria from its set.
                                                      Example                                            Criteria Independeiice
                           Suppose    for    the    criteria    in    2a    ratio    weights     are
                       assigned   to   each   set   as    shown    in    Figure    3a.    The    new          Two    criteria    are    dependent    when    (1)     the     extent     to
                       criteria   set   (A,   B2   C,   D2)   are   ranked    (3,1,2,4)    and    as     which    one    criterion    is    satisfied     is     influenced     by     the
                       signed    weights     of     (1:     1.5,1:1,1:1.25,1:3).     The     weights     extent   to   which   another   criterion    is    satisfied    and    (2)    the
                       in    Figure    3c    are    obtained    by    multiplying    the     weights     utility   associated   with   a   given   level   of    satisfaction    on    one
                       in  sets  A,  B,   C,   and   D   by   values   of   I/   1.5,   1,   1/1.25,     criterion    is    influenced    by    the    degree     to     which     another
                       and    1/3    respectively.    The    final    normalized     weights     are     criterion    is    satisfied.     When     the     assumption     of     criteria
                       obtained    obtained    by     dividing     each     weight     in     Figure     independence,    postulated     in     equation     1,     does     not     hold,
                       3c   by.   the   sum   of    all    the.weights    in    Figure    3c.    The     total   project   value   is   no   longer   equal   to   the    sum    of    the
                       normalized weights are given in Figure 3d.                                        values associated with the individual criteria.


                   150






                                          FIGURE 3. Ratio Method of C)Itetia Weightitig.

                                            3a. Ratio values assigned to criteria  in  Figure  2a.

                        A     1:1             B,     =   1:1            C =  I                   DI    =    1:1
                                              B-2    =   1:2                                     D2    =  1:1
                                              B3     =   1:4
                                              B4     =   1:4

                                               3b.   Ratio weights assigned to new  criteria  set.

                        A =  1: 1.5           B,     =   1:1            C =  1: 1.25             DI    =    1:3

                                                      3c. Ratio weights of all criteria.
                        A =  1: 1.5           B,     =   1:1            C =  1: 1.25             Di        1:3
                                              B2     =  1:2                                      D2        1:3
                                              B3     =  1:4
                                              B4     =  1:4

                                                          3d. Normalized ratio weights.
                        A     0. 16           B,     =  0.24            C =  .20                 DI        0.08
                                              B2     =  0.12                                     D2        0.08
                                              B3     =  0.06
                                              B4     =  0.06



         An   additive   model   with    interaction    terms    may       Criteria interdependence ha   s  been  treated  in  several
       compensate for  criteria  dependence  if  enough  degrees  of    ways    in    project    evaluation    models.    Some     ap-
                                                                                  3 7,1 5 , 1 6
       freedom  can  be  obtained   to   accurately   estimate   co-    proaches              assume  that  all   criteria   are   in-
       efficients empirically.  However,  if  coefficients  must  be    dependent. This  biases  the  evaluations  in  direct  propor-
       subjectively  estimated,  the  multi-dimensionality  of   the    tion to the magnitude of the interdependencies.    1 7   Other
       interaction  term  would  increase  both   the   number   and    evaluation          models4 eliminate     criteria     causing
       difficulty of  the  estimates.  While  we  have  very  little    dependencies.   Fishbum"   has   suggested   a   method    for
       information  about   the   performance   characteristics   of    identifying  such  dependencies  but  there   has   apparently
       judges  in  weighting  multidimensional  criteria,   we   may    been  no  experimental  validation  of  the   technique.   His
       draw   some   insights   from   research   into    subjective    method,  which  uses  the  concept  of  indifference   between
       probability estimation.' ' Several researchers' '   I 2  have    pairs  of  gambles,  is  suitable  when  each   criteria   has
       shown  that  men  are  conservative  probability*  estimators    discrete levels and when the  number  of  criteria  is  small.
       and that this  conservatism  increased  with  the  number  of    -Unfortunately,  bias  reduction  may  be  more  than   offset
       data  to  be   simultaneously   considered..Future   research    by   the   information   loss   resulting    when    dependent
       should  determine  (1)  if  the  same   problem   exists   in    criteria are discarded.  This  loss  can  be  reduced  by  (1)
       utility assessment and criteria weighting and  (2)  the  best    discarding criteria only  when  there  is  a  high  degree  of
       methods  for  obtaining  these  estimates.  Until  then,  the    interdependency  and  (2)  discarding  those  criteria  having
       criteria independence problem will  have  to  be  treated  in    the smallest influence on project evaluation. ,
       some  other   way.                                                  We   propose   the   following   untested   procedure   for
                                                                        discarding criteria:
          *Conservative  eIstimators overestimate  the  importance  of      1. Select and estimate the relative importance  of  a  set
       diagnostic data  and  underestimate  the  importance  of  non           of criteria using the procedures suggested  earlier;  a
       diagnostic data.                                                        subset of those criteria will be independent.
                                                                                                                                   151





                    2. Select   pairs   of   criteria    having    a    major    de-     C, has the largest number of dependencies              (four)    so
                       pendencies.   This   can    be    accomplished    empirically     it  is  the first  to  be  considered  (step  5).  WI  -<  W2  +
                               ta                              f-                                                    s
                       if  da      are   available.   If   no    ,    experts    can     W,7  +   Wg + WI 0 so C, i  discarded (step 6).
                       subjectively select those pairs.    1 9                               C2   is the  next  criteria  to  be  considered  (step   7).
                    3. Remove   from   consideration   those   criteria   that    do     W2 > W4 + WI o so C4 and C, 0 are discarded.
                       not  have   at   least   one   major   pairwise   dependency.         Since   C2  no   longer   has   pairwise   dependencies   it
                       These criteria can be considered independent.                     forms     a new   subset    leaving    only    the    dependency
                    4. Divide   the   remaining   criteria   into   subsets   having     between     C,7 and Cg to be  rectified.  W,7  >  Wg  so  Cg  is
                       high   intradependence    but    low    interdependence    by     discarded.  The  new  group   of   criteria   subsets   is   C2,
                       having  experts  sort  3x5   cards,   each   containing   the     [C3,C6  1,  CS,C7,C8-  W3  <  W6  so   C3   is   discarded.   The
                       name  of  one  criterion,  into  groups   such   that             final  set  of  independent  criteria   is   C2,Cs,C6,C,7,   and
                       a.  the  extent  to  which   one   criterion   is   satisfied     Cs
                           strongly  implies  or  is  implied  by  the   extent   to     Criteria Measurement
                           which   another    criterion    in    that    group    is
                           satisfied.*                                                       Measures of the degree to which criteria          have     been
                       b.  the  utility  function   of   each   criterion   in   the     satisfied must be reliable, valid, and easy           to    obtain.
                           subset   is   influenced   by   the   degree   to   which     Some evaluation models'        5,16   use  ordinal  values  as   Xi
                           another criterion in the subset is satisfied.'                entries   in   some   variation   of   equation   1.   These    are
                    S. Select  the  criterion,  C,  with  the  largest   number   of     obtained   by   ranking   projects   according   to    extent    to
                       major    pairwise    dependencies.     We     will     either     w@ich  they  satisfy   each   criterion.   Unfortunately,   ordinal
                       discard   this   criterion   or   all   the   criteria   with     scale  values  should  not   be   added21  because  the   resulting
                       which it has major dependencies.                                  project  scores  will  be  biased  in  proportion  to  the   degree
                    6. If its  weight,  as  determined  in  step  1,  is  less  than     to   which   the    intervals    between    project    ranks    are
                       the  sum  of  the  weights   of   all   dependent   criteria,     unequal.                       IS3    Se
                       discard criteria  C.  If  not,  discard  all  those  criteria         Other evaluation mode             lect  only   criteria   whose
                       having major dependencies with it.                                values   can   be   added.   The    important    but    qualitative
                    7. Repeat  steps  5  and  6  for  the   criterion   having   the     criteria  are  replaced  by  less  appropriate  but   more   easily
                       next largest number of dependencies.                              measurable   criteria.   In    such    an    exchange,    important
                    8. Repeat    step    7    until     all     dependencies     are     information may be lost.
                       eliminated.                                                           As-  an  alternative,  we  suggest  that  criteria  should   be
                                                                                         measured   on   an   interval   scale   whenever    possible    and
                                          Example                                        otherwise,   ordinal   scale   values   should    be    transformed
                                                                                         onto   an   interval   scale   using   the   method   proposed   by
                      Suppose we have a set of      IO criteria, Cl,. -    CIO)                       12  'A  set   of   statements   (verbal   descriptors)
                                                                                         Eckenrode.
                  with weights WI,. . ., WI 0 assigned in step 1.        Step 2          are  assigned  values  on  an   interval   scale   which   indicate
                  yielded subsets [C,,C2,C4,C,7,Cg,Cl 0 1 , [C3,C6 ], [Cs j ,            the  degree  to  which  a  project   possessing   that   descriptor
                  and [Cs]. Step 2 yielded major pairwise dependencies                   satisfies  the  criterion.  Sensitivity   can   be   increased   by
                  for the first subset as shown below:                                   increasing  the  number  of  descriptive   phrases   as   long   as
                                            c                                            this  number  does  not   exceed   the   evaluator's   ability   to
                                      ci     2     C4     C7     C8       C9     CIO     discrin-dnate.    Previous     research 23, indicates   that    men
                  C,                        x      x       x               x    x        may   have   difficulty    discriminating    beyond    approximate.
                  C2                    x           x                           x        ly seven criteria.
                  C4                    x    x                                               In   order   to   test   the   effectiveness   of   these   two
                  C7                    x                                   x            methods,   nine   of    thirteen    members    of    a    committee
                  C8                                                                     evaluating   medical   research   proposals   used   the    hierar.
                  C9                   x                   x                             chical  and  ratio  methods  to  estimate  weights   for   the   40
                  CIO                  x     x                                           evaluation  criteria  in  Figure  1.   They   also   rank   ordered
                                                                                         each  of  the  40  criteria.  This  rank  ordering   was   a   good
                                                                                         approximation   of    their    true    feelings    because    their
                    *This   method   for   detecting   criteria   dependencies   was     cognitive   limitations   were   not   exceeded.   They    compared
                evaluated by GusWson. He attempted to predict patient lengt              two  criteria  at  a  time  until  the   ordering   was   complete.
                of stay by a Bayesian model  that  assumed  data  were  conditional-     These   rankings   were    compared,    via    Spearman    Coffela-
                ly independent. in one case, he  acted  as  if  all  data  were  in-     tion  Coefficient,   with   those   derived   by   the   subjective
                dependent.  In  the  other,   he   used   procedure   4a   to   form     weighting methods.
                conditionally  independent  subsets  of  data.  The  second   method
                predicted length of stay better than the first. This would  indicate         The  results  indicate  (Figure  4)  that  the           ratio"
                that  the  proposed  approach  may  be  effective  for   identifying     method  does   predict   rankings   more   effectively   than   the
                major dependencies.                                                      "hierarcmcal"   method.   The    average    Spearman    coefficient
                152





    was  0.676  for  the  "ratio"  method  versus  0.309  for   the    from them to an  interval  scale.  For  13  of  these  criteria
    "hierarchical"  method.  The   standard   deviations   of   the    the scale went  from  0  to  100:  for  11  of  them,  it  went
    coefficients  indicates  that  the  ratio   method   has   less    from -100 to +100.
    variation  between  subjects  (0.021)  than  does  the  hierar-
    chical method  (.295).  This  implies  that  the  ratio  method       The results (Figure 5) indicate  that:  (1)  The  0  to  100
    may  more  consistently  model  the   decision   maker's   true    scale has less  overall  variability  than  the  -100  to  +100
    feelings about criteria weights.                                   scale.  (2)  On  the  0   to   100   scale,   the   end   point
       Inter-rator  variability  was  examined  for   twenty   four    descriptors  have  less  variability  than   the   intermediate
    qualitative criteria in Figure  I  using  a  diverse  group  of    descriptors.  (3)  It  would  appear   that   in   each   case,
    twelve  health  related  professionals   including   engineers,    subjects  perceive  the   descriptors   to   be   approximately
    economists,  physiciars,  planners,   and   hospital   adiwnis-    equally  spaced  in  importance.  This  finding   is   somewhat
    trators.  Verbal  descriptors  were  established  for  the   24    discouraging  because  it  indicates  that  subjects  may   not
    qualitative   criteria.   Each   committee   member   estimated    accurately  perceive  differences  between  these   descriptive
    the  importance  of  these   descriptors   by   drawing   lines    phrases.  Group  discussion   between   decision   makers   may
    FIGURE   4.--Evaluation,   via   Spearman   Correlation   Coeffcient   of   the   Degree   to   Which   Criteria   Rankings   Were
                                       Approximated by Methods for Criteria PVei,@hting.

           l'O







           .80-

                                                                                              00 00 00                        00
                                                                                           100


           .60 -
                                                                                                                     r=.676
                                                                                                                                 Ratio
    0                                                                                                                           Method



           .40 -
    CD
    !7
    0


    E                                                                                                                      Multi-Level
    a)     .20-                                                                                                                 Method
    a




               1             2              3             4              5             6             7              8                9

                                                                       Subjects



         -.20 -





         -.40 J




                                                                                                                                     153





                be one way  to  improve  their  perception  of  the  values.       the  criteria  being  -measured  have  clearly  defmed-  end
                (4) The variation  between  subjects  appears  to  be  quite       points, there  are  several  utility  estimation  techniques
                large. This wide variability between  subjects  may  be  at-       that may be used.' 0,20'When the  range  of  values  is  not
                tributed to individual  differences  in  utility  functions.       clearly specified, the end  point  can  be  approximated  by
                This may be especially pronounced  in  a  group  as  diverse       asking experts to estimate the value of the,  criterion  for
                in background  as  the  one  tested.  Much  more  investiga-       which they  would  be  very  surprised  to  find  a  project
                tion  is  needed  into   performance   of   subjects   using       exceed.
                descriptive phrases. However, these  initial  data  indicate
                that  subjects  can  give  more  than  a  simple  preference       Model Modification
                ordering to the phrases.
                                                                                      If men are conservative estimators of  criteria  weights,
                  FiGuRE 5.-LRelation between      Value  of  the  Descriptors     they will not  attribute  enough  importance  to  diagnostic
                              and Variation between Subj:ects.-                    criteria* and will attribute too  much  importance  to  non-
                   100 -                                                           diagnostic critera. By raising the weight of  each  criteria
                                                                                   to a constant power greater than one       'we are in  effect,
                                                                                   increasing the value of diagnostic criteria  and  decreasing
                                                                                   the  value  of  non-diagnostic  criteria      '  Equation     2
                                                                                   represents such a modification  of  the  weighted  aggregate
                                                                                   model:


                    50

                                                                                               n              n+in
                                                                                          E    'E wag(x,) +       @aRj                          (2)
                                                                                                                n



                a.
                                                                   Standard        If the value  of "a"  that  maxin-dzes  model  effectiveness
                0                                                  Deviation
                                   x                                               were  constant  between  decision  makers,   it   would   be
                                 10            0       @O         4'0        so    practical to  estimate its  value  and  thereby  improve  the
                                                                                   model's  evaluation  capability.  The  value  of  "a"   that
                bD
                                                                                   opti@zed  the  performance  of  equation  2  was  calculated
                                                                                   in order to investigate this question.
                                                                                      Ten  members  of  the   proposal   evaluation   committee
                                                                                   rated on a 0 to  100  scale  twelve  hypothetical  projects,
                  -50 -                                                            each described by five of the criteria in  Figure  1.  These
                                                                                   ratings   were   compared   to    estimates    made    by    the
                                                                   x               subject's  weighted   aggregate   model   (equation   1)   where
                                                                                   (1)  criteria  interdependence   was   not   investigated,   (2)
                                                                                   utilities were . assessed by method of order     10,  and    (3)
                                                                                   criteria  weights  were  established  using  the  ratio  method.
                 -ioo                                                              The results  are  ind'icated in Figure 6.
                                                                                      A    "comrrdttee    model"    was    then    developed     by
                                                                                   averaging  the  individually  determined  weights  and   utility
                                                                                   curves.  The  resulting  evaluations   of   the   twelve   hypo-
                                                                                   thetical  projects  were  compared,  via  the  Pearson   product
                                                                                   moment  correlation  coefficient  with   the   average   ratings
                Utility Assessment Techniques                                      assigned   to   each   project   by   the   ten   members.   The

                   Before an additive  model  can  be  employed,  all  criteria
                measures  must  be  transformed  to  have  the  same  units  of
                value.   One   such   transformation   would   be   to   relate        *Diagnostic criteria are those having a major influence on the
                extent of criteria satisfaction to a utility scale .3,2 0    When  rating given a project.


                154




        Pearson  product   moment   correlation   betw&en   model   and        We   next   investigated    the    variation    between    com-
        ratio estimations was 0.9.*                                         mittee   members   in   the   optimum    values    of    "a".    A
                                                                            significant  variation  would  require   separate   estimates   of
        We   next   calculated   Spearman   Correlation    Coefficients     "a"  for  each  comn-iittee  member.  This   would   be   a   time
        indicating  association  between   averaged   subject   ratings     consun-dng   task   for    both    the    conurdttee    and    the
        and    committee    model    evaluations    developed     using     experimenter.    Individual    evaluation    model     performance
        several values  of  "a"  in  Equation  2.  Results,  Figure  7,     was  measured   at   several   values   of   "a".   The   results,
        indicate   that   model   performance   first   improves   with     Figure 8, indicate that there  is  substantial  variation  in  the
        additional  weight  being  given  to  more  important  criteria     optimum   value    of    "a"    between    individual    subjects.
        and  then  drops  off  as  values  of  "a"  exceed  1.50.  This     Conservatism  does  not  appear  in   all   subjects.   In   fact,
        data would lead us  to  believe  that  subjects  are  conserva-     some  subjects  appear   to   be   radical   in   their   criteria
        tive in weighting criteria and that  equation  2  is  a  useful     weightings.  At  the  very   least,   this   would   indicate.that
        modification of the weighted aggregate model.                       values  of  "a"  for  equation  2  must  be  developed  for   each
        FIGuim    6.-Correlation    Between    Experimentally    Derived    Project    Ratings    and     Ratings     Computed     via     the
                                                       Weighted Aggregate Model.



               1.00-
        .2



        0

                 .80-

        E
        0



                 .60-
        p
        CL

        0


        EL       .40-






                 .20-





                                                                                       -T                                                    T
                     1            2             3            4              5          .6            7            8            9             10
                                                                  SUBJECT NUMBER 2





                                                                             comrrdttee  member  if  they  are  to  be  used   at   all.   This
          *These correlation coefficients are useful  only  as  standards    finding   is   not   too   surprising   when   viewed   with   the
        against which to compare evaluation models  with  "a"  0  l.O..@e    results   of   similar   research   on   subjective    probability
        results cannot, for instance, be used to imply that the committee    estimations.   The   optimum    value    of    a    modifier    of
        model is an effective predictor of  committee  decisions  because
        comniitte-es do not necessarily operate on a majority rule basis.    subjective   probability   estimates   was   influenced   by   the

                                                                                                                                            155





                  importance  of  the  criterion  under  consideration.   They                      Fortunately, it does  not  appear  that  there  i
                  also found substantial  variation  between  subjects.                           improvement  to  be  obtained  by  using   equation
                                                                                                  last column of Figure  8  indicates  for  each  su
                                                                                                  percentage improvement  that  could  be  obtained  Y  Usi
                 FIGURE   7.   A   Compafison   of   Committee   Model    Per-
                               fornzance Using Various Values of "a"                              equation 2 rather than equation 1. In 9 of the
                                                                                                  subjects the improvement is 5'Yo or  less.  This  relatively
                      Value of "a"                          Spearman    Correlation               meager  improvement  in  performance  indicates   that   the
                                                                  Coefficient                     additional  work  required  to  improve  the   basic   model
                                                                                                  may be justified only  when  the  projects  under  consider.
                          o.6                                       .908                          ation will require a large  investment.
                          0.8                                       .923
                          0.9                                       .923                          FurtherModelModi
                                                                                                                         .fications
                          1.0                                       .922
                          1.1                                       .935                            Equation I assumes that all  potential  benefits  will  be
                          1.2                                       .937                          achieved and that the  time  required  to  achieve  each  of
                          1.5                                       .95@                          them is the same. Neither  of  these  assumptions  is  true.
                          2.0                                       .915                          Model  performance  might   be   improved   by   considering
                          2.5                                       .886                          expected benefits modified by  a  present  worth  factor  as
                          5.0                                       .702                          in equation 3:


                                                    n                                      -rti  n+,n                                  -rti
                                                E=z               P(Silyil,---,Yin)@Vig(Xj)e     +E    P   (Si yil       Yin) WjRje
                                                                                                  j=n+l                                                               (3)
                                                                                       C(t)



                          where    Si     =     success of project in  achieving  benefit  i             r      =   the  exponent  of  the  present  worth  factor  re-
                                   Yik    =     degree of satisfaction  of  the  kth  factor                        lating  benefit  utility  to  time   for   achieve-
                                                influencing the success  of  project  in                            ment
                                                achieving benefit i                                      C(t)   =   present worth of project costs'        AU     other
                                   ti     =     time required  before  benefit  is  achieved                        symbols the same as in equation 1.


                                                    FIGURE 8. Performance of Individual Evaluation Models at Va?ious
                                                Values  of  "a  "   Circled   Value   is   Best   Spearman   Correlation   for   Each   Subject.


                                                                                       VALUE    OF     "a"
                                                                                                                                                              Deviation
                      Subject                                                                                                                          from  opt.   for
                            No.      .2         .4          .6          .8        1.0        1.2         1.5        2.0         2.5          3.0           a=]. 0
                              1                 -   -      .955                   .955       .955        .910       .815                                     2
                              2     .771                   .764       .764        .585       .705        .705       .655                                    24.5
                              3                 .809       .832       .770        .760       .707        .5 80      .222                                    12.5
                              4                 -   -      .895       .910        .910       .910        .910       .924                    .937             4
                              5                 -   -      .820       .820        .856       .856        .856       .8LO       .820         .781             0
                              6                 -   -      .960       .960        .960         .9@7      .948       .926                                     5
                              7                 -   -      .895       .895        .895       .895        .916       .938      .9@41         .912             5
                              8     .944        .930                              .930       .930        .950       .965                                     5
                          1   0                 -   -      .810       .810        .868       .853        .810       .783                                     0
                          1   1     .925        (aD        .925       .925        .93        .935        .920       .850                                     0



                   156



             The  basic  concept  behind   equation   3   is   not   new   but         11. Philips,  L.D.,  Hays,  W.L.,  and  Edwards,   W.,   "Conservatism
      much  work  is  still  needed  to  validate   its   potential   and   to         in  Complex  Probabilistic  Inference,"   IEEE   Transactions   on
                                                                                       Human Factors in Electronics, HFE-7, 1966.
      develop   methods   for    estimating    its    parameters.    At    the         12. Philips,  L.D.,  Some  Components  of   Probabilistic   Inference,
      same  time,  there  is  evidence  to  indicate   that   the   model   is         Doctoral Dissertation, The University of Michigan, 1965.
                   1 1 4
      practical.'  ,     "     The     research     reported     here      has         13. Schum,   D.A.,    "Behavioral    Decision    Theory    and    Man-
      evaluated    methods    of    weighting    and     measuring     benefit         Machine    Systems,"    Technical    Report,     Systems#46-4,Rice
      criteria.   We   have    suggested    but    not    evaluated    methods         University,   Program   in   Applied   Mathematics   and   Systems
                                                                                       Theory, July, 1968.
      for    establishing    independent    criteria.    The    results     of         14. Smith,   Lee    H.,    "Ranking    Procedures    and    Subjective
      experiments    conducted    at    the    research    laboratories     of         Probability   Distributions,"   Management   Science,   Vol,   14,
      Monsanto Company        2 4  tend    to    support    the     hypothesis         No. 4, December 1967.
      that   R&D   planning   and   control   models   that   are   based   on         15. Mottley,   C.Nt.   and   Newton,   R.D.,   'The    Selection    of
      subjective   probability   estimates   may   reliably   be    used    as         Projects  for  Industrial  Research,"  OR,  7:6  Nov.-Dec.   1959,
      an  aid   in   project   selection   and   funding.   Other   behavioral         pp.  740-751.   (Propose   Method:   Decision   Theory,   Rating).
                                                                                       16. Hertz  and  Cerlson,  Phillip  G.,  "Selection,  Evaluation,   and
      research9  @l  0  @l  1  indicates  that   the   posterior   probability         Control  of  Research  and  Development   Projects,"   in   Burton
      of  project  success,  P(Sil,  -  -   -,   Yin)   can   be   effectively         V.   Dean   (ed.),   Operations   Research   in    Research    and
      estimated    by    combining,     subjectively     estimated     likeli-         Development, Wiley, 1963.
                                   C,                                                  17. MostcHer,  R.  and   Wallace,   E.J.,   Inference   and   Disputed
      hoods through Bayes' Theorem as follows:                                         Auti2orship:    The    Federalist,    Reading:     Addison-Wesley,
                                                                                       1964.
                                                                                       18. Fishbum,Peter,   "Independence   in   Utility    Theory    with
      P(Silyil,.-.1yin) @ P(Yalsi) ... P(yilsi) P(Si)                      (4)         Whole  Product  Sets,"  Operations  Research,  Vol.  13,  No.   1,
      P(Silyil, - - - Yin',   P(YitlSi)     P(YinISi) P(Si)                            1965.
                                                                                       19. Gustafson,   D.H.,   Unpublished   Doctoral   Dissertation,    The
                                                                                       University of Michigan, Ann Arbor, 1967.
                                  Bibliography                                         20. Fishbum,  P.C.,  "Methods  of  Estimating   Additive   Utilities,"
                                                                                       Management Science, Vol. 13, No. 7, March, 1967.
             1. Friedman,  M.   and   Savage,   L.J.,   "The   Expected   Utility      21. Nadler,  G.,  IVork  Design:  A  System  Design  Strategy,  to  be
             Hypothesis  and  the  Measurability  of  Utility,"   Journal   Of         published   soon,   G.P.   Huber,   Chap.   7,   General   Models-
             Political Economics, Vol. 60 (1962), pp. 463-74.                          Decision Making, in the above book.
             2. Cramer,  R.H.  and  Smith,  B.E.,  "Decision   Models   for   the      22. E@ienrode,   R.T.,   "Weighting   Multiple   Criteria,"    Manage-
             Selection   of   Research   Projects,"   Engineering   Economist,         ment Science, Vol. 12, No. 3, November, 1965.
             Vol. 91 2, 1964.                                                          23. Miller, G.A.,  "Magical  No.  7  +  or  -2:  Some  Limits  on  Our
             3. Miller, J.R. 111,  "A  Systematic  Procedure  for  Assessing  the      Capacity  for  Processing  Information,"  in  Alexis  and  Wilson,
             Worth   of   Complex   Alternatives,"   1967,    Defense    Docu-         OrganizationalDecision-making, Prentice-HaU, 1967.
                                                                                       24. Souder,  W.E.,  "The  Validity  of   Subjective   Probability   of
             mentation Centre, AD 662001.                                              Success  Forecasts  by  R&D   Project   Managers",   IEEE   Trans-
             4. Schoner,  Bertram,   The   Selection   of   R&D   Projects,   un-      actions  on  Engineering  Management,  Vol.  16,  No.   1,   1969.
             published  doctoral  dissertation,  Stanford   University   1965.
             5. Baker,  N.R.   and   Pound,   W.H.,   "R&D   Project   Selection,      25. Edwards,    Ward,    "Conservatism    in     Human     Information
             Where   We   Stand,"    IEEE    Transactions    on    Engineering         Processing,"     in     Formal     Representation     of     Human
             Management, EM I 1, pp. 124-134, December 1964.                           Judgment, Benjamin Kleinmuntz, ed., Wiley, 1968.
             6. Cetron, M.J.,  Martino,  J.,  and  Roepcke,  L.,  "The  Selection
             of     R&D     Program     Content-Survey     of     Quantitative
             Methods,"    IEEE    Transactions    on    EngineeringManagement,
             EM-14, pp. 4-14, March 1967.                                                  Comments on an Evaluation Model for the
             7. Heilman,  L.P.,  "A  Value  Measure   for   Selecting   Proposals                    Regional Medical Program
             for    Research    Grant    Support,"     Operations     Research
             Division, The Johns Hopkins Hospital, Baltimore.                                  VERNON E. WECKWERTH, Ph.D.
             B. Churchman, Ackoff, and Arnoff, Introduction        to   Opera-
             tions  Research,  John  Wiley  and  Sons,  Wiley,  1957,  Chapter         How  generic   one   wishes   to   make   a   model   depends   on
             6.
             9. Abernathy,   W.J.   and   Rosenbloom,   R.S.,    "Parallel    and      how  far  one  is  displaced  from  the  reality  of  application.
             Sequential   R&D   Strategies:   Application    of    a    Simple         The  creator  of  a  model  in  the   Ivory   Tower   can   easily
             Model,"   IEEE   Transactions    on    Engineering    Management,         assume  away  the   inconsistencies   of   the   world.   To   the
             Vol. 15, 1, 1968.
             10. Huber,  G.P.,  Sahney,  V.K.  and  Ford,  D.L.,   "A   Study   of     day-to-day   doer   of   what   could   be   called    evaluation,
             Subjective   Evaluation   Models,"    Social    Systems    Resear         there  is  no  way  to   assume   away   the   problems   in   the
                                                                             ch
             Institute   University   of   Wisconsin,   Madison,   Firm    and         world.  Judgment  is  totally   pragmatic.   The   applied   model
                      I
             Market Workshop Paper; 6817.                                              either represents what is or it is rejected.
                                                                                                                                                        157




             As  a  -group,  you  have   been   subjected   to   some   high process  is   dynamic,   that   outcomes   (and   benefits)   are
             level  forms  of  abstraction  in  terms  of  starting  points, what  we  seek.  Change  is  a  means  or   an   observation   of
             preliminary  strategies,   ends-in-view,   and   implementation means, not an end.
             with  stated  intents  of   transformation   of   the   system.    The  generic  nature   and   benefits   of   the   model   for
             This  introduction  will,   by   virtue   of   that   type   of evaluation  proposed.here  are  one  of   a   system   possessing
             presentation, try to be as abstract and obtuse.                 six ordered elements:
             You  have  been  told,  and  by  report  most   of   you   have    I     .Context - That  piece  of  the  world  under  considera-
             acquiesced at least,  to  the  proposition  that  the  RMPs  do          tioii as it is found at a given point in time.  This  is
             not  form  a  closed,   but   an   open   system.   That   open          the "where" for the RMPS.
             system is a  seductive  proposition.  It  is  as  seductive  an         2. Content  -  The  inputs  of  men,  money,  and  material
             alternative  as  many  propositions  are  when   the   ends-in-          in   whatever   extant   form   they   are    possessed,
             view are mundane or repetitive. If  the  system  is  one  of  a          whether  or  not  they  are  identified,   ordered,   or
             static nature  -  closed,  just  input,  throughout,  and  out-          measured.  This  is  the  "who"  and  the   "what"   for
             put  -  which  is  routine,  reproducible,  repetitive,  stand-          the RiNIPs.
             ardized like a  ball-bearing  production  system,  then  it  is         3. Process -  The  way  the  content  is  put  tocether  in
             even easier to be seduced.                                               some  functional,  or-anized  way,  both  in  terms   of
              I  propose, however,      that    the    open-ended     system          the  static,  i.e.,  repetitive  closed  system  meaning
             embrace is as deceptive    in  the  argument  for  it  as   the          like a production process, as well as in  terms  of  the
             argument that any living,  on-going process  like  life  itself          dynamic  system  of   self-modification   and   directed
             is  better  than  a  dead-end.  Even  the   old   truism   sum-          chance. This is the  "how  and  when"  of  the  F-LNIPS.
             marized  that  belief  from  antiquity  -  you  only  have  one     These  three  elements   are   in   fact   the   indepe?ident
             life to live - you can't  live it over  a-ain  -  you  are  all  variables  for  any   RMP.   Each   RMP,   by   its   existence,
             different. Each RMP is    unique   and   dynamic.    For    our  structure,  and  function,  delimits  and  encompasses  at   any
             own mental health, could we believe otherwise?                   point in time the dependent elements which are:
              There are two points to be made:                                       4. Output   -   This   is   the   product   produced   from
               1.  A  model  is  only  a  model.  It  can   be   made   suf-          content in the process in  use  within  the  context  of
             ficiently complex  so  that  it  fits  within  a  predetermined          the  operation.  These  are  the   observable,   record-
             degree  of  closeness  to  perceived  reality   so   that   you          able,   reproducible,   measurable   "why's"   of    the
             choose to believe it and use it, i.e., you  choose  to  believe          RMP's using  the  classical  definition  of  evaluation,
             that the model fits  your  perception  of  things  rather  than          i.e.,    cornparing    accomplishment    with     stated
             concluding  that  life  is  a  haphazard  sequence  of  chaotic          objective. These typically  form  the  evidential  basis
             happenstances.  It  depends  on  your  view  of   the   meaning          of  hard  fact  observation,  on  which  "output   only"
             of  change  -  from  what  to  what  in   what   direction   at          evaluation is based.
             what  rate.  In  fact,  one  could  play  with  the  words  and          5. Outcome  -  These  are  the  time-delayed  impacts  that
             redefine status quo to be  a  constant  rate  of  change.  What          demonstrate  whether   the   outputs   were   any   more
             then  happens  to  the  obligation  to  transform  the  system?          than just  outputs  at  the  points  in  time.  Outcomes
             It   is   merely   the   difference   between   evolution   and          (over  time)  show  the  time-delayed  impacts  of  out-
             revolution.   Orderly   change   with   a   built-in   planning          put  on  health  states,  disease   incidence,   updated
             sequence  is  a  necessary  part  of  any   dynamic   organiza-          practice,  altered  organization,  complete   and   con-
             tion.  I  am   concerned   that   what   the   "change"   model          tinLIOUS  care  delivery,  equalized  access,  cost  ef-
             implies is best  described  as  the  "rocking  chair  model"  -          fectiveness,  etc.  These  Should  be  the  "why's"  for
             giving the health field  a  sense  of  movement  but  no  sense          the  RMP,  but  these  kinds  of  "why's"   are   either
             of  direction.  Restated,  "evaluation  of  transformation   of          too soft in the data  sense  or  take  so  long  in  the
             the  system"  requires  an  articulate  statement   of   change          time sense that they are  onlv  rarely  used.  The  out-
             from where we are  to  where  we  intend  to  go  by  a  series          put "why's"  are  accepted  as  the  basis  for  funding
             of defined steps.                                                        perpetuation and classical evaluation.
                2.  If  the  end-in-view  is  looked  at  only   as   input,          6. Benefit - This is the ultimate "why."  It  is  also  the
             throughout,  output,  rather   than   in   the   structure   of          vaguest and "softest" element  in  evaluation.  It  gets
             input, content in a context,  then  I  propose  that  it's  the          at the associated, serendipitous, as  well  as  intended
             wrong  model.  I  propose  that  the  definer   of   a   closed          effects  that  are  evident  in  an   altered   context.
             system  has  forgotten:  the  context   of   uniqueness,   that          Benefits   can   be   represented   in   imputed    cost

             158



            benefit  terms  or  in  a  gestalt   sense   of   total   changes       The    framework    of    the    conceptual     model     represented
            or  differences  in  the  context   at   a   subsequent   po-int      here  which   has   as   its   basis   a   markovian   process   is   a
            in   time.                                                            model   which   may   not   be   explicit   enough    for    day-to-day
          Obviously  each  RMP  is   unique   if   one   considers   a   suf-     doing   in   RMP    but    the    sequence    -    context,    content,
       ficiently    large    number    of    items    of     context.     The     process,   output,    outcome    and    benefit    -    is,    however,
       14 wheres"    are    unique     by     combinatorial     reduction     to  applicable  at  all  levels  -  be  it  to  projects,  to   the   local
       absurdity.                                                                 advisory  or  regional  advisory  groups,  to  the   core   staff,   to
          Each  RMP  is  also  unique   if   one   considers   the   specific     the   board,   to   separate   RMP'S,   or   to   the    RMP    as    a
       combination   of   the   process   (how's)   chosen.   The   how   for     program.
       each    RMP    given    merely    combinatorial    structure     makes       It  should   be   clear   that   I   believe   that   evaluation   is
       each obviously different from any other.                                   merely   a   means   of   responding   to   the   question    of    the
          Given     these     unique,      independent      variables      in     46 social  good"   of   the   RMP.   It   can   be   answered   relatively
       combination   the   outputs   will   be    by    definition    unique,     or  absolutely.  It  is  simply  a   judgment   or   opinion   of   the
       depending   on   how   crude   or   fine   one   chooses    to    make     person  with  the  right  to  decide.   This   point   is   made   very
       the output units.                                                          clearly in the pa per, A Tool or a Tyranny.
          The   outcomes   will   also   be   obviously    unique,    depend-       One    last    comment    before    the    paper:    Evaluation    is
       ing, of course, on what time frame is used.                                distinct    from    assessment.    Assessment    means    to    produce
          The   benefits   must   of   necessity   be   unique   since    the     the   evidential   base   by   which   statements   such    as    more,
       context    was.                                                            less,  or  equal.   can   be   made.   Evaluation   means   to   attach
          Obviously  one  can  chose  or   not   to   be   seduced   by   the     such  words  as   good   or   bad   to   those   assessment   findings.
       age-old   proposition   that   each   is   different   from   everyone     It  is  necessary  to  be  clear  on   the   value   judgment   meaning
       else,  i.e.,  each  RMP  is  an  open,  not  a  closed   system.   The     of   evaluation   versus   the   quantitative   meaning   of    assess-
       issue  is  not  that  RMP's  are  open   or   closed   and   therefore     ment.  For  example,  it  is  possible   that   the   same   level   of
       different  but   how   different.   How   different   must   they   be     assessment   data   could   be   judged   to   be    "bad"    in    one
       so   that   being   different   makes   a   difference?   'Me   burden     context-content-process    combination     and     for     the     same
       of   an   evaluation   is   to   categorize,   order,   measure,   and     level   judged   to   be    "good"    for    a    different    context-
       interpret  the  differences   -   either   relatively   or   absolute-     content-process     combination.     Obviously,     an      evaluation,
       IY.                                                                        in   my   opinion,   can   be   good   or   bad,   better   or   worse,
          The  evaluation   issue   at   hand   is   answering   the   simple     whether   the   assessment    data    is    identical    in    measured
       question,   "What   social    good    has    the    RMP    produced?",     quantity   or   order.   This   ends   the   introduction   and   leads
       where   in   fact   the   evaluators   have   the   right   and    the     to  the  delivery  of   the   formal   paper   which   Mr.   Ichniowski
       obligation to defme "good.                                                 asked   me   to   discuss   with   you,   weaving    into    it    your
          Or  restated,  what   are   the   outcomes   (or   benefits)   upon     questions and comments.
       which  RMP  is  to  be  judged?  On  what  basis  are   they   to   be
       held accountable?                                                                    On Evaluation: A Tool or a Tyrannyl
          If  it  is  to  be  on  the  basis  of  a  change  or  the  fancier                 VERNONE.WECKWERTH,Ph.
       euphemism,   "transformation    of    the    system,"    there    must
       be a clear statement of what "good" means in              terms     of
       changed  to  what,  from   what,   at   what   cost/unit   of   change        Evaluation  is  a   ten   letter   word   -   in   English.   Beyond
       in  what  time  frame   -   not   just   a   nondirectional   rockiniz     that   statement   the   only   consensus   about   evaluation   is   a
       chair    model.                                                            lack  of  consensus.  This  paper  is  a  series  of  loosely   related
                                                                                  topics   which   attempts   to   give    some    limited    perspective
          If it is  to be  on  the  basis  of  process,  then  the   rate   of
       change and the time horizon must be defined.                               into  what  evaluation  means,   how   and   why   it   is   done   and
       .   One   would   have   to    conclude    that    the    goals    and
       purposes   of   RMP   were   intentionally   stated   in   the   vague
       way  they  were   because   there   was   no   desire   to   be   held        iThis  paper  is  distributed  for  general  interest.  Reproduction
       accountable  or  there  was  no  clear   raison   d'etre   for   them.
       @'Apparently,   it   is   now    becoming    necessary    to    define     in whole or part is permitted if proper credit  is  given.  'Ibis  dis-
                                                                                  tribuflon neither  expresses  nor  implies  approvAl  of  its  contents
       "good"  in  terms  of   the   process   of   change   without   saying     by  the  Project,  the  University  of  Minnesota   or   the   Granting
       from what to what at which rate in what time.                              Agencies.

                                                                                                                                                      159





               in what  ways  the  vernacular  use  of  the  term  in  manage-    value in the value system of the one with the right
               ment relates to  the  discipline  use  in  research.  It  high-    decide what is to be measured.
               lights four points:                                                   All of us know the single most common a
                  I .There is no one way to do evaluation.                        of evaluation is to the evaluation of the qualit)
                  2. There  is  no  generic  logical  structure   which   will    care. Quality of care, we know, serves to expl,
                     assure a  unique  "right  method  of  choice."               are high, productivity  low  or  demands  too  great.  It  will
                  3. Evaluation   ultimately    becomes    judgment and    will   serve  here  as  the  example  to  trace  the  development   of
                     remain so, so long as  there  is  no  ultimate  criterion    how we  arrived  at  where  we  are  in  the  Art  and  Science
                     for monotonic  ordering  of  priorities,  and:               of evaluation.
                  4. the  crucial  element  in  evaluation  is   simply:   who
                     has the  right.  i.e.,  the  power,  the  influence,  the         EVALUATION OF THE QUALITY OF CARE.
                     authority, to decide.
                                                                                     Consider  the  word  quality..  It  has  the  same  root  as
                                                                                  qualities. Originally, qualities were  selected  as  the  basis
                                   INTRODUCTION                                   for the first quality  of  care  studies.  The  first  question
                  A discussion of evaluation will lead  to  no  useful  result    asked upon beginning a quality  of  care  study  is,  "what  is
               unless  one   states   at   the   beginning   what   evaluation    to  be   included   to   be   measured?"   That's   where   the
               means;  why  evaluation  is  being  done;  to,  by,  with,  and    laundry lists  began.,  Out  of  that  long  list,  a  set  was
               for  whom;  what  is  the  intended  outcome   of   evaluation;    chosen  by  whosoever  had  the  right  to  decide.  Typically,
               how  does  one  "evaluate   evaluation"   and   who   has   the    the qualities  were  chosen  because  they  either  had  to  be
               right  to  decide  the  what,  why,  where,  when,   how,   and    present or were desirable. Thus:
               who involved in evaluation.                                           Development one:  A  list  of  qualities  was  presented  (a
                  Evaluation  includes  within   it   consideration   of   ap-    value system  value  decision)  in  which  merely  presence  or
               proaches,   methods,   techniques,   and   uses;   a    process    absence of each quality was recorded.
               versus   a   goal   approach;   program    veitus    individual       An array was generated with  a  laundry  list  on  the  left
               objectives;  needs,  demands,   desires,   and   their   inter-    and two  columns  to  check  either  absent,  score  it  0,  or
               relationships.  It  includes  objectives  versus   goals;   ac-    present, score it 1.
               tivities  versus  accomplishments:   inputs   versus   outputs;       The  measure   of   quality   was   therefore   simply   the
               outputs   versus   outcomes,    outcomes    versus    benefits;    number  of  qualities  present  divided  by  the  total  number
               effectiveness  versus  efficiency;  structure  versus  qualifi-    of  qualities.  Low  quality  meant:  a  proportionately  small
               cation; and  so  forth.  It  includes  the  context,  the  con-    number  of   qualities   present;   High   quality   meant:   a
               tent,  and  the  process;  the  served  and  the  server;   the    proportionately large number of qualities present.
               individual  and  the  group;  the  quantity  and  quality;  and       The  first  use  of  evaluation  of  quality  was  to   make
               others.  It  includes  when  and  where,   wi'th   or   without    present the qualities that were absent.
               feedback, and how often.  It  includes  a  research  versus  an       As time  passed,  it  became  obvious  that  some  qualities
               administrative   meaning.   It   includes   vernacular   versus    were more important than others.
               discipline definition. It includes much more than this.               Development   two:   A   weight   was   attached   to    the
                                                                                  qualities reflecting the importance of each quality.
                                                                                     Obviously,  these  weights  were  attached  based   on   who
                            Dictionary  Definition  of   Evaluation               had  the  right  to  decide.  The   array   was   modified   by
                  The dictionary  says  that  to  evaluate  means  "to  deter-    adding a column of weights.
               mine  or  fix  a  value  of"  or  "to   examine   and   judge".       The  measure   of   quality   thus   became   the   sum   of
               These two meanings give  the  first  insight  into  evaluation.    weighted  presence  of  qualities.  As   time   passed,   these
               The term, "evaluation" has value as its root.                      weights  became   somewhat   "standardized"   and   there   de-
                  Using  the   dictionary   definition,   one   can   separate    veloped  what  we  now  know  as  the  setting   of   standards
               papers  and  practice  into  those  to  whom  value  means:  1)    of quality of care. It was  a  way  of  sayin2  what  qualities
               a  number  value,  or  2)  a  value  system  value.  These  two    had to be present. High value on a quali-,     was   reP@.,ctej
               groups  can  each  be  divided  into  those  who  are   process    in a  large  weight.  Sometimes  qualities  wel,-c  ;udged  and
                                                                                                                                   i
               versus  goal  oriented.  What  is  commonly  n-dssed  is   that    weighted so highly  that  absence  was  identical  to  a  veto.
               any element (variable, quality,  attribute)  that  one  selects       Development  three:  Place  a  sufficiently   large   weight
               to  be  included  for   number   value   measurement   is   the    on any one quality so that  if  it  were  absent  the  "quality
               result of someone's priority in its selection, i.e,  it  is  of    of care" would assuredly be "low".

               160




      Quite  soon  the   simple   dichotomy,   absent   or   present,    two   completely    separate    endeavors    with    the    practi-
   ias  as  unacceptable  as  was  the  equal   weighting.   It   was    tioners  worse  o@@f   than   before,   since   "evaluation"   must
   [atural  to  expand  the  measure  of  presence  from   0   or   1    now  mean  something   detached   from   day   to   day   practise,
   inly to 0, 1,  2,  3  ...  to  as  many  "units  of  more-so-ness"    and in use most likely plinative in addition.
   s  was  useful.  These  degrees  of  more-so-ness  did  not   have
   o  be   whole   units   or   integers.   These   "measures"   tied            WHAT CAN THE PRACTITIONER DO?
   asily  into  "standards"  since  some  standards  were   in   fact         Every practitioner has taken  at  least  the  first  steps  in
      level  of  the  degree   of   presence   rather   than   merely    evaluation.    Each     practitioner     must     determine     how
   iresence or absence.                                                  sophisticated  he  wants  to  got  and  be   prepared   to   defend
      Development   four:   Specify   a   measure   of   the   degree    where  he  stops,  if  he  stops  short  of  research  design.  The
   @f presence for qualities.
                                                                         steps are simple:
      Such   a   development   was   conceptually   easy   to    come         I .Choose the qualities.
   )y,  but  operationally  very  difficult  to   achieve.   However,         2. Attach weights reflecting priorities.
   @hat   mechanical   difficulty   didn't   deter   the   doina   of
                                                              t,              3. Specify measures of degrees of presence.
   Evaluation   of   quality   of   care.   The   procedure    merely         4. Combine   the    created    array    in    some    functional
   )ecame a  listing  of  included  qualities;  the  listing  of  as-          form(s).
   ;ociated   weights,   and   an   associated   measure    of    the         S. Generate distribution(s) of those function(s).
   legree   of   more-so-ness   but   combined   in   some    "arith-         6. Set  the   cut   off   points   to,   determine   where   the
   netical or number value way".                                               quantitative     representation     concurs     with      his
      Once   that   "arithmetical   way"    was    determined,    one          judgment of desired quality. '
   nerely  proceded  to  specify  the  distribution  of  the   values         He  can  call  in  help  at  any  step;  develop  any   number
   Lnd  define  low  and  high  quality  on   the   scale   of   that    of  experimental  designs   and   number   value   functions,   but
   'neasure.                                                             ultimately  that  evaluation  will  bofl  down  to  who   has   the
      There  were,  however,  in  the  40's  and  50's   many   other    right to decide and who renders the judgment.
   'orces   operating;   new   knowledge   of   statistics,    proba-
   )flity   theory,   experimental   design,   and   other   measure-             ACCEPTED OPERATING DEFINITION
   nent   technology.   People   were    increasingly    dissatisfied                          OF EVALUATION
   with   simple   arithmetic   ways,    including    the    implicit
   assumptions  of  independence  among  qualities   in   the   list.         Dictionary  definitions  help  to  give   insight   into   the
      Those   faced   with   evaluation    were    soon    developing     "whats"   of   concepts.   Operational   definitions    help    to
   Sophisticated   research   designs   with    fancy    mathematical     give insight into "how's" of concepts.
   -nodels,  formulae,   and   techniques.   The   limit   functions,         The   most   commonly    accepted    operational    definition
   Interdependence   of    qualities    handled    by    multivariate     -of   evaluation,   the   "how",   is:   Compare    accomplishment
   correlations,   covariance,   factorial   designs   with    inter-     with  stated  objectives.  This  is   itself   a   goal   oriented
   ictions,  simple  and  main  effects  plus  factor  analysis   all     definition.  The  objectives  are  analogous  to   the   qualities
   Decame   involved.   In   fact,    these    developments    became
   the  fife  blood   of   the   biostatisticians   and   the   death     or elements chosen in the quality of care example.
   potions  of  most  of  those  involved   as   delivering   practi-         Since the  operational  definition  is  so  simple  -  why  is
   tioners - both clinical and administrative.                            evaluation  so  tough?  Let's  look  first  at  that   operational
      Development   five:   Only   qualities   with    experimentally     definition.  In  it  five  assumptions  are  made:  1)  objectives
   determined   measurability,   validity,   and   reliability   were     are  stated;  2)   in   measurable   terms;   3)   accomplishments
   permitted to enter quality of care evaluations.                        are  documentable;  4)   in   the   same   measurable   terms   as
      As   a   result,   the   evaluation   of   quality   of    care     the  objectives;  and   5)   one   knows   what   compare   means,
   developed  to   such   a   mathematically   sophisticated   extent     i.e., what is to be done?
   that  those  who  first   desired   it   and   created   it   were          WHAT USUALLY LEADS To DIFFICULTY?
   bypassed  and  found  that  it  couldn't  be  applied  on  a   day
   to  day  basis.  Hence,  evaluation  became   so   detached   that         First:  Objectives  aren't  stated.  Goals  versus  objectives
   now it is  not  recognized  as  a  part  of  the  ongoing  process     are   rarely   differentiated.   Purposes,   goals,   salutes   to
   of  clinical  management,   or   program   administration,   i.e.,     mother and  country  -  and  lots  of  other  things  are  usually
   planning,   orgaiiizing,    assembling    resources,    directing,     stated -  but  not  objectives.  An  analogy  may  be  helpful  to
   controlling,  replanning,  reorganizing,  etc.  It  is   seen   as     distinguish  objectives  from  goals.   Consider   the   sequence,

                                                                                                                                         161





              1/4, 1/9, 1/16, ... I/n2, .... In this case, that  sequence  of               SOME COMMENTS ON MEASURES
              terms will approach a limit.  That  limit  is  analogous  to  a        Frequently,  a  quality  selected  in  evaluation  has   no
              goal.  The  individual  terms  in   the   sequence   are   like    direct measure or has  one  which  is  too  costly  or  tedious
              objectives.                                                        to  obtain.  There  frequently,  however,  are  associated   or
                 Second:  Even  if  objectives  are  stated,  most  of   them    indirect measures which  can  be  used  in  lieu  of  a  direct
              are  not  independent.  In  fact,  they   frequently   are   in    measure.
              conflict  with  each  other  and  rarely   would   their   sum-        Some   measures  which  are  indirect  are   called   proxy
              mation  add  up  to  the  program  goals.   Additionally,   the    measures.  This  obviously  means  that  they  stand  in   lieu
              state  of  the  art  (or  science)  of   evaluation   has   not    of  what  is  desired  to  be   measured.   Frequently.   proxy
              developed   means    of    measuring    most    value    system    measures  in  evaluation  are  used  to  predict   or   monitor
              objectives.   Thus,   our   measurement   ineptness    reflects    activities, and are useful because  of  their  high  associated
              both our ignorance'and our errors.                                 though not causal relationship.
                 Third:  Even  given  stated   objectives   and   appropriate       For  example,  the   number   of   individuals   usinc,   an
              measures,  we  likely  can't  enable   the   documentation   of    emergency   room   in   hospitals   is   associated    withcthe
              accomplishment.    Frequently,    the    measures    are    too    phases  of   the   moon.   For   administrative   purposes   of
              complex  or  the  day  to  day  documentation  is  either   too    staffing and the provision of  service,  it  is  not  necessary
              tedious, or not  visibly  relevant  to  thejob  being  done  on    to  know  the  direct  or  causal  elements.  However,  if  one
              an ongoing  basis.  As  a  result,  we  substitute  approximate    were to  change  the  pattern  of  service  "demand"  it  would
              measures or frequently just  get  lost  in  the  data  acquisi-    be  necessary  to  know  cause  -  and  the  relationship,  and
              tion  problems  and  consume  so  much   time   and   resources    not   operate   purely   with   proxy    measures.    Commonly,
              that we judge  that  documentation  isn't  worth  it  -  unless    "Comparison"  in   evaluation   highlights   differentials   in
              it is an experiment in which service is    only   a   necessary    such  proxy  measures.  Actions  are  then   frequently   taken
              evil or a necessary  context.                                      on  forces  putatively  "causal"  but  to  the  dismay  of  the
                Fourth:   In   the   rare   event   that    evaluation    has    action  taker,  produce  no  change  because   -   in   medical
              measurable   objectives    and    documented    accomplishment,    jargon - he treated the symptom and not the disease.
              commonly  nobody  knows  what  to  do  with  it!  Or   if,   in    -  These  experiences   further   alienate   the   practitioner
              fact,  someone   knows,   the comparison  will   still   depend    and  result  in  Ws  questioning   even   more,   "NNhy   eval-
              entirely  on   the   judgment of  whoever  has  the  right   to    uate?"
              decide what to do with it.
                A  facetious  and   trivial example   may    help:    suppose                    USES OF EVALUATION
              that  an  MCH   Program   has an  objective  that  75%  of  all
              mothers-to-be are to  be  seen  by  an  O.B.  physician  before       No attempt   is made to provide  a  laundry  list  of  uses.
              the third  trimester.  We  find  that  73%  do  in  project  A,    An attempt is  made,  however,  to  fit  "evaluation",  in  the
              and 77% do in  project  B.  Now  what?  If  n  is  big  enough,    non-experimental   design   meaning,   into    day    to    day
              the difference  may  be  statistically  significant.  So  what?    operations.
              Is  the  project  with  77%  awarded  a  gold  star   or   more       First,  we  must  answer,   "For   what   purpose   is   the
              money?  Does  the  project  with  73%  get  a  budget  cut?  In    evaluation  done?"   Regrettably,   the   answer   that   would
              fact, is it not true that  since  both  missed  the  objective,    now   be   given   (if   honestly   ascertainable)   is,   "The
              that both  are  bad?  Why  is  doing  more  an  ultimate  good?                   law requires it."  That  is  regrettable.  In  a
              After  all,   the   77%er   allocated   more   resources   than    sense, the  requirements  in  the  law  reflect  a  failure  on
              should  have  been  to  that   objective   and   that   project    behalf  of  those  responsible  for   programs   to   document,
              could be  "penalized"  for  n-tisahocation  while  the  73t7oer    accomplishment   in   an   orderly,   measurable,    and    ar-
              should  be  given  more  resources  because   it   was   under-    ticulate  manner  that  met  the  desiderata  of   those   with
              allocated.                                                         right to make laws.
               The overriding question  being  asked  is,  is  the  classical       With  the  legal  emphasis  on  evaluation  and  the   mean-
              operating  definition  of   evaluation:   Compare   accomplish-    ing  of  the  term  to  be  the  rigid   mathematical,   numer-
              ment with stated objective the end of e       valuation?     Is    ological, hard fact one, the  day  to  day  intuitive  or  soft
              evaluation to  be  only  descriptive?  Is  it  merely  to  tell    data  meaning  and  use,  has  been  both  lost  and   rendered
              how it  was?  If  not,  is  it  to  include  ground  rules  for    unacceptable.
              translating  description  into   prescription,   i.e.,   admin-       Evaluation  has  always  been  -  in  the  dictionary  mean-
              istrative action?                                                  ing of the  word  -  present  in  anyone  who  was  responsible


              162




      in  his  work,  and  had   a   personal   accountability   for   his    you  stated  it  would.  You   have   evaluated   at   the   program
      acts.  Evaluation  is  inherent  in  the  process  of   adniinistra-    level.
      tion - be it clinical or program management.                              Although  there  i.s  a  reasonable  basis  for  saying  there  is
        Anyone    who    manaaes    successfully    either    a    program    a  single  generic  process  in  doing  evaluation,  the   qualities
      or  a  patient  goes  through   some   orderly   staces,   beginning    chosen   for   patient   management   are    so    different    from
      with  planning:  that  is  deciding  what  is  to   be   done;   by,    those     chosen     for     program     management     that     the
      with,  for,   and   to   whom,   with   what   materiel,   at   what    singleness  of  the  process   is   lost.   In   fact,   because,the
      time,   in   what   sequence,   at    what    places,    for    what    priorities  assigned  to  the  qualities  in  patient  versus   pro-
      intended outcome.                                                       gram  evaluation  are   so   discrepant,   conflict   has   resulted
        Thus   the   planning   is   the   what   step   in   the   admin-    in the whole health care delivery system.
      istrative   or   management   process    with    the    how    steps
      being   organizing   the   what,   asseniblinc,    the    resources,                                     THE QUALITIES
                                                   C,                                  HOW TO CUIOOSE
      directing the delivery and controlling (or      supervising,      or      Since  all  of  us  come  from  rigorous  scientific  fields,   we
      monitoring) the operation (or performance).                             almost    without    thought    believe    we    choose    qualities
        An   inherent   part   of   the   management   process   is    its    based  on  the  facts.   What   one   means   by   "based   on   the
      evaluation.   That   examination   and    judgment    in    delivery    facts" necessitates some expansion.
      of care  is  used  as  feedback  to  alter  the  process  or  treat-      For this paper, consider four groupings of facts:
      ment   of   choice   in   order   to   replan,   reorganize,   reas-      First:  Theoretical  facts.  Starting   withgivens   and   a   set
      semble, redirect, control, etc., ad infinitum.                          of   known   theoretical    relationships,    one    by    deductive
        Clearly,  evaluation  has  been,  is,  and  always  will   be,   a    logic   can   arrive   at   some   qualities   which   are   to   be
      part  of  such  a  management  process  -  be  it  for   a   program    included in evaluation.
      'or a patient.                                                            Second:   Dogmatic   facts.   Dr.   Lebon   (that   spells   nobel
        The  similarity  in  the  process  can  be   seen   if   we   move    backwards  -  and  he  has   one   of   those   prizes   and   don't
      from  the  individual  care  of  a   patient,   through   a   cohort    you forget it) says this  is  a  fact  -  and  it  is.  In  general,
      of  patients  to  a  program.   Consider   yourself   first   as   a    these  are  the  qualities  which  those  in  positions  of   power,
      physician   beginning   with   a   work-up.    You    first    chose    influence, or authority include in evaluation.
      input facts, i.e.,  qualities,  such  as  lab  tests,  signs,  asked      Third:   Pragmatic   facts.   Those    which    are    based    on
      symptoms,   soundings,   touchings,    etc.,    plus    using    the    astute  observations,  with  data   acquired   from   day   to   day
      history  to  assess  the  patient,  derived  mentally   a   set   of    practise   which    every    intelligent    practitioner    gathers.
      weights  of   what's   important,   arrived   at   what's   relevant
      (by  degrees   of   presence   plus   weighted   priority),   deter-    These   form   the   basis   for   selecting    another    set    of
      mined  a   most   probable   "value"   or   judgment   (or   evalua-    qualities.  In   general,   they   derive   from   "experience   and
      tion)   and    rendered    a    care    plan.    You    subsequently    demonstrated use ..."
      compared   this   to   what   happened   to   the    patient    and,       Fourth:   Experimental   Research   Facts:    These    are    the
      depending  on  the   outcome,   either   altered   the   care   plan    facts  derived  from  research   studies   which   meet   the   most
      or   reinforced   your    confidence    in    your    own    medical    rigid   of   experimental   design   requirements.   The   resulting
      judgment, i.e., you evaluated on a one case basis.                      qualities  are  chosen   by   approaches   and   methods   such   as
         Consider  next  a  cohort  of  patients.   You   look   at   them    factorial  designs,  controlled  probability   selection,   or   any
      as  a  group.  You  select  another  set  of  qualities   (some   of    of  the  research  statistical  methods  that  strikes   fear   into
      which   are   different   from   the   case    specific    qualities    most day to day practitioners.
      chosen  in  the  one  patient  sense)  and  look   at   the   cohort       From  these  four  fact  bases,  one  can   get   the   qualities
      from  a  view  of  those  qualities  being  a  set  of  intertwining    to  be  used  in  any  evaluation  schema.  It  is  here  also  that
      degrees   of   presence   and    priority.    You    mentally    and    standards  with  which  we  are   so   obsessed   in   health   care
      numerically  measure  and  then   compare   the   results   of   the    delivery are included.
      cohort to what is "good medical practise".
          At  the   program   director   level,   you'd   look   at   more             W14Y MUST ONE USE EVALUATION?
      than  only   physician   case   management   for   either   individ-       If  one   is   the   perfect   clinical   practitioner   or   the
      uals  or  for  his  cohort  and  include  the  other   health   care    perfect  program  director,  his   intuitive   ongoing   soft   data
      functional  services,  living  conditions,   or   what   have   you,    system   would   be   "evaluating"   without   need   for   a   hard
      that  are  qualities  of  the   "program"   and   go   through   the    fact  base.  But,  since  perfection  is  not   a   human   reality,
      same  process   to   deternii.ne   whether   it   accomplished    what  one  must  set  up   a   hard   fact   data   system   to   document

                                                                                                                                               163





                accomplishment.  The  less  prestitious  one  is,  the   more        Consider the old fashioned  grade  school  report  card..,
                subject one is to the  "tyranny  of  hard  fact  evaluation".    The "qualities" are analogous to items like courses
                   Since  one  cannot   get   continuous   evaluation,   some    math, others like art, others like deportment.
                choices of time intervals  must  be  made  -  hourly,  daily,        By analogy, three groupings of qualities of repo
                                                                                                                                             rt
                weekly,  monthly,  quarterly,  yearly,  or  what   have   you    card items are apparent:
                for  ongoing  programs.  Evaluation  of  single   shot   pro-        I .Those   that   have   an   inherent   measurement   ab-
                grams are relatively easy if only a "final" evaluation is  to           soluteness  in  them  (even  though  the  measure   may
                be made. One must determine if  feedback  is  to  be  used  -           be arbitrarily defined) like  feet,  inches,  etc.  The
                of what kind,  and  how  often.  If  so,  how  does  feedback           units have  a  meaningful  metric  on  the  scale.  The
                fit into  a  subsequent  round  of.  evaluation?  Is  it  now           mathematical formulas work beautifully.
                another  quality   or   element?   If   one   does   feedback        2. Those qualities  that  have  an  inherent  relative  or
                "evaluation" with  the  intent  to  alter  the  program,  how           inore-so-ness  meaning  to  them  but  lack   absolute.
                does one now evaluate the effect of evaluation?                         ness  such  as  strongly  agree,  agree,   indifferent,
                                                                                        disagree, strongly  disagree.  Again,  the  mathematics
                                                                                        is reasonably easy to apply.
                            APPROACHES To EVALUATION                                 3. Those  qualities  that  are  named   or   categorizable
                   No  attempt  to  be  either  scholarly  or   complete   is           only. These are those qualities  that  either  have  no
                intended  here.   Only   three   commonly   used   approaches           inherent measure of  absoluteness  or  relativeness  or
                are included:                                                           that  as  yet  aren't  understood  well  enough  to  be
                   First:  Very   commonly,   programs   are   subjected   to           measured. It is with these, where real difficulties  in
                periodic  review.  These  "evaluations"   are   made   by   a           the  mathematics  are  found  because   the   weighting
                squadron of outsiders. Let us call  this  the  J.D.A.  -  the           is not inherent nor is there a logical  way  to  attach
                judgment day approach.                                                  priority values.
                   The big brother squadron,  usually  called  a  site  visit        Since  every  program  or  practise  includes  all   three
                team,  comes   in   on   judgment   day.   The   concern   is    kinds  of  qualities,  we  must,   in   our   wisdom   choose,
                obvious, "Will one be judged for sins or  virtues  using  the    weigh,   scale,   combine,   and   then   compare    to    the
                same qualities that one has used to live by?"                    objectives, i.e.,judge the result.
                   Second:  Another  commonly  used  approach   is   one   of        We  render  an  evaluation.  So   what?   We   have   gone
                being reviewed by a  hand  picked  panel  called  peers.  Let    through  a  magnificently  structured   and   logically   jus.
                us call this the B.R.A. - the  bunny  rabbit  approach.  It's    tifiable  process  with  bewildering   numerological   finesse
                title comes from  the  setting  in  which  Johnny  brought  a    to arrive at  the  end  point  -  a  judgment  or  opinion  of
                rabbit  to  kindergarten  for  drag  and   brag   (Show   and    what to do with it.
                Tell). Mary asked if it were a boy rabbit or a  girl  rabbit.            WHAT DIFFERENCE DOES IT MAKE?
                Johnny said,  "I  don't  know".  The  precocious  Mary  said,
                "Since this is a participatory democracy, let's vote."               It  makes  a  difference  only  if  the  person   in   the
                   Although  both  these  approaches  have   been   practised    position with the right to decide agrees.
                successfully (at least in the evaluation of  those  with  the        This  formalized  ritualistic  numerological  game  called
                right to decide), the invalidity is obvious: for  the  first,    evaluation, is a series of decisions of those with  the  right
                one only needs to have the  right  to  choose  the  qualities    to  decide  and  ultimately  rests  on  the  judgment  of  the
                and the measures and the weights  and  the  cut  off  points;    person who can determine the outcome by:
                and for the second, one merely  needs  the  right  to  choose        1. Choosing the doers of evaluation.
                the majority of the panel.                                           2. Choosing the elements for inclusion, and/or
                   Certainly  no   one   could   object   to   those   simple        3. having the  right  to  decide  what  comparison  means.
                requests if the "right to decide" is not  the  crucial  issue        Such   evaluative   manipulation   can   occur    whenever
                in  evaluation,  and  if  evaluation  does   not   ultimately    there  is  no  ultimate  criterion  which  assures  a   unique
                become  judgment,  i.e.,  the  opinion  of  the  person  with    ordering of  priorities,  and  the  resulting  correct  method
                the right to decide.                                             of choice.
                   Third:  The  third   commonly   used   approach   is   the
                R.C.A. - the report card  approach.  It  is  essentially  the      WHAT DOES IT MEAN To HEALTH PROGRAMS?
                approach  used  for  evaluation  of  the  quality   of   care        1. If one doesn't play the game,  or  even  worse  realize
                example at the beginning of the paper.                                  what  the  ground  rules  are,  one   may   lose   the,

                164




           funding  or  have  funding  reduced  since   it's   easy   to         By  analogy,  we   are   merely   counting   how   many   times
           relate  dollars   to   points   scored   on   a   hard   fact     the bird flaps his wings,  without  asking,  did  the  bird  fly  -
           evaluation index.                                                 let alone how far and how high.
       2.  It's  an  effective  way  for  funders  to  mold   or   shape         Clearly,   outcomes   as   the   measures   of    effectiveness
           the program, i.e., dictate the hea      Ith   care   delivery     must  be  the  starting  point  for  evaluation   before   any   of
           system.    They    need    only     specify     the     proxy  ,  the  measures  of   input   or   output   analysis   of   the   ef-
           indicators   or   elements,   their   weights    and    their     ficiency kind have any meaning or usefulness.
           measures,    and    attach    adequate     punishment     and
           rewards    so    that     grantees     desiring     continued          TWO AIDS To ASSIST IN EVALUATING
           support  will  allocate  the  resources   to   maximize   the                      HEALTH DELIVERY
           evaluation  index.  It's  +,he   health   care   version   of         In the  face  of  such   a   bewildering   maze   of   considera-
           "shape up or ship out".                                           tions,  two  simple  lists  of  elements  are  helpful  in  retain-
        3. Quite   clearly,   those    elements    that    are    easily     ing ones sanity: The first are the five A's.
           measurable  will  get  the  attention  and  be   assured   of         In   the   evaluation   of   any    health    care    delivery,
           inclusion  in   such   an   evaluation.   I   am   personally     questions   of   appropriateness,   availability,    accessibility,
           concerned  that  what  is  really  important   in   life   is     and   acceptability   to   both   seeker   and   server   must   be
           inversely related to what is easily measurable.                   answered.  These   are   a   dependent   sequence.   For   services
                    OTHER CONSIDERATIONS                                     can  be  deemed   appropriate   yet   be   unavailable.   Or   they
                                                                             can  be  defined  to  be  available,   yet   not   accessible.   Or
        Currently  in  vogue  also  is   efficiency   of   health   care     they  can  be  defined  to  be  appropriate,  and   available   and
      delivery.  One  of  the  chosen   qualities   is   efficiency   in     accessible, but still not  acceptable  to   either   or   both   the
      every  evaluation.  The  usual  operational  definition   is   one     serving staff  or  the  seeking  cli-ent.     However,     overriding
      lifted  bodily  from  engineering  -  the  ratio  of   output   to     these four  A's  is  the  one  called  accountability.  It  is  the
      input.                                                                 essence    of    the    moral    contractual     agreement     made
        The   hazard   of   this   measure   is   clearly   from   whose     between the seeker when he seeks and server when
      perspective   output   is    viewed.    From    the    doer,    Ms     serves.
      activities  are  always  viewed  as  output.  From  the  receiver,         The  second  list  is  the  generic  structure  of   evaluation
      those outputs are always viewed as inputs.                             implicit  in  this  paper   and   necessarily   a   part   of   the
        The  classical  data  which   allegedly   measures   output   of     process  of  evaluation.  There  riiust   be   six   interdependent
      laboratories,  groups  of  personnel,  institutions,   etc.   such     elements to any evaluation undertaking:
      as  visits,  lab  tests,  encounters,  and  the  rest  are  really         First: Context (what, where, when, and who).
      only  inputs  to  the  health  of  the  seeker  of  service.  Even         Second:     Content     (program     elements     being      or
      more  interestingly,   within   the   sequence   of   doers,   the     intended to be provided and why).
      prior doers, output is  also  viewed  as  an  input  to  the  next         Third:  Process  (how  care  is   organized   and   delivered).
      one  in  sequence.  Thus,  the  lab  technician  believes  he   is         Fourth:  Output  (how  many  times  did  the  bird   flap   its
      highly  productive  because  of   outputting   many   lab   tests,     wings).
      and  the  engineering   definition   gives   him   a   very   high         Fifth: Outcome (did the bird fly).
      efficiency  rating.  The  physician  or  nurse,  however,  lool-,s         Sixth:   Benefit   (how   high   and   far,   with   what   re-
      at the lab  tests  merely  as  inputs  and  they  in  turn,  value     sources).
      their visits  and  activities  as  the  real  outputs  upon  which         Clearly,   context,   content,   and   process    combine    in
      efficiency should be based.                                            many  ways  to  produce  the   output,   the   outcome,   and   the
         What  is  incredible  is  that  none  of  these   measures   of     benefits.
      efficiency  really  get  at  the  question  to   be   answered   -
      namely,  are  any  of  these  inputs  or  outputs   effective   in                               SUMMARY
      maintaining  or  altering  the  health  status  of  the  recipient
      of service.                                                                This  paper  was  intended  to  give  some   limited   perspec-
         Clearly,   effectiveness   must   first   be   defined   before     tive  into  the  what,  why,  and  how  of  evaluation.  It   high-
      efficiency  has  any  useful  meaning.  It  appears  that  we  are     lighted   the   reasons   for    misunderstanding    between    the
      producing  a  health  delivery  system  which  is  unit  by   unit     hard  fact  approaches  to  evaluation   and   the   day   to   day
      approaching     100%     efficiency      while      simultaneously     uses.  It  is  not  easy  to  describe  a  program  even  in  terms
      'narching toward the other extreme in effectiveness.                   of  telling  how  it  was.  For  ongoing  programs   it   is   even
                                                                                                                                             165





               more difficult to tell how it is. It is virtually  impossible      between  the  SCIENCE   (retrospective   description)   ail(
               to tell how it will  be,  or  as  some  glibly  say,  how  it      the ART (prescriptive action) of adn-dnistration.,Ho
               should be.                                                         fully,  program  evaluation  will  continue  to  serve   an(
                  It is  necessary  for  each  of  you  as  accountable  and      develop as the management tool  it  first  was,  and  stii@i
               responsible  health  devotees  to  DESCRIBE   how   it   was,      intended to be and will  not  become  the  program  tyranfii
               but   it   is   more   important   to   structure   ways   to      of the 1970's.
               PRESCRIBE  how  it  will  be.  This  may  be  the  difference




















































             166








                                EVALUATION OF CORONARY CARE TRAINING


                                                             Participants
   .Zodger M. Shepherd, M.D. - Moderator                            MarieHa Larter, M.S.
   Assistant Program Director, Continuing Education                 Coordinator, Washington/Alaska Coronary
   ,Presbyterian Medical Center                                         Care Unit Nurse Training Program

                                                                    Carol Larson
                                                                    Allied Health Specialist
   Daniele Deverin                                                  Continuing Education and Training Branch
   Cybern Education, Inc.                                           Regional Medical Programs Service



            Evaluafion of Coronary Care Training:                          technique has  a  disruptive  effect  and  reduces  the
                 Some Direct Observations of                               tendency  of  other   physicians   to   implement   the
               Performance in Hospital Practice                            advocated procedure at all.
                RODGER SHEPHERD, M. D.                                  2.   Availability   of   Equipment:   Standardized   tech-
                                                                           nique  depends  on  standard   materials.   Instructors
      The objective  of  our  Intensive  Care  Training  Program           have observed the lack of  certain  critical  materials
   is to enable physicians in cadres  from  small  general  hos-           or instruments  during  introduction  of  a  new  tech-
   pitals to perform certain intensive care skills in their  own           nique.   The   attendant   frustration   during    this
   hospital settings.  These  skills  include:  use  of  central           critical  phase  may  abort  or  seriously  retard  the
   venous   catheter,   use   of    intra-arterial    monitoring           adoption of the  new  practice  in  spite  of  adequate
   catheter,  interpretation  of  blood  gas  data,   continuous           ly trained personnel.
   EKG  monitoring,   airway   care,   controlled   ventilation,        3. Supporting  Services:  Interpretation  of   blood   gas
   cardloversion,  and  others.  The  staff  of  our   ICU   had           data  depends  on  complete  confidence  in  the  data.
   visited  small  hospitals  and  identified  these  skills  as           We  have  encountered  one   hospital   setting   where
   feasible but underused in smaller hospital ICU'S.                       the  student's  training  in  interpretation  of  blood
      The  training  program  is  conducted  in  three   phases.           gas   data   was   not   implemented   until   we   had
   During  the  first  phase,  the  cadre  and  project  clarify           rectified certain analytical problems in  the  clinical
   mutual objectives. During  a  second  phase,  each  physician           laboratory.
   from  the  cadre  undergoes  a  week4ong  program   of   one-
   to-one  instruction  at  a   metropolitan   medical   center.
   During a third phase,  an  instructor-in-residence  is  main-                 Report on Xerox Study of Eleven
   tained in the cadre's own  intensive  care  unit  around  the             National Coronary Care Training Centers
   clock for 10-12 days.
                                                                                       DANIELE DEVERIN
      The  direct  observations  of   these   instructors   have
   provided  valuable  anecdotal  data  on  both   the   project
   and the resulting student performance:                               In 196.7,  Xerox  Education  Division  was  contracted  by
      1. Standardization of Technique:       The   same   single    Public  Health  Service  to  conduct   a   2!year   evaluation
         standard technique for insertion    of  central  venous    study of  eleven  national  coronary  care  training  centers.
         catheter  is  advocated  during   each   individualized                           OBJECTIVES
         instruction.  The  mastery   of   this   technique   is
         certified  by  the  instructor.  However,  the  student        The study  was     designed  to  fulfill   the   following
         may  not  implement   this   technique   in   his   own    objectives:
         hospital  setting.  It  has  been  observed  that   the        1. To determine   the effectiveness of the  training  pro-
         failure of  some  physicians  to  support  standardized           grams  in  imparting  the  knowledge,  attitudes,   and

                                                                                                                               167





                        skills needed for a nurse to  perform  in  a  CCU  at  an     ments  constitute  the  basic  evaluative  data  bank  of   th(
                        acceptable level.                                             study.
                    2.  To determine  the  effectiveness  of  the  training  pro-        The  follow-up  portion  of  the  study  was  conducted   ir
                        grams  in  developing  a  high  quality  of   performance     two  ways:  mailed  questionnaires  were   sent   out   to   al
                        in the training graduates.                                    graduates  of  the  programs,  except  as  noted   below.   Jr.
                    3.  To  determine  the  most   effective   training   program     addition,   other   questionnaires,    including    the    per,
                        for achieving these aim@.                                     formance  checklist,  were  sent  to  their   hospital   super.
                    4.  To   determine   the   distinguishing    qualities    and     visors.   A   systematic   mail   and    telephone    procedure
                        characteristics of a successful CCU nurse.                    assured a return rate of at least  90%.  In  order  to  monitor
                    5.  To   determine   the   most   effective   and    reliable     the reliability of  the  mailed  returns,  and  to  assess  the
                        methods  for  the  selection  of  the   "best"   training     effect  of  non-respondent.  bias,  a  10%  random  sample   of
                        applicants.                                                   the graduates was  selected  for  personal,  on-site  follow-up
                                                                                      visits.  The  results  of  these  visits  confirmed  the   high
                                                                                      degree of reliability in the mailed returns.
                                      MET  HODOLOGY                                     Data-Collection.   The   survey    period    extended    from
                    A  systematic  model  was  designed  to  analyze  the   three     August  1968  to  September  1969.  In   the   eleven   centers
                interrelated primary spheres of concern:                              under study, a total of  57  sessions  were  monitored,  for  a
                    1.   Input    variables:    Trainees'    demographic    data,     total of 862 trainees.  The  456  sponsor  hospitals  were  all
                        education,  personality,   expectations   and   attitudes     included in the survey.
                        towards CCU nursing, etc.                                       Data-Processing.   Standardized   procedures   were    estab.
                    2.  Process  va?iables:  Training  Centers'  facilities,  ap-     lished  for  handling   and   coding   of   raw   data.   Data-
                        proach, curriculum, etc.                                      processing  was  completed  at  the  end  of  October  1969.  A
                    3.  Output  variables:.   Knowledge   gained,   post-training     correlation  matrix-was  designed  and  run  on  85  variables.
                        expectations   and   attitudes,   clinical    performance
                        both in-training and on-the-job, etc.                                                 FINDINGS
                    In   addition,   Environmental   variables   were    studied.       Trainees.  The  "typical"  trainee  was  female  (989o),  the
                They consist of the sponsor hospitals'          facilities,   ap-     mean  age  for  the  group  was  34  years,  and   the   median
                proach  to  nursing,  etc.  that   influence   both   input   and     was  28  years.  About  half  of  the  trainees  were,  or  had
                output.                                                               been married; of  these,  60%  had  families,  half  of  Nvhich
                    The project  staff  then  prepared,  piloted,   and   revised     consisted  of  2  or  more  children.  83%   had   obtained   a
                nine   data-gathering   instruments.   A   standard   personality     hospital diploma,  5%  had  an  associate  degree,  18%  had  a
                test, the  16  PF,  was  also  selected.  This  process  involved     baccalaureate   degree   and   2%,    a    masters.    Previous
                discussions  with  PHS  contract  officers   and   with   various     coronary   care   experience   was   as   follows:   17%    had
                consultants, visits to  CCU'S,  a  review  of  pertinent  litera-     worked  in  a  CCU  for  an  average  of  8  months,  and   36%
                ture  and  existing  research  information,   and   an   analysis     had  worked  in  ICU/CCU's  for  an  average  of   14   months.
                of the content to be covered.                                         Most  values  of  the  16  f.F.  were  close  to   the   normal
                    In general,  data  were  collected  on  the  trainees  before     mean, except on  the  general  intelligence  scale  when  their
                and  after  training,  and  at  follow-up,  between   three   and     mean  was  substantially  higher  than  the  mean  of  any  oc-
                four  and  a   half   mon'ths after training. Data   were    also     cupational sub-group reported.
                collected  on  the  training  programs,  and   on   the   sponsor      Sponsor   Hospitals.   Of   the   456   hospitals    surveyed,
                hospitals to which the trainees were          returning     after     55%  had  sent  one  nurse  to  training,  27%   had   sent   2
                completion of the program. In terms           of   the   specific     nurses, and the remainder  3  or  more  nurses.  Hospital  size
                problems addressed in this survey, two       instruments      are     varied  considerably:  16%  had  less  than   100   beds,   26%
                of special interest.                                                  between  100  and  199,  21%   between   200   and   299,   and
                    The   knowledge   test   was    especially    designed    and     the   remaining   37%   had   300   or   more   beds.   Results
                standardized.  The  test  contained   12   weight-ed   sub-tests,     obtained  before  training  showed  that  27%   of   the   hos-
                with   each   sub-test   containing   a   number   of    weighted     pitals   had   a   separate   CCU   and    41%    a    combined
                items.  It  was  used  both  before  and  after   training.   The     CCU/ICU.  These  figures  increased  slightly   at   follow-up.
                performance  checklist  was  designed  to  tap  the   degree   to     The  most  surprising   finding   of   the   survey   was   the
                which   the   training   graduates   performed    specific    CCU     number  of  training  graduates  still  not  working  in  coro-
                nursing  functions  at   follow-up.   Together,   these   instru-     nary  care  at  follow-up.  With   a   90%   response   (N=779)

                168






   only 5 1 0 nurses were found to  work  in  coronary  care,  or    would seei-n, then,  that  the  sponsor  hospitals  view  the
   65%  of  the  follow-up  population.  The  reason   given   in    major  function  of  traiiiin-   as   developing   tec   nica
   65% of the cases was absence of CCU.                              competence, while general nursing  qualities  are  viewed  as
      Trainees'  Preparedizess   upon   CCU   entry.   Evaluation    bihereiit in the potential trainee. This  will  be  discussed
   of  objective  one  was  done  on  the  basis   of   knowledge    later tinder objective four.
   tests, trainees' attitudes, and supervisors' general  ratings.       A fair comparison  between  nurses  trained  at  different
      Out of a possible  220  points,  the  862  nurses  averaged    centers require  that  some  allowance  be  made  for  skills
   a pre-score of 127.4 points, or 57.9,7o,  wlgch  increased  to    the trainees brought  with  them  on  entering  the  program.
   74% on the post-test. The variability of test  scores  at  pre    In all cases, it was found that the centers rated highest  in
   and  post  for  the  group  indicates  that  instruction   had    nursing  performance  had  trained   the   most   experienced
   brought about a leveling of test performance.                     population,  while  the  lowest  ratings  were  obtained   by
      Trainees'   expectations   of   CCU   activities    changed    those centers having trained  the  least  experienced  group.
   markedly  between  pre  and  post-training,  with  a   general       Model  program.  The  study   failed   in   providing   an
   shift  towards  a  "middle-of-the-road"  attitude,  indicating    analysis  of  the  model  program,  objective  three  of  the
   a  tendency,  over  the  training  cycle,  for  attitudes   to    study.  Both  dependent  and   independent   variables   dis-
   become more realistic, which is felt  to  be  a  positive  and    played  inadequate  variance  characteristics.  Further,  the
   desirable training output.  63%  of  the  trainees  found  the    training  centers  were  quite  similar,  at  least  on   the
   program  excellent  and  35%  rated   it   as   "good".   When    variables tapped by the  instruments.  This  result  was,  of
   asked,  after  training,  whether  they   would   select   CCU    course, disappointing,  but  it  should  be  noted  that  the
   training  again,  63%  answered  "definitely  yes",  and   25%    basic reason for this failure is  the  success  of  the  pro-
   "probably".                                                       grams in fulfilling the overall objectives.
      The  following   curriculum   areas   were   mentioned   at       Optimal  characteristics  of  a  CCU  nurse.  The   fourth
   post-training  as  needing   more   preparation:   Fluid   and    objective was examined  from  the  standpoint  of  high  per-
   Electrolyte   Balance   (53%),    Interpretation    of    ECGs    formance,  satisfaction  with  CCU  nursing,  and  motivation
   (49,7o),   Recognition   and    Treatment    of    Arrhythmias    to continue work in coronary care.
   (35,7o).  While  these  figures  may  have   represented   the
   trainees' anxiety at assuming  new  responsibilities,  similar       Since  performance  ratings  were  typically  either  good
   results  were  obtained  among  nurses  working  in  CCU's  at    or excellent,  a  detailed  study  was  made  of  those  per-
   follow-up. Fluid and Electrolyte  Balance  was  still  on  the    formance items  rated  as  "deficient"  by  the  supervisors,
   top  of  the  list,  non-coronary  complications   were   men-    yielding a picture of what  a  successful  CCU  nurse  should
   tioned  by  38%  of  the  working   graduates,   basic   elec-    not  be,  and  inversely  what  characteristics  she   should
   tronics and interpretation of ECG by 37%.                         possess.  The  largest  number  of  deficient  ratings   were
      Suggestions   by   supervisors   regarding   possible   im-    found  in  the  broad  area  of  "Communication  and   Inter-
   provements  in  the  programs  agree  in  general  with  those    action  with  Staff";  next  in  line  was  'Terformance   of
   of the trainees,  the  main  one  being  for  more  stress  on    day-to-day  assignments"  with  stress  on  general   nursing
   the technical aspects  of  CCU  nursing.  Rating  the  nurses'    competence, and skill in handling  and  verifying  the  tech-
   preparation on  a  seven-point  scale,  77.5%  of  the  super-    nical   equipment;   finally    "Communication    with    the
   visors selected the two top categories.                           patient and his relatives".  Thus,  a  successful  CCU  nurse
      Trainees'Perfonnance   in    CCUS.    Performance    check-    would appear to need excellent  nursing  skills,  an  ability
                                                                     to relate well  with  the  members  of  the  CCU  team,  with
   fists were received by each hospital, one  for  each  trainee.
   A total of 487 checklists were completed and ret      .   urned   the  patient  and  his  relatives,  as  well   as   technical
   (56% of those mailed).  By  far  the  main  reason  given  for    competence.
   non-completion of the form was "CCU not open".                       73%  of  the   training   graduates   working   in   CCU's
      Overall  mean  performance  was  rated   from   "good"   to    stated  that  they  definitely  wanted  to  pursue   coronary
   "excellent";  however,  while  the   mean   ratings   do   not    care  as  a  specialty.  A  number  of  problems   were   ex-
   deviate  siiznificantly  from  one  another,   there   was   a    pressed,  however,  the  great  majority  stressing  staffing
   general tendency for nurses  to  be  rated  higher  for  tech-    difficulties,  and  lack   of   support   and   communication
   nical,  CCU-specific  activities   than   for   non-technical,    within the hospital in general, and the Unit  in  particular.
   jeneral nursing  activities.  There  is  an  apparent  contr.a-   A smaller number of  nurses  also  expressed  frustration  at
           between   these   high   "technical"   ratings,    and    the  occasional   "dullness"   of   Unit   work.   Successful
          ions for more program  depth  in  the  same  areas.  It    trainees derived great  satisfaction  from  bedside  nursing,

                                                                                                                               169




                      and   from   the   challenge   and    diversity    offered    by      relating to a set of regional objectives  A      pool  of   ol
                      coronary care work.                                                   800 test  items  were  edited  for  content  and  format;  th(
                       Since  the  follow-up  period  extended  from  3   to   4   1/2      revised items.  were  then  rated  by  their  authors  as  bel
                      months,   long-range   tenure   could   not   be    ascertained.      either    "essential",    "desirable",    or     "supplementar
                      When  asked  about  their  plans  for  a  two-year  period,  61%      knowledge  for  a  CCU   nurse.   Only   those   items   rated
                      of  the   nurses   stated   that   they   wanted   to   continue      "essential"  or  "Mghly  desirable"  were  retained,   leavini
                      coronary care nursing.                                                pool  of  2SO  questions  weighted  in  the  following   fasMo
                       Selection   criteria.   The   main   criteria   used   by   the
                      sponsor  hospitals  when  selecting  potential  trainees   were:      CCU Concepts                1)
                      motivation  and  interest   in   CCU   nursing;   stability   in      Anatomy and Physiology      3)  Summarized   as   CCU   Concep
                      present  position,  and  demonstrated  excellence   in   general      The Classic MI              8)
                                                                                            Diagnostic Tests            5)
                      nursing skills.                                                       Rehabilitation              8)
                      The  results  of  the  study   point   to   the   necessity   of
                      providing the potential  trainee  with  a  clear  perception  of      Complications of an MI      13)
                      what  her  role  will  be,  prior  to   selection.   They   also      (excluding       arrhythmias)   Summarized   as    Complicatio ri
                      suggest  a  need  for  closer   communication   between   train-
                                                                                            Electrocardiography         8)
                      ing  centers  and  sponsor  hospitals,   before,   during,   and      Equipment and Safety        3)  Summarized   as    Arrhythmias
                      after training.                                                       Arrhythmias                 19)

                                                                                            Chemical Therapy            13) Swnmarized as Chemical
                             Evaluation of CCU Nurse Education in                                                             Therapy
                                      Washington and Alaska                                 Other Therapy (i.e.
                                      MARIELLA LARTER                                       pacing,  resuscitation)   19)   Summarized  as  Other  Therap!

                      In   July   1969,   the   Subregional   CCU   Nurse    Education      Evaluation of CCUNurse Education
                      Project  of  W/ARMP  became  operational.  The   goal   of   the      Test  items  were  randomly  assigned  to  version  A   or   I
                      project  was  to  train  873  nurses  per  year  in  basic  CCU,      of  the  exam.  Each  exam@  is  equally  weighted  by  conten
                      and  to  train  them  in  sixteen   (16)   subregional   centers      but the individual questions  remain  different.  After  fiel(
                      rather than in a "core"  or  "Seattle  based"  setting.  In  the      testing  on  student  nurses  with  no  CCU   background   an(
                      last year, all  but  one  of  the  sixteen  centers  has  become      on  graduates  of  a  USPHS  five-week  CCU   nursing   course
                      jational   and   an   additional    three    communities    have      fourteen  items  from  each  version  were  eliminated.  Aftel
                      become  subregional  education  centers.   Each   center   plans      item analysis of the test results of 200  nurses  involved  ir
                      its  own  objectives,  curriculum,   elegibility   requirements,      subregional   courses   was   completed,   eleven   additional
                      course length, and teaching methodology.                              items were deleted from each version.
                      The   plan   as   outlined   in   the   following   pages    was      Test  A  and  Test  B  now  contain  100   items   each,   and
                      developed   by   the   Subregional   Project   staff   in   con-      are of equal  difficulty  according  to  standard  statistical
                      junction  with  the  Office  of  Research  in   Medical   Educa-      measures.  In  addition  to  answering  each   question   with
                      tion  of  the  University   of   Washington,   Charles   Dohner,      what she supposes to be the  correct  answer,  each  nurse  is
                      Ph.D., director.  From  its  inception  the  evaluation  was  to      asked to rate her certainly  about  that  answer  on  a  scale
                      meet two goals:  1)  to  evaluate  the  impact  of  the  project      from one to three, or  absolute  certainty  to  guessing.  The
                      on  regional  CCU  nurse  training;  and  2)  to  provide  feed-      computer   summary   of   her   scores   then   computes   not
                      back   to   course   instructors   on    the    strengths    and      ordy how  many  questions  she  answers  correctly,  but  also
                      weaknesses  of  their  courses  and  of  individuals  in   them.      how  many  questions  she   was   certain   about,   and   how
                      The  evaluation  design  at   present   involves   measures   of      many  which  she  says  she  was  certain   about   that   she
                      knowledge,  attitude  and  skill.  A  patient  care   assessment      actually  answers  correctly.
                      too] is presently under development.                                  We  can  thus  measure  with  our  instruments   three   areas
                                  KNOWLEDGE TESTING                                         of potential  change  from  pre  to  post  course:  1)  change
                                                                                            in knowledge  (right-wrong  score);  2)  change  in  expressed
                      Practicing   physicians   and   nurses   from   throughout   the      certitude  and  guessing;  and  3)  change   in   ability   to
                      region  were  asked  to   submit   multiple   choice   questions      evaluate  her  knowledge  about  CCU  nursing.   These   three

                      170






        factors are recorded for the overall exam as well as  for                             SKILL TESTING
        five content areas within the exam for each  student  and            A skill test was designed to evaluate  the  functioning
        each class.                                                      of  nurses when presented  with  simulations  of  clinical
           The instructor  can  go  over  with  each  student  her       emergencies.  The  testing  involves  an   evaluation   of
        areas of greatest  knowledge  or  growth  in  knowledge,  her    psychomotor abilities as well as  the  rationale  for  initia-
        expressed  confidence,  and  whether   that   confidence   is    tion of  certain  therapeutic  measures.  The  skill  test  is
        well founded or not, She  has  considerable  data  about  the    designed to be administered  in  a  mock-up  setting  using  a
        continuing  education  needs   of   her   students   at   the    standard   hospital   bedside   area,   an   arrhythmia   anne
        completion of her course as well.                                resuscitation doll, a bedside monitor,  a  defibrillator,  and
           The regional mean for  the  pre  test  (Test  A)  is  32%.    standard   emergency   equipment   (i.e.   suction,    medica-
                                                                         tions).
        The regional mean for the post test  (Test  B)  is  53%.  The
        reliability  coefficients  of  these   exams,   when   admin-        In initial field testing, nurses suggested the  following:
        istered to a group of ten or  more  individuals,  range  from        1. that they be in uniform when tested, and
        0.89 to 0.95.                                                        2. that the evaluator "role play" as  a  new  orientee  to
                                                                               a coronary care unit,  rather  than  assume  a  strictly
                          ATTITUDE TESTING                                     observational andjudgemental pose.
           A  standard  semantic  differential  scale  is   used   to        Taking these  suggestions,  a  group  of  28  nurses  from
        evaluate the attitudes of  nurses  on  ten  concepts  in  CCU    both  metropolitan  and   rural   hospitals   were   evaluated
        nursing.  Those  concepts  include:  coronary  care  nursing;    using this tool in their own clinical setting.  The  range  of
        coronary   heart   disease;   cardiac   monitoring;   cardio-    scores was  8  to  30,  out  of  a  possible  32  points.  The
        pulmonary    resuscitation;    doctor-nurse    relationships;    mean score was 22 pohits.
        change in nursing;  independent  nursing  decisions;  patient        The  evaluator  summarized   her   conclusions   regarding
        individuality;   patient   teaching;   and   emergency   sit-    initial use of the tool as follows:
        uations.  One  other   concept,   "death",   was   eliminated
        because of  possible  disagreement  about  what  a  desirable        1, Greater  consideration  needs  to  be   given   to   the
        change  in  attitude  would  be  at  the  completion   of   a          standing orders under  which  a  nurse  functions  in  a
        course.                                                                given   agency;   accepted   therapy   for   nurses   to
           The  attitude  scales  are  filled  out  on  an  anonymous          initiate varies greatly from agency to agency.
        basis at pre and post  tests;  results  are  scored  by  sub-        2. No  more  than  one  agency  can  be  evaluated   on   a
        region. Analysis of results in the  first  formal  evaluation          given day in  view  of  unit  pressures,  staffing,  and
        period  (September  1,  1969  -   February   1,   1970),   in          patient  census;  the  cost  of  sending  an   evaluator
        which  time  over  500  nurses  participated  in  Subregional          any distance is considerable  unless  other  duties  can
        courses, revealed that overall there was  not  a  significant          be performed concurrently.
        change in  attitude  from  pre  to  post  course.  Individual        3. It  is  very  difficult  to  remain  neutral  even  when
        courses  showed  significant  attitude  changes  in  one   or          involved  in  role  playing;   there   is   a   constant
        more "concept" areas; in  discussing  some  of  the  puzzling          temptation  to  correct  errors  and  teach  during  the
        attitude shifts  with  course  instructors,  it  was  learned          testing.
        that the majority  of  these  shifts  reflected  an  attitude        4. The  nurses  tested  need  to  be  thoroughly   familiar
        consciously conveyed  to  the  students  by  the  instructor.          with  all  equipment  used  in  the  testing  situation;
                                                                               thus,  hospital  equipment  or  a  like  brand  must  be
                                                                               brought to each testing site.
        Evaluation of CCUNurse Education                                     5. Many  nurses  responded  appropriately   to   situations
           Future planning calls  for  use  of  attitude  measure  at          but for the wrong reasons.
        pre  course  and  at  six  months  after  course  completion.        6. A weigliting  scale  needs  to  be  further  refined  so
        Knowledge  and  attitude  scores  have  thus  far  showed  no          that there is  a  greater  spread  in  scores  and  dif-
        correlation; if this  lack  of  either  significant  attitude          ferentiation between levels of performance.
        change or  correlation  between  attitude  score  and  knowl-        7. The  skill  test  is  an  excellent  teaching  tool  but
        edge continues, a new strategy  to  measure  this  area  will          needs further revisions to  increase  its  effectiveness
        be devised.                                                            as an evaluation instrument.


                                                                                                                                    171





                Evaluation of CCU Nurse Education                                        strate changes in patient care as a  result  of  postgraduate
                     Future plans call  for  the  random  skill  testing  of             learning experiences for nurses.
                nurses at the completion of basic courses utilizing  equip-
                                                                                         Additional Matetials Available on Reqtiest:
                ment they have used in mock-up drill sessions.
                                                                                            I . Objectives  for  nurse  training  upon   which   knowl-
                             OTHER EVALUATION TOOLS                                             edge tests are based.
                                                                                            2.  Sample  computer  printout  and  explanation  of   data
                     A. A personal profile  sheet  revealing  17  pieces  of                    contained in it.
                demographic information about  each  nurse  is  filled  out                 3.  Copies  of  Test  A  and  Test  B  for  review.  (Copy-
                at the completion of courses. Correlations  are  being  run                     righted material - not to  be  retained  or  duplicated
                between these 17 variables and  combinations  of  variables                     in any fasl-don)
                compared with pre test score, post  test  score,  expressed                 4.  Answer sheet incorporating certitude measure.
                certitudes, accuracies, and changes in score. Data will  be                 5 . Attitude test.
                available soon.                                                             6.  Sample of  results  of  attitude  testing  returned  to
                                                                                                course instructor.
                     B. Use of  chi  square  measure  in   conjunction   with               7.  Skill test.
                certitude  score  has  been  employed  by  the  School   of                 8.  Personal profile sheet.
                Medicine. A high  and  significant  correlation  was  found
                between a low chi  square  and  the  overall  knowledge  of
                                                                             C,                 fvrite.-   MARIELLA 1ARTER, R.N.
                the   students   tested.   This   measure   is   being   incorporated                      Subregional CCU Project W/ARMP
                into the CCU nursing data analysis.                                                        180 "U" District Building
                     C.  A  patient  care   assessment   too]   is   under   develop-                      1 107 N. E. 45th Street
                ment.  It  is  hoped  that  this  tool  can   be   used   to   demon-                      Seattle, Washington 98105
                                                 CORONARY         CARE         NURSING          EXAM...CONTENT          AREAS
                                                             RELATIVE WEIGHTING, OBJECTIVES
                CCU Concepts .... relative weight I                                         b:  Evaluates  the  techniques  used  in  the  physical  and   psy-
                     a:  Synthesizes  a  concept  of  intensive  coronary   care   in           chological preparation of the  patient  for  diagnostic  tests.
                       relation to its implications for the professional nurse              (content  areas  include:  history  and  physical;  serum  enzymes,
                     b: Values the necessity for assuming  responsibility  and  self-       ESR,  NVBC,  temperature   elevation;   circulation   time;   chest
                       direction for continued learning in CCU nursing.                     X-ray; serial EKG'S; heart and breath  sounds;  vital  signs,  CVP,
                Anatomy and Physiology .... relative weight III                             jugular   veins, urine  sp.  gravity,  1&0;   nursing   care   plans
                     a:   Comprehends   basic   anatomy   and   physiology   of   the       related  to  scheduling  of  tests;  teaching  plans  to   minimize
                       cardiovascular system                                                fear,  discomfort,   emergencies)
                     b:  interprets  significant   interrelationships   between   the    Complications   of   an   Acute   Myocardial   Infaiction   (excluding
                       cardiovascular, pubnonary, renal, and nervous systems             arrhythnijas) relative  weight  XIII
                     c: Interprets significant concepts of stress.
                Uncomplicated  Acute  Myocardial  Infarction.   .   relative   weight       a:  Applies  the  problem  solving  method  to  the  identification
                Vill                                                                            and  treatment  of  the   complications   of   coronary   heart
                     a: Synthesizes  a  concept  of   coronary   artery   disease   in          disease:
                       relation to its implications for  professional  nursing  care.           I . congestive heart  failure
                     b: Develops a  systematic  approach  lo  the  assessment  of  the          2.  cardiogeriie shock
                       individual  patient's  status  upon  admission  and  in   sub-           3.  acute pulmonary edema
                       sequent days of hospitalization.
                     (content  areas  include:   epidemiology;   pathophysiology   of           4.  pulmonary-systemic emboli
                     coronary  heart  disease;  physiologic  stress  responses;  psy-           5.  pericarditis
                     chology of rife-threatening diseases; history and classic  signs           6.  cardiac rupture
                     and symptoms of acute MI; cardiac  ischemia  as  it  relates  to           7.  cardiac arrest
                     relief of pain, anxiety, and administration of  oxygen;  dietary           8.  extreme emotional reactions
                     modifications, activity  restriction,  fluid  balance;  planning    Electrocardiography .... relative weight VIII
                     individualized care)                                                   a:  Synthesizes  basis   principles   of   electrocardiography   to
                Diagnostic Tests .... relative weight V                                         serve as a basis for the evaluation of cardiac  status  of  the
                     a: Analyzes the major diagnostic tools  used  in  the  diagnosis           individual patient
                       of coronary heart  disease  in  terms  of  their  implications       (content   areas   include:   electrophysiology;   hemodynamic   vs
                       for planning nursing care.                                           electrical   properties   of   the   heart;   depolarization    and

                172





       repolrization  of  the  myoca-rdium;  correlation  of  the  electro-        d:  Develops  a  systematic  approach  to  the  identification   and
       physiology of  the  heart  with  the  electrocardiographic  tracing;            treatment of cardiac emergencies.
       basic  principles  of  polarity,  amplitude,  and  configuration  of        e:  Differentiates  the   nurses'   responsibilities   in   elective
       the PQRST in terms of lead axis and cardiac vector)                             cardioversion  and  the  preventive  use   of   pacemakers,   as
     :quipment and Safety .... relative %vei.-ht III                                   opposed  to  the  emergency  situations  involved   with   these
       a:   Applies   fundamental   principles   of    electrocardiographic            therapies.
           techniques  to  achieve  maximurh   effectiveness   and   safety        f:  Utilizes   the   problem    solving    method    to    determine
           of  electrocardiographic  monitoring  and  twelve  lead   equip-            priorities in nursing care  in  the  post-resuscitative  period.
           ment.                                                                Rehabilitation. . . relative weight VIII
       (content   areas   include:   grounding;   monitoring   capabilities        a:  Develops  and  communicates  a  nursing  care  plan   that   in-
       and  limitations  as  opposed  to  the   standard   EKG;   essential            corporates   preventative,   therapeutic,   and   rehabilitative
       features;   purposes   and   standards    of    electrocardiographic            aspects.                                                     I
       equipment;  interference  and  means  of  eliminating  it;  how   to        b:  Evaluates the  patient's  CCU  experience  in  relation  to  his
       use  standard   monitoring   equipment;   safety   for   staff   and            total fife situation.
       patients with monitoring equipment)
                                                                                   c:  Determines  implications  for   the   planning   of   comprehen-
     ,rrhythmias ... relative weight XIX                                               sive nursing care.
       a:  Applies  the  problem  solving  method  to  the   identification        d:  Values  the  role  of  the  professional  nurse  in  the  health
           and  treatment  of   the   complications   of   coronary   heart            team, especially in  relation  to  her  potential  contributions
           disease, specifically cardiac arrhythmias.                                  regarding  the  individual  needs  of  the  patient  and  family
       b:  Evaluates  alterations  in   the   electrocardiographic   rhythm            and continuity of care  into  the  post  hospitalization  phase.
           strips  and  rhythms  displayed  on  the  oscilloscope   accord-        e:  Reviews  select  basic  nursing  knowledge  and  skills  in  the
           ing to their  significance  to  the  patient's  total  condition            light of  their  implications  for  the  patient  with  coronary
           and  th@  implications  for  medical  and   nursing   therapies.            heart disease.
       c:  Develops  a  systematic  approach  to  the   interpretation   of
           arrhythmias.                                                            (content   examples-vital   sips,   pulses,   tracheal   auctioning,
       d:  Utilizes  the  problem  solving  method  in  the  treatment   of        oxygen    administration,    respirators,    patient    positioning,
           arrhythmias.                                                            venipuncture,    IV    therapy    and    administration,    rotating
       (content areas include:  arrbythmias  by  site  of  origin,  effect,        tourniquet, skin care, passive exercising)
       treatment, and implications for nursing care)
     hemical Therapy .... relative weight XIII                                 Summary  of  Content   Areas   and   Relative   Weighting   onboth   pre
       a:  Develops  a   systematic   approach   to   the   classification,    and post Tests:
           analysis  of,  rationale  for,  and  the  nursing   implications
           involved  with  chemical   therapies   in   the   treatment   of                   CCU     Concepts:                  1/100
           coronary  heart   disease   and   the   frequently   encountered                   A - P:                             3/100
           complications.                                                                     Classic MI:                        8/100
     ther Therapy .... relative weight XIX                                                    Diagnostic Tests                   5/100
       a:  Appreciates  the  nurses  role  in  the  early  recognition  and                   Complications:                   13/100
           treatment  of  conditions  that  may  precede  life  threatening                   Electrocardiography                8/100
           conditions.                                                                        Equipment & Safety                 3/100
       b:  Appreciates  the  importance  of  effective  habit  patterns  in                   Arrhythmias:                     19/100
           the handling of emergency situations.                                              Chemical Therapy                 13/tOO
       c:  Appreciates   the   importance   of    frequent    review    and                   Other    Therapy:                19/100
           continued refinement of emergency procedures.                                      Rehabilitation:                    8/100


















                                                                                                                                            173




                           A SYSTEMS APPROACH TO CORONARY CARE EVALUATION


                                                                Participants
      Morton Robins - Moderator                                           M. A. Rockwell, Ni. D.
      Acting Chief, Study Design and Analysis                             Director, Rand Health Program
         Staff                                                            Rand Corporation
      Regional Medical Programs Service



           A Study of Coronary Care Unit Effectiveness                       The  CCRNIP,  aware  both  of   the   importance   of   col-
                          M.A. ROCKWELL                                   lecting  performance  data  in  CCUs   and   the   difficulties
                                                                          experienced  by   many   units   in   collecting   such   data,
                                                                          embarked   upon   development   of    a    standardized    data
         This  report  describes  a   continuing   project   conducted    collection  and  reporting  system  for   CCUS.   In   December
      by  The  Rand  Corporation   for   the   California   Committees    1969,  a  contract  was  given  to  The  Rand  Corporation   to
      on  Regional   Medical   Programs   (CCRMP)   to   measure   the    develop  a  prototype  system   and   test   its   feasibility.
      operational  effectiveness  of  coronary  care   units.   During    Medical  guidance  of  the  project   was   provided   by   the
      the past  two  years  the  project,  which  began  as  a  feast-    CCU Steering Committee of the CCRMP.
      bflity  study,   has   become   a   community   action   project       During  the  past  two  years,   a   prototype   data   col-
      involving more than I  00  hospitals.  This  report  traces  the    lection  form  has  been  designed,  tested  and  revised.   On
      evolution  of  the  study  from  its  initiation   up   to   the    January  1,  1970,   a   prototype   data   collection   system
      present,  describes  what  has  been  accomplished,   and   out-    became  operational  and  participation  in   the   study   was
      lines future objectives.                                            opened  to  any  California  CCU  that   wished   to   partici-
         Our  study  is  based  on   the   belief   that   every   CCU    pate.
      should  continually  monitor  its   performance.   Data   should       The  current  system  requires  that  about  100  items   of
      be  collected  describing  patients  admitted   to   the   unit,    information   be   reported   on    each    acute    myocardial
      how  rapidly  they  reached  the  CCU  following   their   onset    infarction  patient  admitted  to  the  CCU  (only   10   items
      of  symptoms,  their  clinical  -course  and  treatment   during    of  information  are  collected  on   non-Ml   patients).   The
      their  CCU  stay,  and  their  clinical  course  and   treatment    data  forms  are  mailed  to   The   Rand   Corporation   where
      during  their  CCU  stay,  and  their  discharge  status.   Col-    they   are   keypunched.   Every   three   months   the    key-
      lection and  analysis  of  such  data  is  necessary  to  ensure    punched   data   are   processed   by   computer   to   produce
      that the unit is performing effectively.                            summary  reports.  Each  hospital  receives  a  15-page  report
         In   1968,   the   CCRMP   found   that   most   CCUs    were    describing the patients admitted  to  the  unit  and  the  out-
      trying to  collect  and  analyze  such  data  but  many  of  the    come of  their  hospitalization.  Each  unit  can  compare  its
      units  were  having   problems   in   their   data   collection.    experience  with  that  of  the  participating   group   as   a
      First,   development   of   the   necessary   data    collection    whole.
      forms  and  procedures  proved   to   be   too   difficult   for       Preliminary  indications  are  that  the   data   collection
      many   units.  Second,  many  CCUs   soon   collected   such   a
      large  volume  of  data  that  it  could  not  be  analyzed   by    system   has   become   an   important   part   of   the    CCU
      manual    techniques    but    required    computer     methods.    operation  in  many  hospitals.   Although   participation   in
      Most  units  did  not  have  access  to  the  necessary   equip-    the study  is  voluntary,  the  number  of  participating  hos-
      ment  and  expertise.  Third,  once  the  data   was   collected    pitals  reached  120  by  June  1970.  Thus,  about  two-thirds
      and analyzed,  it  was  often  difficult  to  interpret  because    of  California's  CCUs  are  now  involved  in  the  study.  In
      there  was  no   standard   against   which   to   compare   the    addition,   units   from   the   WasWngton-Alaska   RMP,    the
      results.  It   seemed   desirable   to   allow   each   CCU   to    Northern  New  England  RMP  and  Missouri  either   have,   or
      compare its  results  with  those  of  similar  hospitals,  Such    are soon expected to join the study.
      comparisons,   however,    required    data    collection    and        We  believe  that  the  study  has  had  an  important  and
      analysis  procedures  to  be  standardized,  a  task   obviously    beneficial  effect  on  CCU  effectiveness.   First,   it   has
      beyond the capability of an individual CCU.                         helped  some   CCU   directors   improve   the   operation   of
                                                                                                                                      175



              their units by, for example, finding ways of  speeding  the    ways of  reducing  the  cost  of  CCU  care  without  com.
              patient  admissions.  Second,  periodic   summary   reports    promising its quality. These include: (1) using  specially
              have served as a focus for  teaching  conferences  for  CCU    trained  CCU  technicians  to  supplement  nurses  in  the
              physicians and nurses. Third, data collected  by  the  sys-    units, and (2) using  automated  monitoring  equipment  to
              tem have helped  the  CCRMP  assess  the  effectiveness  of    eliminate the requirement for continuous  surveillance  of
              their nurse training program.  Fourth,  data  collected  by    ECG monitors.
              the system should make it possible to  investigate  several





































                   176







                          EVALUATION OF INSTRTJCTIONAL TECHNOLOGY PROJECTS


                                                                 Participants
     James E. Dyson, Ph.D. - Moderator                                      James Barrett, Ph.D.
     Director, Continuing Education'Division                                Continuing Education Division
     Colorado-Wyoniing Regional Medical Program                             Colorado-NVyoming Regional Medical Program
                                                                            M. Gene Aldridge
                                                                            Continuing Education Division
     Cecilia C. Conrath                                                     Colorado-Wyorfflng Regional Medical Program
     Chief, Continuing Education and
       Training Branch                                                      William Engbretson, Ph.D.
     Regional Medical Programs Service                                      President, Govemor's State University



                          Swnmary of Session                                   While  Dr.   Barrett   interviewed   participants   a   written
                        CECELIA CONRATH                                     recording  of  the  answers   was   projected   on   an   overhead
                                                                            projector   by   Gene   Aldridge    also    of    the    Colorado/
         The    workshop    session    on    Instructional    Technology    Wyoming   RMP   staff.   This   enabled   a   running    inventory
     was    developed    by    the     Colorado/Wyo@ng     RMIP.     The    to  be  kept  in  front  of  the  participants  as   the   session
     objectives  of  the  session  are   given   below   in   order   of    progressed.
     priority.                                                                 At   the   conclusion   of   the   problem   census   a    long
         1. To  learn  interests  and   needs   of   workshop   partici-    distance  telephone  conference  was   held   with   the   follow-
            pants   for   help   in   evaluation   using   instructional    ing Consultants:
            technology.                                                             William  J.  Paisley,  Ph.D.
         2. To   help   participants   learn   functions   of    various            Director, ERIC Clearinghouse on
            types   of   instructional   technology,    approaches    to            Educational Media and Technology
                                                                                    at the Inst. for Comm. Research
            evaluation  of  such   technology   and   relative   effects            Stanford University
            of various approaches.                                                  Stanford, California 94305
         3. To   present    information    on    effective    evaluation
            procedures.                                                             E@abeth Norman, Ph.D.
         4. To      develop an   awareness    of    consultation/referral           Associate Professor of Nursing
            resources  nearby   within   region   and   on   an   inter-            College of Nursing
            regional basis.                                                         Northeastern      University
         The   whole   idea   was   to   show    how    questions    and            Boston, Massachusetts 02115
     concerns  can  be  quickly  identified,  how   resources   can   be            Rick   Breitenfeld,    Ph.D.
     located   and   used   effectively,   and   to   demonstrate   that            Executive Director
     the  basic  strategy  of  evaluation  grows  out   of   the   needs
     of the participants.                                                           Maryland Center for Public Broadcasting
         The  session   opened   with   a   brief   statement   of   the            Owings Mills, Maryland 21117
     statu s   of   instructional   technology   within   RMPS   by   the           Gerald W. Gaston, t).D.S.
     Chief   of   the   Continuing   Education   and   Training   Branch            OSRMP-CAI Project Supervisor
     followed  by   an   outline   of   the   session   by   Dr.   James            The Ohio State University
     Dyson,    Associate    Director    of     Continuing     Education,            Columbus,     Ohio     43210
     Colorado/Wyoming    Regional    Medical    Program.     A     prob-
     lem  census  of   interests   and   needs   of   participants   was            David L. Bell
     conducted   by    Dr.    James    Barrett    of    the    Colorado/            Box 488
                                                                                    Altadena, California 91001
     Wyoniing staff.

                                                                                                                                           177





                  The results were  not  entirely  successful  because  of    lems, i.e. temporarily losing California participants a
               the small attendance at  the  session.  This  technique  is    poor voice transmission interfered with the reception.
               productive  with  a  minimum  of  10  and  upward   in   an
               almost  unlimited  number.  There  were  only  5   partici-       Gene Aldridge assembled kits of material on eva
               pants and two left early.                                      uation of instructional technology and learning theory
                  Issues concerned with cost  effectiveness  of  different    Bibliographies on the general field of learning, teaching
               media,  adaptability  and  conversion  from  one   modality    with films, guides for TV teachers and considerations for
               to  another,  and  status  of  evaluation   research   were    judging audiovisual presentation standards were among
               brought up during  the  conference  call.  Technical  prob-    materials distributed.






































                178








                                       EVALUATION OF PHYSICIAN EDUCATION


                                                             Participants
      Daniel Fleisher, Ni.D. - Moderator                             William B. Munier, M.D.
      Director of Health Professions                                 Staff Assistant, Continuing Education
      Temple University                                                 and Training Branch
                                                                     Regional Medical Programs Service



                        Summary of Session                           the  projects  was  very  well  constructed  and  contributed
                       WILLIAM B. MUNIER                             the majority of  desirable  aspects.  Following  critique  of
                                                                     all three, a fairly  complete  list  had  been  developed  of
                                                                     what  constituted  an  effective   project.   It   had   been
         The  objective  of  the  Special  Interest  session   on    developed   by   the   participants   themselves    following
      Evaluation of  Physician  Education  was  to  increase  the    careful analysis of three projects representative  of  actual
      knowledge of  the  participants  about  the  essentials  of    RMP grant requests.
      sound  educational  projects.  The   methodology   employed       It was felt that the  active  involvement  of  the  people
      was  that  of  active  involvement   of   participants   in    attending the session  was  more  likely  to  increase  their
      deciding on what constituted a sound  project.  No  evalua-    knowledge   than   would   a   didactic   presentation.   The
      tion of learning was planned.                                  actual proceedings at  the  session  involved  active  debate
         Specifically, three surrogate projects  were  presented,    concerning  which  aspects   were   good   and   which   were
      one at a time. In  each  case,  desirable  and  undesirable    not. Errors in  judgement  by  a  given  participant  -  from
      aspects were listed, as  volunteered  by  the  participants    the   moderator's   point   of   view    -    were    quickly
      following  review  of   the   projects.   Explanation   and    lampooned by others. The resulting  list  at  the  conclusion
      analysis of the projects was  led  by  the  moderator,  Dr.    of the conference  was  educationally  quite  sound.  Insofar
      Daniel Fleisher of  Temple  University,  Philadelphia,  as-    as.no evaluation of learning  was  planned,  the  product  of
      sisted by Dr. William Munier.                                  the  session  was  good,   and   all   present   participated
        Two  of  the   projects   were   poorly   designed,   and    actively,  the   conference   was   subjectively   judged   a
      contributed the bulk of the  undesirable  aspects.  One  of    success.




















                                                                                                                          179







                                      EVALUATION OF MULTIPHASIC SCREENING


                                                                Participants

       Donald N. Logsdon, M.D.                                          Frank R. Mark, M.D.
       Associate Director, Multiphasic Health                           Chief, Operations Research and Systems
         Screening Center                                                  Analysis
       Department of Community Health                                   Regional Medical Programs Service
       Brookdale Hospital Center




               Evaluation of Multiphasic Health Testing                    Tayback  considered  establishment  of   a   multiphasic
                                                                        screening (testing service to be  based  upon  the  follow-
                       DONALD N. LOGSDON
                                                                        ing operational model.
                                                                           Therefore,   evaluation   of   MHT    projects    funded
          In  the  chapter  entitled  Evaluating  the   Quality   of    through NCHSR&D should proceed on three levels.
       Medical  Care  by   Avedis   Donabedian   from   the   recent    ACHIEVEMENT       OF       TECHNOLOGICAL        OBJ.ECTIVES
       book  Presymptomatic  Detection  and   Early   Diagnosis   by       PHS was  at  that  time  proceeding  on  the  assumption
       Shark   and   Keen,   the   conclusion   is   reached    that    that MHT is basic to the attainment of  a  national  health
       "alth.ough the assessment  of  the  quality  of  medical  care   objective - periodic assessment of  the  health  status  of
       remains difficult and imprecise, there are  several  ways  in    each adult, 35-69 years of age.  The  system  must  undergo
       which one may arrive at judgment  sufficiently  valid  for  a    continuous  improvement  resulting  in  added  validity  of
       variety  of  administrative  decisions".   Among   the   ways    the health testing  and  in  improved  cost  effectiveness.
       suggested were "studies of the effect  of  greater  precision    Specifically,  it  was  recommended  that  the  achievement
       and detail in standards on the reliability  and  validity  of    of  technological  advances  in  MHT  be  measured  by  the
       judgments   (measurement)".   As   applied   to    MHT    the
       current  operating  programs  have  attempted  several  eval_    completion of defined tasks and with  time  specified  end-
       uation studies which I  will  briefly  describe  and  comment    points.  During  the  twelve  month  period,   January   1,
       on.                                                              1969 - December 31,  1969,  the  following  tasks  were  to
                                                                        be initiated and progress reports submitted by the  end  of
          Dr.  Matthew  Tayback,  in  several   meetings   sponsored    the period- These tasks are not a complete  description  of
       by the U.S. Public  Health  Service  in  1967-68,  set  forth    the technical problems which need solution.
       criteria which he suggested  for  determining  the  value  of
                                                                        Glucose Tolerance Test
       Multiphasic  Health  Testing.  He  restated  the  proposition       It is im
       that evaluation should rest  on  the  success  of  attainment              perative to  detern-dne  the  relationship  which
       of  project  objectives,  namely,  (1)  per  cent  of  target    exists  between  the  result  of  the  abbreviated  glucose
       sample reached  (2)  precision  and  accuracy  of  individual    tolerance  test  as  employed  in  MHT  and  standard  oral
       measurements  (Quality  Control)  (3)   yield   of   screened    glucose  tolerance  test  as  performed   in   conventional
       positives per  major  procedure  (4)  per  cent  of  screened    hospital or private laboratory centers.
       positives who  make  contact  with  personal  physician,  and       The effect of time  of  day  on  the  abbreviated  tests
       (5)  per  cent'  of  screened  sample   with   minimum   sig-    must be clarified.
       nificant  benefit   in   health   knowledge   due   to   MHT.
       Although it is highly pertinent and  eventually  critical  to    Standardization of Norms
       consider  cost-benefit  characteristics  or  end  results  of       Interpretation of results from clinical lab  tests,  in-
       MHT,  such  data  will  not  be   forthcoming   for   several    cluding blood  chemistry,  hematology,  and  non4ab  tests,
       years.  In  the  meanwhile  MHT  technology   needs   to   be    such as spirometry, by the practicing physician  is  diffi-
       advanced on the principle of its cost-effectiveness  and  its    cult when the normal  population  ranges  for  a  specified
       capability to efficiently process large populations.             measurement is not given in the report.

                                                                                                                                181





                               A.       Service inputs         provided to  a       B.    Target Population         produces
                                                                                                  of
                                               of                                         Specified Quantity
                                           Specified                                      Composition and
                                            Quality                                            Behavior



                                C.                             which     when       D.   Personal Physicians       produces
                                        Individualized        interpreted  by                Utilizing the
                                            Health                                          Information in
                                         Information                                         A Specified
                                                                                                Manner







                                                          E.
                                                                Desirable    Benefits
                                                                  Which   Can   Be
                                                                     Specified



                  A   standard   procedure   for   reporting   MHT    results    Quality Control
               should   be   adopted.   The   exact   measurement    obtained       Each  major  component   of   MHT   requires   a   protocol
               should be reported and the  normal  range  for  the  age,  sex    for  establishment  and  control  of  quality  of  measurement
               and ethnic group category given.                                  and  test   information   generated.   Each   project   should
                  Since  the   distribution   of   deflned   measurement   by    develop  a  manual  of  procedures  in  respect   to   quality
               age, sex, and ethnic groups  has  not  been  determined,  this    control.  PHS  should  then  produce  a  standard  manual   on
               should be developed as soon as possible.                          quality control and annually update this document.

               Standardization and Documentation of                                  ACHIEVEMENT OF PROGRAM OBJECTIVES
               Computer Programs                                                    Pending  demonstration  of  benefits  relating  to   reduc-
                  Inefficiency (excessive cost) is generated  by  failure  to    tion in disease, disability  and  age  specific  death  rates,
               develop  systems  which   can   be   replicated   with   minor    MHT must receive  process  evaluation  on  the  basis  of  the
               adjustments.                                                      attainment  of  program  objectives  and  the  cost-effective-
                  Existing   computer   monitoring   of   SMAIZ    and    VCG    ness of services.
               interpretation needs to  be  validated  with  a  view  towards       Such  process  evaluation   will   be   possible   by   the
               selection  of  a  standard  program  so  that  widespread  use    following strategy:
               can  be  made  of   the   standard   @rograms   with   minimum       1. Each project should  set  forth  the  target  population
               further  investment  for   software   development   in   these    it seeks to  reach  with  its  screening  program  and  should
               specified areas.                                                  specify  the  fraction  of  the  target  population  which  it
               Cost Analysis                                                     proposes to reach.
                                                                                    2.  Periodically   (quarterly)   demographic   characteris-
                  Major   components   of   MHT   needed   to   be   defined.    tics of  the  screened  population  were  to  be  reported  to
               Each component must then  serve  as  a  unit  for  the  deter-    PHS. Comparison of 2  with  I  will  indicate  the  extent  to
               niination of cost.                                                which  the  target  population  is   reached.   The   minimun,
                  Cost analysis   data  should  be   generated   within   the    set of  variables  for  which  information  is  sought  should
               next 12 months.                                                   include  age,  sex,  race,  income,  occupation,   source   of
                  In  view  of  the  limited  staff,  this  task  should   be    regular medical  care,  utilization  of  medical  care  within
               accomplished   through   a   contract   negotiated   with   an    past six  months,  date  of  last  general  physical  examina-
               interested and competent cost analysis service.                   tion, and follow-up results.

               182




           3. The yield  of  significant  positive  findings  per  exami-        3.   Documentation   and   evaluation   of   the   experiences
        nation  procedure  for  age,   sex   and   ethnic   groups   will     with  this  type  of  health  service  as   compared   with   the
        permit  assessment  of  the  cost   per   abnormality   detected.     existing  health  services  of  the  Hospital   Ambulatory   Care
           4.   The   patient-physician   contact    ratio    by    major     Program.  A  central  record   system   would   enable   monitor-
        screening classification type is  a  measure  of  the  extent  to     ing of the two types of care.
        which   significant   screening   findings   receive    follow-up        4.   Further   utilization   of   paraprofessional   personnel
        exai,ns.                                                              and  instrumentation  in  health  care.  The  use  of   physician
           5. A survey  of  selected  classes  of  the  screened  popula-     assistants,   nurses,   technicians   plus   hardware   can    be
        tion,  prior  to  and  following  the  date  of  visit   to   the     tested.
        screening  center  could  provide   suggestive   clues   relative        The  above  factors  are  considered  important   in   assess-
        to the health attitude and knowledge of consumers and -               ing  the  difference  MHT  makes  in  the  delivery   of   health
        could  provide  information   of   consumer   reaction   from   a     services. It is recognized  that  the  addition  of  a  Follow-up
        single exposure to MHT (a consumer study).                            Clinic  would  not  alone  provide  an  answer   to   the   ques-
           6.  A   questionnaire   survey   of   physicians,   who   have     tion  of  benefits  in  terms  of   biologic   outcome   or   end
        received  MHT  reports,   with   a   view   towards   determining     results.  However,  as  the  methodology   for   this   type   of
        their  attitudes  and  knowledge  of  the   usefulness   of   MHT     evaluation  is  adequately  developed,   a   prospective   longi-
        would complement information obtained through 5.                      tudinal   study   of   morbidity,   mortality,   and   disability
                                                                              could be attempted.
           ACHIEVEMENT OF LONG RANGE OBJECTIVES                                  We  began  to   evaluate   MHT   at   BHC   as   part   of   a
           Control  of  clinically  significant   chronic   disease   and     primary  health  care  system  at  intermediate  points  and   to
        prolongation of  life  are  end  results  of  MHT  which  can  be     determine feasibility of assessing end results.
        demonstrated    only    by    ambitious    research     involving        In  December  1969  a  subcontract   was   signed   and   work
        careful  prospective  follow-up  of  large  samples   of   adults     begun for biostatistical  retrieval  and  analysis  of  the  data
        over long periods of  time.  Such  a  project  is  under  way  at     on   the   14,000   screenees   processed   at   the    Brookdale
        the  Kaiser-Permanente  medical  service  in   and   around   San     Hospital   Center   MHS   program   from   the    beginning    in
        Francisco.                                                            February  1968  through  October  1969.   Initially,   the   data
           it   was   recommended   that   aside    from    Kaiser-Perma-     were  examined   in   terms   of   frequency   distribution   for
        nente, no  extensive  investment  of   funds   should   be   made     continuously   distributed   quantitative   variables   by   age,
        at this time to demonstrate long term effects of MHT.                 sex,   and    ethnic    background.    Dichotomous    qualitative
                                                                              variables  were  tabulated  an(f  percentage   positives   calcu-
                         PROCESS EVALUATION                                   lated  also  by  age,  sex,  and  ethnic   background.   Measure-
           The Brookdale Hospital, Multiphasic         Health     Screen-     ments  of  central  tendency   and   variation   were   performed
        ing   program   has   been   a   successful   demonstration    of     on   continuously   distributed   measurements.   This   included
        AMHT.   The   questions   now   to   be   answered   are:    What     mean,  standard  deviations,  median,  5  and   95   percentiles.
        difference  does   MHT   make   in   the   delivery   of   health     The   number   and   percent   of   screenees   with   clinically
        services  in  an   urban   environment?   Can   MHT   become   an     significant   overt   and   occult   abnormalities    based    on
        effective  component  in'  a  primary  health  care  system?   it     currently  acceptable  critera  was  detem-dned   also   by   the
        was established  by  PHS  support  to  test  the  feasibility  of     variables  of  age,  sex,   and   ethnic   background.   Investi-
        operating   a   MHT   program   in   an   "open"   medical   care     gations  will  also  be  made  for  correlation  analysis,  i.e.,
        system with an adjacent poverty population.                           history vs. test results, and  screening  results  vs.  physician
           In  order  to  answer  these  questions  it  would  be  neces-     diagnosis.  This  effort  has  been  successful   for   Brookdale
        sary  to  "close  the  infomwtion   loop"   by   establishing   a     MHS   and   should   have   application   to   other   demonstra-
        Follow-up  Clinic  which   would   have   enabled   the   program     tion programs in MHT.
        to accomplish the following.                                             Problems of data retrieval and analysis include:
                                                                                 1.    Quality    of    input-measurement    and    keypunching
           1.  Provision  of  the  necessary  follow-up  medical   evalu-
        ation  and  management   for   the   screened   poverty   popula-     errors.
        tion.                                                                    2.  Storage  on  historical  tapes,  i.e.,  completeness   and
           2. Validation  of  the   screening   results   by   comparison     documentation
        with  results of  diagnostic  studies  for  the  poverty  popula-        3.  Retrieval     group   intervals,   criteria   of   normal,
                                    physicians.                               abnormal.
        tion by Follow-up Clinic

                                                                                                                                            183






                 4.  Analysis  -  Mean  or  median,  standard  deviation   or       2. In regard to the latter,  the  input  for  a  time-effort
              percentfle, test of significance.                                  study  has  been  built  into  the  computerized   module   for
                                                                                 sui)vort  of  follow-up  activities.  These   components   must
                 An  econon-dcal  evaluation  can  be  approached   by   cost
                                                                                 be  costed  out  in  a  AMHT  in  addition   to   those   items
              per test as indicated in the  cost  finding  protocol  for  the    included  in  the  recent  reports  on  costs  of  the  Kaiser-
              past project year. The SRI method is being tested.                 Permanente  MSP  in  the   New   England   Journal   of   Medi-
                 Effectiveness is being  evaluated  in  terms  of  the  yield    cine.  These  additional  items  will  obviously  increase  the
              of  unknown  and   uncontrolled   occult   and   overt   condi-    cost   of   AMHT   programs   involved   in   motivation    and
              tions  detected  at  the  MHT  Center.  This,  of  course,   is    follow-up  and  the  question  to  be  answered  is  how  much.
              related to the prevalence of  disease  in  the  target  popula-       In  addition,  a  cost  effectiveness  report  can  be  pre-
              tion. High  yields  are  expected  for  certain  conditions  in    pared  for  the  follow-up  activities  wherein  comparison  of
              poverty  populations,  groups,  due  to  prevalence   and   the    costs  for  furnishing  the  follow-up  services  using  alter-
              lack  of  adequate  medical  care,  e.g.,  rates   for   hyper-    native methods  will  be  used  with  the  objective  of  being
              tension  and   hypertensive   heart   disease.   The   methodo-    able  to  minin-dze  the  resources   expended   and   maximize
              logical   problem   of   determining   "unknowness"   can    be    the  number  of  individuals  receiving  medical   follow   up.
              solved  by  the   use   of   questionnaire   information   from       Investigate   Consumer.   and    Physician    Reaction    to
              patients  rather  than  from  M.D.'s.   Efficiency   is   being    AMHT.-
              calculated on the basis of  the  cost  per  positive  screening       1.  During  the  1969  project  year  50%  of  the  individ-
              test and cost per valid diagnosis.  Of  course  the  latter  is    uals  screened  resided  in  the  Hospital's  core  area,   and
              dependent  on  adequate  follow-up   reporting.   These   eval-    25%  were  Black  or  Puerto   Rican.   A   number   of   tech-
              uation   efforts   should   be   performed   as   the   program    niques for increasing registration  from  the  hard  core  high
              activities are  carried  out.  Simultaneously,  the  end-result    priority areas  were  tested.  Good  progress  has  been  made,
              evaluation is being explored for  feasibility  in  an  environ-    but  it  is  apparent  that  "hard"  data   on   the   behavior
              ment   which   prohibits   randomization   into    study    and    factors  are   necessary   to   improve   performances   beyond
              control  groups   for   longitudinal   study.   Present   plans    this point.
              could  include  labehn-  a  sample  of  the  screened   poverty       Similar  considerations  are  involved  in   improving   the
              population   for   monitoring   over   time    and    comparing    70% figure for successful follow up.
              their  experience  in  morbidity,  mortality,  and   disability       The  data  generated  by  the  screenee  process   must   be
              with  non  poverty  population   and/or   national   statistics    evaluated by  a  physician  in  the  context  of  his  examina-
              for the same age, sex, and ethnic group.                           tion  of  a  specific  individual.  The  physician's  knowledge
                 For   preliminary   results   for   total   population   see    of   and   attitude   toward   AMHT   therefore   becomes    of
              Appendix I.                                                        central importance.
                 Detennine  the  cost  of  MHT  in   affinwry   Health   Care       The  staff  of  the  Brookdale  MHSC  is  actively   engaged
                                                                                 in  assisting  the  research  group  at   Columbia   University
              System:                                                            School  of  Public  Health  and  Administrative   Medicine   in
                 1.  The  Brookdale  MHSP,  as  a  result  of  the  SRI  Cost    the   development   and   implementation   of   two    relevant
              Finding  Study  of  ANIHT  has  begun  to  examine   the   cost    studies:     Consumer      Reaction      to      AMHT      #HSM
              of        the        totalprogram.Informationisbeingcollectedon    110-69-212,    and    Physician's    Attitude    Toward     and
              the  total  expenditures   for   this   program   through   the    Acceptance of AMHT #HSM I 10-HSRD-57 (9).
              Brookdale Hospital  Center  business  office.  However,  it  is       2.   The   Physician   Attitude   Study   is   designed   to
              apparent that  a  true  cost  analysis  of  this  program  will
                                                                                 deternune:
              require  the  establishment  of  bookkeeping  -  cost  account-       a. What   are   the   social   and   psychological    factors
              ing  procedures  separate   from   the   Hospital   System   in         which  affect  the  physician's  cooperation   with,   his
              order to identify  the  various  costs  involved.  In  addition         acceptance   of,   and   his   behavior   concerning   the
              to the usual items that are  included  under  direct  cost,  it         MHT  at   Brookdale,   including   those   factors   which
              will  be  necessary  to  itemize  those   costs   involved   in         facilitate  his  utilization   and   acceptance   of   the
              recruiting the target population to utilize  the  facility  and         service, as well  as  those  factors  which  are  barriers
              the follow-up activities.                                               to effective utilization?


              184





             b. What factors differentiate those  physicians  who                           In  your   opinion,   did   the   summary   contain   more   information
                actively participate and accept the screening pro-                       than  was  necessary,  just  about  the  right   amount   of   information,
                gram from those who  do  not,  and  what  factors                        or not enough information?
                                                                                            Ntore information than necessary                   .......... 123 39%
                differentiate     those     physicians     who     change      their        About right amount of information         . . . . . . .  .  .133  42%
                attitude and behavior concerning ANLHS?                                     Not enough information        . . . . . . . . . . . . . . . .59   19%
             c. How    can    an    automated    system    such    as    multiphasic
                screening    be    made    more    useful    and    acceptable    to                                                                    315   100%
                the practicing private physician?                                           How easy was it for you      to   follow   the    general    layout of the
             d. How    do    physicians    adapt    their    practices     to     an     summary? Was it very easy, fairly easy,        somewhat    difficult,    or
                automated health testing program?                                        very  difficult?
             e. What    inferences    can    be    drawn    from    this    specific        Very     easy                       .......................... 111 34%
                study    to    the    more    general    area     of     physician's        Fairly   easy  .........................                    112   34%
                response    to    automation     in     medical     practice     and        Somewhat difficult                       .................... 70  22%
                                                                                            Very   difficult                     ........................ 32  10%
                what impact does it have on medical practice?                                                                                           -     -
             In   order   to   perform   this   type   of    study    the    project                                                                    325   100%
         must   actively   engage    the    support    of    the    local    Medical
         Society. This can present a difficulty.                                            In  your  opinion,  should  the  normal  range  of  results   be   indi-
                                                                                         cated on the sumniary?
             The     study     of     physicians'     reactions     to     automated        Yes   ..............................                        290   90%
         multiphasic    health    screening    presently     provides     for     an        No    ................................                      31    10%
         initial   and   a   follow-up   survey   10   months    later    of    1200
         physicians   in    Kings    County,    New    York.    The    two    inter-                                                                    321   100%
         views    will     determine     their     attitudes,     knowledge,     and        Was  the  blood  glucose  test  and  result  clear  to   you?
         utflization    of    the    Automated    Multiphasic     Health     Screen-        Yes   ..............................                        280   90%
         ing    Center    at     Brookdale     Hospital     in     Brooklyn,     New        No    ...............................                       30    10%
         York.
             The    re-interviews    were     intended     to     concentrate     on                                                                    310   100%
         changes   in    attitudes,    behavior,    and    perception    of    auto-        Was   the    histogram    arrangement    of    the    hearing test results
         mated   screening    resulting    from    exposure    to    the    program.     clear  to   you?
         When   the   study   was   planned   it    was    anticipated    that    at        Yes   ..............................                        186   63%
         the   time   of   the   first   interview,   at   least   half    of    the        No    ..............................                        111   37%
         doctors    would    in    the    interim    become    exposed    to     the                                                                    297   100%
         Brookdale    program    and,    as    a    consequence,     alter     their
         image of it.                                                                       How    useful    was    the    medical    history    questionnaire? Was it
                                                                                         very  useful,  somewhat  useful,  not  very  useful,   or   worthless?
             However,   results   of   the   first   wave   of   interviews    indi-        Very useful    ..........................                   34    12%
         cate    that     diffusion     of     the     screening     program     has        Somewhat useful                       ....................... 92  34%
         occurred    more    rapidly    than    anticipated.    This    fact     has        Not very useful                       ....................... 92  34%
         bearing   on   the   timing   of   the   re-interviews    and    in    part        Worthless                        ............................ 55  20%
         motivates this suggested modification.                                                                                                         273   100%
             Of    the    first    712    completed     interviews,     only     101        What  did  you  think  of   the   fraction   arrangement   of   positive
         physicians   have    not    been    exposed    to    the    program    (86%     responses  by  body  system?  Did  you  think  this  was  a  good  way   of
         were   exposed).   There   is   no    reason    to    expect    that    the     presenting  the  medical  history  information   or   not   a   very   good
         remaining   interviews    will    show    much    departure    from    this     way?
         7:1   ratio.    Therefore,    we    cannot    expect    dramatic    changes        Good      way  .........................                    127      50%
         between   the   first   and   second   interviews   as    a    result    of        Not a very good way                        .................. 127      50%
         contact   with   the   program.   Some   early    results    from    inter-
         views of doctors follows:                                                                                                                      254     100%



                                                                                                                                                              185






                  How helpful did you find this reference  manual  in  reading                 Pretest   in   local   laboratory.    Example:    cholesterol
               the patient summary? Did you find it very helpful,  or  somewhat                studv.
               helpful, or not at all helpful?                                              d. Design   general   outlines   of   AMfiT   internal   quality
                  Very helpful              .........................     75     44%           control     system:     calibration,      serum      monitor,
                  Somewhat helpful                  ..................... 74     43%
                  Not at all helpful                ....................... 23     13%         laboratory   responses;   design   a   system   of   external
                                                                                               evaluation.
                                                                        172     100%
                                                                                                   OTHER STUDIES OF MULTIPHASIC
                  Now  about  what  you  think   should   be   done   by   screening                         HEALTH TESTING
               programs like  Brookdale's.  Do  you  think  that  a  screening  pro-
               gram  like  Brookdale  should  be  free  of  cost  to  examinees   or       1.  HIP  -  Utilizing  MHT   to   define   the   health   status,
               should there be a charge?                                                practices,  and  attitudes   of   a   defined   poverty   population
                  Should be free                ....................... 152      50%    covering  a  broad  age  range  (12  yrs.  +)   from   an   absolute
                  Should be a charge              ..................... 152      50%    standpoint   and   relative   to   a   nonpoverty   group   in   the
                                                                        304    1007@    same    medical    care    environment.     Action     to     modify
                                                                          -             adverse  aspects  of  the   health   co
                                                                                                                              mponents  is  to   be   insti-
                  Do you think  that  a  screening  program  like  Brookdale  should    tuted  and  evaluation  is  in   terms   of   change   as   compared
               refer  both  normal  and  abnormal  patients  for  follow-up   by   a
               physician or only patients  with  some  positive  condition?             with   what   occurs   in   the   non-poverty   group.   An   under-
                  Both normal and abnormal               .............. 241      67%    lying  question   is   whether   through   the   MHT   program   and
                  Positive condition only             ................. 103      33%    activities  generated  by   it,   the   anticipated   gaps   between
                                                                                        the  two  groups   can   be   narrowed.
                                                                        344     100%       The   program   expects   to   begin   processing   patients   in
                                                                                        November 1970.
                  C7inical    Laboratory     Qua-lity   Control   Studies.,                2.   Meharry   Medical   College   MHT   Project   -   Evaluation
                  After  several  attempts  over   two   years,   there   has   been    of  this  project  will  be  performed  as  part  of  the  study  on
               relatively  little   success   in   providing   assistance   to   the    comprehensive health services by Dr. Sam Wolfe.
               Clinical  Labs   in   AMHT   for   developing   a   sufficient   pro-       3.   North   Florida   .,   RMP,   Gainesville,   Florida,    Dr.
               gram  of  quality   control.   The   Clinical   Chen-dstry   Section,    Richard    Gordon    and    Co-workers.
               NCDC,    has    repeatedly    demonstrated    their    interest    in       In   summary   MHT   is   a   complex,   relatively    expensive,
               providing   this   support,   but   various    bureaucratic    delays    experimental   system   of   health    services.    Evaluation    in
               have   prevented   any   progress.   The   problem    of    assisting    terms  of  program  effectiveness   and   efficiency   is   feasible
               these  labs   remains,   and   a   modest   beginning   is   proposed    but  the   methodology   for   successful   end   result   or   out-
               for the next project year.                                               come  evaluation  has  yet  to  be  demonstrated   for   the   total
                  This   effort   would   initially   consist   of    a    six-month    system.   MHT    is    advettely    affected    by    two    circum-
               study   and   evaluation   of    AMHT    interlaboratory    standard-    stances:
               ization  utilizing  the  Brookdale  Hospital  Clinical   Lab   as   a       1.  It  appears  to  o    easy    and    glamorous    which    is
               starting   point,   and    then    extending    the    protocol    to    probably   the   result   of    over-selling    the    technological
               include   the   other   AMHT   labs.   The   brief    outline    that    developments,   when   in   fact   there   are   multiple    techno-
               follows describes the activities and resources required:                 logical  problems  still   to   be   solved.   The   major   program
                  Study     and     Evaluation     of      AMHT      Interlaboratory    problems   involve   the   recruitment   of   the   target   popula-
               Standardization:                                                         tion   and   providing   adequate   follow-up   for   the   individ-
                  Preparatory    efforts    -    Brookdale     AMHT     and     NCDC    uals tested.
               through individual and group interaction.                                   2.  The  latter  relates  to  the  major   uncontrollable   vari-
                  a. Develop     recommendations      for      reference      method-   able  in  assessing  the  value  or  benefit   of   MHT   and   that
                    ology, enzyme units.                                                being  the  lack  of  proven  therapy  for  most  of   the   chronic
                  b. Anticipate     problems     in      SMA      technology      and   conditions detected.
                    calibration.                                                           After  struggling  with  evaluating   MHT   for   several   years
                  c. Design   and   prepare   Multiphasic   Text   Panel   for    the   I  usually  caution  people   about   trying   to   implement   this
                    eucidation    of    methodologic,    technical,     and     cah-    system  of  health  services  and  especially   to   think   through
                    bration    problems.    Check    stability     of     materials.    the planned evaluation.



                186




                                  References                                       4.  Metropolitan   Life   Statistical   Bulletin,   Dec.   1969    Bio-
                                                                                       chemical    Profiles
          1. Sharp  CLE  &   Keen   H.   Presymptomatic   Detection   and   Early  S.  Use  of  "Normal  Range"  in  Multiphasic  Testing  Files,  et   al
           Diagnosis  Williams  &   Wilkins,   Baltimore,   MD.   1968                 JAMA. Sept. 2, 1968. Vol 205 No 10, p. 684
          2. Collen   et   al,   NEJM,    283:459-463    1970.and    Vol.    280;  6.  Clinical   and   Biological   Observations    on    Working    Men,
           1043-1045       1969                                                        Clark,  T.W.  et  al  Arch  Environ.  Health,  Vol  19.  Nov  1969,
          3. Neuffield Foundation Report Screening in Medical Care                     p. 700




           APPENDIX 1
                              NUMBER AND PERCENT PREVALENCE OF CLINICALLY SIGNIFICANT ABNORMALITIES
                                                                     ON 13,000 SCREENEES
                                                          THE       BROOKDALE       HOSPITAL        CENTER
                                                       MULTIPHASIC       HEALTH       SCREENING        CENT
                                                            FEBRUARY     1968     -     NOVEMBER      1969
           Test                                                               Brookdale   Hospital   Center                    KaiserPermanente*
                                                                                     Total  all   ages                            Total all ages
                                                                              No.                    170                  170                  Cost
           Blood  pressure  >  160/95                 ................        4058                   31.5                    4.1
           Electrocardiogram     ....................                         3203                   25.0                  17.-3                  $S.90
           Chest X-ray                        ........................        1053                   8.8                     7.4                   6.20
           Cervical cytology,  III  ..................                             8                 .13
           Visual Distant >20/40    ..................                        1917                   14.9                   15.8                   1.85
           acuity:   Near   >iO/50  ..........        ...         ....        817                    6.8

           Tonometry:
           >   21.9    mmHg:    OD  ..................                        545                    4.7
                              Os    ..................                        659                    5.7
           >   23.8    mmHg:    OD  ..................                        194                    1.7                     0.3                 183.00
                              Os    ..................                        25S                    2.2
           Spirometry:
           Pred.   FVC   <80,7o  ....................                         2505                   24.1                    2.2                  14.10
           Pred.    MW<     90%  ....................                         2327                   22.4
           Audiometry  >   30   db  ..................                        3050                   28.3                   16.2
           Dental: Teeth, poor or bad  .                ..............        1454                   11.7
           X-Ray
                     Edentulous  ....................                         1333                   18.0
                 Alveolar bone loss severe               .............        1439                   19.5
                  Other   X-ray   abn .                 ................      8785                   37.8
           Cytology II-IV     ......................                               31                    .25
           *NEJM Vol 280 No. 9 p. 4S9463







                                                                                                                                                        187














                                                                               CLINICAL LABORATORY TESTS


                         Tests                                                   Abnormal     limits                   Clinically significant abnormalities

                                                                                                         Brookdale Hosp. Center                   KaiserPermaneizte
                                                                                                               Total  all  ages                      Total all ages
                                                                                                        No.                                   %                 Cost


                         Hemoglobin:
                                Females    ....................                  < 12 gms. %            737                9.5                10.3
                                Males   .....................                    < 13 gms. %            249                5.3                3.1
                                Total   .....................                                           986                7.9


                         Hematocrit:
                                Females    ...................                   @      38%             614                7.9
                                Males   .....................                    <      40%             122                2.6

                         WBC    .........................                        <  4  &  >  12,000/                       3.4                2.2
                                                                                          cu mm


                         RBC:
                                Females    ...................                   <  4.2  ni
                                Males   .....................                    <4.5     m


                         Cholesterol:
                                > 95 percentile for age                      .......................    622                5.0
                                Males   .................................                               235                5'0
                                Females    ...............................                              387                4.99


                         VDRL                          .......................   Positive               81                 .7                 I.S


                         Urine:
                                Culture:
                                Females    ...................                   > 16  col.             679                8.8                3.3
                                Males   .....................                    > 16  col.             6  3               1.5                0.4


                                Glucose    ...................                   1+  to  4+             780                6.3                8.2
                                Protein    ...................                   1+  to  4+             550                4.4                6.4
                                Acetone    ...................                   1+  to  3+             155                1.2




























                   188





                                      EVALUATION OF STROKE - REHABILITATION

                                                                    Participants
        Jerome Tobis, M.D. - Moderator                                            B. Lionel Truscott, NI.D.
        chairman Department of Physician Medicine                                 Director, Stroke Program
        and Rehabilitation                                                        North Carolina Regional Medical Program
        University of California at Irvine                                        Philip A. Klieger, M.D.
        Charles M. Wylie, M.D.                                                    Assistant Director for Organizational
        professor, Public Health Administration                                    Liaison for Stroke and Rehabilitation
        University of Michigan                                                    Regional Medical Programs Service

        Bertram L. Tesman, M.D.
        Assistant Coordinator, Stroke Program
        California Regional lvfedical Program -
        Area VIII


        Evaluating Stroke and Rehabilitation Programs:                            moderate stress, and use that  stress  to  galvanize  us  into
                          An Overview                                             improvements  rather  than  into  fits  of  depression.   This
           CHARLES M. WYLIE, M.D.                                                 i-night  be  regarded  as  the  power  of  positive   thinking
                                                                                  about evaluation.
          At this late stage  of  the  conference,  evaluation  is  no
        longer an attractive  and  novel  word.  The  discouraged  or                   WHO SHOULD DO THE EVALUATION?
        bored may suspect the reality  of  the  Turkish  proverb:  If               First, a  brief  word  about  the  site  of  evaluation.  To
        a stone falls on an egg, alas for the egg; if  an  egg  falls             increase the likelihood of acting on the  findings,  it  seems
        on a stone, alas for the egg. If  we  fail  to  evaluate  our             essential that the effect of  RMP's  on  the  national  health
        program,  alas  for  the  program;  if  we  do  evaluate  our
        program, alas for the program.                                            levels be assessed by those  working  in  the'federat  office,
          To evaluate or not to evaluate -  that  is  not  the  ques-             the  effectiveness  of  regional  efforts  be   evaluated   by
        tion for  those  of  us  who  wish  to  continue  working  in             those in regional  offices,  while  the  evaluation  of  local
        RMP'S. Society has  always  advised  us  to  be  critical  of             program, be carried out  by  local  personnel.  Too  often  in
        what  we  do.'  The  saying,  afl's  well  that  ends   well,             their  health  activities  federal  personnel  evaluate  state
        reminds  us  that  even   centuries   ago   activities   were             activities, states evaluate the local  picture,  locals  don't
        considered  good   primarily   when   their   outcomes   were             evaluate,  and  little  change  occurs.   Fortunately,   RMP's
        good. Thus  the  salient  question  is:  how  can  evaluation             have learned from raistakes made elsewhere.
        be  a  constructive  force  which  improves  programs  rather                The  evaluation  findings  are  more  likely  to  be  acted
        than a destructive force for the eradicatioti  of  programs'?             on  when  program  personnel  evaluate  the  effectiveness  of
        It will destroy, for example, if it uses criteria  which  are             ,their- own activities. -Examining  evaluation  realistically,
        so strict that we cannot meet them. It will also  destroy  if             however, we must admit that the first priority              of    the
        it uses so much of our resources that  we  have  little  left             agency  staff  is  to  continue  the  program;   program   im-
        to run good programs.                                                     provement  is  only  a  secondary  goal,  and  destroying  the
           Must evaluation affect us adversely,  however  9  It  will             program  is  their  great  fear.  They  may  often  feel  that
        if we insist that it be completely free from stress. It  will             "conventional  wisdom"  from  which  the  program   arose   is
        if we expect it to  resemble  the  French  view  of  love,  a             more  important  than  negative  evaluation  data.  They  will
        pleasant  diversion  between  meals,   or   even   more   the             correctly add  that  some  decisions  must  be  political  and
        Swedish  view,  a  pleasant  diversion  during   meals.   But             humanitarian,  neither  of  which  viewpoints  is   considered
        evaluation  won't  harm  us   if   we   expect   and   accept             in evaluation.
                                                                                                                                                     189





                  'fhe  likelihood  of  corrective  action   may   be   lower    ADEQUACY  of  what   we   do   (how   much   of   the   entire
              with  an  outside  evaluator,  who  may  have   other   biases.    problem we are likely to overcome).
              He may  view  evaluation  as  a  chance  to  test  theories  or       Documenting  a  change  in  resources  is  a   step   which
              methods  which  interest  researchers.  He  may   suspect   the    can be swift and  cheap;  in  our  concern  to  "get  on  with
              evaluation effort, perhaps  from  bitter  past  experience,  as    the job," it is only too  easy  to  stop  evaluation  at  this
              designed to  give  the  program  a  legitimacy  which  it  does    point. To  ensure  the  long  term  survival  of  RMP'S,  how-
              no  t deserve. He may suspect further that  a  critical  evalu-    ever,  and  to  gain  information  on  how  our  programs  may
              ation will be  ignored,  or  that  negative  outcomes  will  be    be improved, we must regard this as only  the  first  step  in
              quietly  forgotten  so  as  to  ensure  the  growth  of  future    providing  more  convincing  information  on  the   value   of
              funds. Such events,  we  may  hope,  will  be  rare  in  RMP'S.    stroke and rehabilitation programs.
                  In  evaluating  stroke  and  rehabilitation  programs,  our
              efforts are likely to aim at three levels of information:                            ACTIVITY CHANGES
                  1.  Changes  in  resources,   including   the   number   or       Many  activities  are  held  to  be  desirable  when   they
                     quality of trained personnel.                               seem likely to delay  the  onset  of  stroke  or  improve  the
                  2.  Changes  in  the  activities  produced  or   the   work    function  and  speed  the  recovery  of  stroke  patients.  An
                     performed by these resources.                               effective  change  in  resources,  as  described  above,  will
                  3. Changes in the end results of these activities.             result in  more  of  these  desirable  activities;  we  should
                  @t  us  consider  the  strengths  and  weaknesses  of  each    show that this has  truly  happened.  The  process  of  evalu-
              level of evaluation.                                               ation  becomes  more   complete   and   impressive   when   it
                               REsoURCE CHANGES                                  shows  clearly  that  the  new  or  improved  resources   have
                                                                                 truly raised the output or  quality  of  activities  as  well.
                  RMP  funds  may   improve   the   quality,   quantity,   or    'Me steps  to  collect  these  data  must  be  planned  before
              both,  of  facilities,  personnel,  knowledge,  or  other   re-    the  resources  are  changed.  This  advance  planning   makes
              sources  involved  in  producing  stroke   and   rehabilitation    it possible to contrast the activities before  and  after  the
              activities.  A  new  hypertension  clinic  may   be   supported    change occurs.
              to prevent stroke, another clinic  established  for  the  early       Let  us  take  the  situation  where  an  educational  pro-
              detection  and  treatment  of   transient   ischemic   attacks.    gram  has  been  shown  effective  in  improving  the   knowl-
              Health  personnel  may  attend  new   courses   which   review,    edge  of  the  participants.  We  wish  to  show   that   this
              for  example,  the  optimum  care  of  stroke  patients.   More    change  in  resource   produces   a   change   in   subsequent
              rehabilitation  personnel   may   be   recruited   to   consult    activities.  One  goal  of  an  educational  program  may  be,
              with  personnel  in  home  care  programs  or   extended   care    for  example,  to  encourage   physicians   to   make   better
              facilities.                                                        diagnoses   on   their   hospitalized   stroke   patients.   A
                  If  RMP  personnel  document  that  such   resources   have    regional committee of experts or of  peers,  let  us  say,  as
              been changed, but  go  no  further  in  the  evaluation  effort    determined   the   content   of   the    optimal    diagnostic
              (like some  annual  reports  in  the  past),  they  imply  that    examination.  The  purpose  of  evaluation  will  then  be  to
              these   changes   will   inevitably   improve   patient   care.    show that physicians  taking  part  in  the  educational  pro-
              However,  there  are  too  many  skeptics  among   politicians,    gram perform an examination  which  is  closer  to  the  ideal
              the general  public,  and  the  health  professions  to  expect    after than before  the  program.  Is  such  a  step  feasible?
              that  such  a  primitive  evaluation,  with  its  possible  but    When  physicians  may  frown  on  taking  a  test  of   knowl-
              still  unproved   assumption,   will   go   unchallenged.   Too    edge  and  attitudes  before   and   after   the   educational
              many  clinics  improve   the   care   of   small   numbers   of    course, they will not rush to  welcome  an  effort  to  assess
              patients who  are  already  under  care,  but  have  no  impact    their   methods   of    diagnosis.    Compromises    may    be
              on the large burden  of  neglected  disease  in  the  surround-    needed,  and  we  may  have  to  monitor  changes  in   groups
              ing  community.  Too  many  health  personnel   may   fail   to    of  health  professionals  rather  than  changes  in  the  ac-
              act  on  new  information,  obtained   in   courses,   or   may    tivities of individuals.
              return  to  environments  in  which  they  cannot  apply  their
              new  knowledge.  Too   many   rehabilitation   personnel   must
              provide  minute  doses  of  advice  or  care  to  their   large                 CHANGES IN END RESULTS
              caseload  of  personnel  and  patients.  All  of  these  relate      Expert  cominittes  have  been   known   to   err   in   the
              both   to   the   EFFECTIVENESS   of   what    we    do    (the    past,  and  a  change  towards  "optimal"  care  may  not  in-
              extent  to  which  we  attain  our  objectives),  and  to   the    evitably improve the health of the recipients of care.  It  is

              190



              therefore,   that   some   RMP's   try   to   show    that     cases   prevented   are   mainly   those   which   will   develop
       essential,                                                            symptomatic   stroke   some   five,   ten,   or   twenty    years
       health status is  raised  when  a  change  in  resources  is  fol                  ct   an   immediate   and   measurable    fall    in
       lowed   by   more   optimal   activities.   The   evaluation   of     later. To expe
       changing  health  status  has  been  reviewed   in   an   earlier     hospitalization rates  or  mortality  for  stroke  is  to  expect
       workshop,  and  it  is  only  too  clear  that  this  effort   is     too  much  of  primary  prevention.  In  its  first  few   years,
       primitive and difficult.  It  seems  likely,  for  example,  that     this  program  must  be  evaluated  in  terms   of   its   inter-
       most  outpatient  care  must  be  evaluated  in  terms   of   ac-     mediate activities  and  short  range  goals,  the  early  detec-
       tivities, since few tangible end results exist.                       tion  and   effective   treatment   of   patients   with   hyper-
       Iease-fatality  ratios  are   high   in   the   acute   stage   of    tension. Primary prevention  is  liable  to  be  wrongly  classi-
       stroke.   An   improvement    in    diagnosis    and    treatment     fied as ineffective if  we  evaluate  it  by  an  immediate  fall
       should  be  reflected  in  lower  death  rates   among   patients     in incidence.
       in general hospitals.  This  will  not  be  the  only  change  in     The   benefits   of   primary   prevention   must   be   balanced
       end  results,  but  it  is  the  change  which  is  most  readily     against  the  costs  involved  in  this  process.  What  must  we
       monitored.  Moreover,  it  is  a  change   which   should   occur     include  amon-  the  costs,  in  addition  to  the  more  obvious
       at the same time as  the  change  in  activities,  and  will  not     steps?  Certainly  we  should  include  the  costs  involved   in
       be  delayed  for  years  after  the  onset,   for   example,   of     diagnosing,  the  false  positives,   the   referrals   who   are
       effective  educational  programs.  We  must  have   a   different     diagnosed  as   normal   by   their   physicians.   Probably   we
       time   perspective   for   programs   of   primary    prevention,     should   include   the   costs   involved   in   diagnosing   and
       however, and I shall discuss this in the next section.                treating  hypertensives  who  do  not   respond   to   care,   or
       In  the  field  of  rehabilitation,  many   measures   exist   to     who respond adversely  to  it.  And  if  we  wish  to  be  strict
       reflect  changes  in  the  physical  status  of  patients.   Most     with  ourselves,  we  should  also  count  against  the   program
       indices are  based  primarily  on  activities  of  daily  living;     the  cost  of  diagnosing  those  who   are   confirmed   to   be
       they   range   from   those   which   describe    a    functional     hypertensive,  but   who   are   given   no   active   treatment;
       profile  of  each  patient  to  those  which  give  one   overall     reassurance,  supervision,  and  periodic  office   visits   have
       score  to  reflect  the  degree  of  impairment.   Most   scoring     no  magical  ability  to  control  the  adverse  effects  of   an
       procedures seem  to  be  repeatable,  but  little  attention  has     elevated blood pressure.
       been  paid  to  their  validity.  The   fact   that   no   single
       method  has  been  used  widely  may  suggest   that   each   has                    COMPARISON GROUPS
       serious  inadequacies.   Nevertheless,   we   cannot   wait   for      If  evaluation   were   partly   a   research   activity,   pro-
       perfection to occur; it is  probably  true  that  any  one  of  a     ducing   new   knowledge   that   can   be   applied   to    many
       number  of  indices  is  better  than  none  at  all,   and   can     similar  situations,  evaluators  would   have   to   insist   on
       contribute  much  to  evaluating  the  end  results  of  rehabgi-
       tative care.                                                          strict  control  groups  with  whom   study   groups   could   be
                                                                             compared.   Evaluation   efforts   have   the   more    practical
              PRIMARY PREVENTION OF STROKE                                   aim,  however,   of   showing   whether   or   not   a   specific
                                                                             endeavor is reaching the  goals  which  have  been  set  for  it.
        Primary   prevention   of   stroke   involves   those   measures     Its  generic  value  has  secondary   importance;   the   evalua-
       taken  to  prevent  the   onset   of   cerebrovascular   disease.     tive  study  does  not  have   to   show   that   other   similar
       From   the   more   distant    viewpoint,    however,    cerebro-     endeavors  are  likely  to  be  effective.  Thus  evaluators   do
       vascular disease is merely  a  part  of  the  natural  course  of     not  feel   compelled   to   use   the   rigorous   methods   and
       hypertensive   and   atherosclerotic   cardiovascular    disease.     strict  controls  of   those   involved   in   experimental   re-
       Since  these  conditions  begin  at  a  young   age,   preventive     search.
       measures  before   onset   are   difficult   to   institute.   In       Nevertheless,   evaluators   must    show    that    activities
       practice,  therefore,  what  we  label  primary   prevention   is     change  and  end  results  improve   because   of   the   program
       the   taking   of    preventive    measures    before    symptoms     being  evaluated,  and  not  because  of  an  artifact  occurring
       begin of cerebrovascular disease.                                     throughout   the   region.    The    evaluation    effort    must
         Probably   the   technic   with   the   strongest    scientific     usually involve, therefore,  a  facility  or  group  of  patients
       support  is  the  early  detection  and   active   treatment   of     which  have   not   received   the   service   being   evaluated.
       hypertensive  disease.  How  should  we   proceed   to   evaluate     Such  a  comparison  group   need   not   resemble   the   treat-
       this  effort?  We  must  first  form  the  realistic  perspective     ment  group  so  closely  as  it  must   in   an   experiment.@It
       that  primary  prevention  is   a   long-term   investment.   The     must  be  similar  enough,  however,  to  be   exposed   to   the
                                                                                                                                             191





               same   extraneous   factors   which    could    produce    the      physician takes an intensive  two-day  course;  the  nurse'
               changes  under  study.  "Before  and  after"  studies   become      has three weeks of training; and  the  physical  therapist'
               much  more  successful  evaluation  efforts  when  they   @how      has two. weeks.  Af  of  September  1970,  seventeen  teams
               that the  change  occurred  only  in  the  group  under  study      plus  selected  guests,  have  been  trained  in   Memorial
               and did not  occur  in  a  somewhat.  similar  group,  perhaps      Hospital of Long Beach.
               located in a different institution or community.                    The  medical  faculty  to  train  these  stroke  teams  in-
                                                                                   cludes specialists in all aspects of  the  stroke  problem.
                                    CONCLUSION                                     'he paramedical faculty includes all  standard  rehabflita.
                 To seek a graceful end,  perhaps  I  should  tell  you  that      tion disciplines, i.e.,  physical  therapist,  occupational
               around  160   A.D.   the   Roman   emperor   Marcus   Aurelius      therapist, nurses,  speech  therapist  and  social  service
               gave  this  advice:  "Thou  hast  embarked,  thou   has   made      workers.
               the voyage,  thou  are  come  to  shore;  get  out."  At  that      After  completion  of  the  training   program   the   core
               time, sailors  feared  to  test  the  effectiveness  of  their      returns  to  its  own  institution  to  utilize  the   team
               navigational  efforts  by  jumping   ashore   promptly.   They      approach  and  to  train  fellow  workers  in  the  method-
               knew ordy too  well  the  uncertainties  and  errors  involved      ology. As a result of this  experience,  the  team  members
               in sailing  in  those  early  years,  and  feared  the  unpre-      have  improved  not  only  their  own  expertise  but  also
               dictable  welcome   that   might   greet   them   on   foreign      their awareness of the techniques of the other  disciplines
               shores.                                                             in dealing with stroke problems.
                 In  the  1970's,  we  may  still  expect  some  voyagers  in      The  stroke  team  training   divides   stroke   care   and
               the  ships  of  stroke  programs  and  rehabilitation  to   be      rehabilitation into three phases. The first phase, Phase 1,
               slow to leave their  vessels  for  fear  that  they  may  have      provides the supportive care to the patient until his vital
               reached  wrong  and  hostile  shores;  even  more   reluctance      signs have become stabilized. This includes  passive  range
               to evaluate the  situation  may  stem  from  doubts  that  the      of motion  exercises,  proper  positioning  and  meticulous
               vessel  has  actually  left  the  port  of  embarcation;   and      skin care. The second phase, Phase 11, consists of a multi-
               perhaps  most  reluctance  to  assess   progress   will   stem      phasic  patient  evaluation  and   implementation   of   an
               from  realizing  that  it  takes  more  than  a   brisk   jump      active  rehabilitation  regimen  designed   to   meet   the
               ashore  to   determine   whether   we   have   or   have   not      individual's specific needs. The last level of care,  Phase
               reached our goals.                                                  111, essentially is a continuation of the second phase, but
                                                                                   emphasis is  placed  on  the  post-hospital  needs  of  the
                                                                                   stroke victim.
                         An Evaluation of a Stroke Program                         The  nurse-coordinator  is  the   catalytic   agent   among
                                     in California                                 the various modalities in the stroke team. She  visits  and
                           BERTRAM L. TESMAN, M.D.                                 assesses each new patient in her facility, initiates Phase I
                                                                                   at the  physician's  request,  assists  in  developing  the
                                                                                   patient care plan  with  the  attending  staff  and  demon-
                Area  VIII  of   the   California   Regional   Medical   Pro-      strates proper care techniques  when  indicated.  In  addi-
            grams  consists  of  Orange  County  and,   for   this   specific      tion  she  is  prepared  to  complete   forms   which   are
            program,   Long   Beach.   This   area   incorporates    approxi-      intended to elicit data for the  stroke  registry  in  Area
            mately  two  million  people  and  includes   35   acute   hospi-      Vill.
            tals  and  approximately  7S   extended   care   facflities.   To      The  physician  is   the   medical   coordinator   of   the
            promote  effective  treatment  of   patients   with   stroke,   a      stroke team who is  responsible  for  leading  the  patient
            training  program  has  beeriset  up  at  Memorial  Hospital   of      care conferences. He serves as  moderator  at  staff  meet-
            Long  Beach.   Although   all   disciplines   of   rehabilitation      ings when stroke data at his particular  hospital  are  re-
            ideally  are  involved  in  stroke,  the  basic   core   of   the      viewed and analyzed. He will  be  available  for  consulta-
            stroke  team  concept  as   implemented   in   Area   VIII   con-      tion about the team approach to  care  of  stroke  patients
            sists  of  physician  niirse-coordinator  and   physical   thera-      for other members of the medical staff at his facility if it
            pist. Each  hospital  in  the  Area  is  invited  to  send  these      is requested.
            three  members  of  the  health   team   to   Memorial   Hospital      The physical therapist  is  responsible  for  a  continuing
            of  Long  Beach  to  take  special   stroke   training;   back-up      assessment of all the stroke patients in the  hospital  and
            teams  also  can  be  trained.   Hospital   administrators   also      he helps establish their  active  rehabilitation  programs.
            are   encouraged   to   attend   the   training   session.    The      He also is available to all staff members for consultation.

            192



            Presently,   one    year    after    the    team    trainin"    was monthly   basis,   the   personnel   from   the    surrounding
            initiated,  an  assessment  of  the  stroke  teams  in  Area   Vill extended care facilities will be invited.  It  is  hoped  that
            reveals  that  only  one  hospital  has  an  active   proaram.   We personnel  in  the  facilities  will  become  more  aware   of
            would  like  to  discuss  some  of  the  difficulties  and   obsta- complete  stroke  rehabilitation  and  also  that  the  physi-
            cles  we  have  identified  as  a  result  of  the  evaluation  and cians on the staff of the acute  hospital  will  become  coc,-
                                                                                                                                          C,
            we  also  would  like  to   discuss   our   resultant   plans   for nizant of those extended care  facilities  which  are  willing
            increasing  the  number  of  effective   stroke   teams   in   this to cooperate in  giving  better  care  to  their  patients  on
            Area.                                                             discharge from the acute hospital.
            The    problems    we     confronted     when     attempting     to We  also  hope  to  develop  a  mobile  van  unit  which  will
            initiate   the   program   were   numerous.   One   year    elapsed transport  a  stroke  teat'n  to  the  various  extended  care
            between   the   time   Area   VIII   submitted   the   grant    and facilities  in  our  community  in  an  attempt  to  introduce
            funds  finally   were   available.   This   posed   a   recruitment the phases of  rehabilitation  that  we  have  been  teaching.
            problem   for   us.   Although   I   had   visited   every    acute We hope that this demonstration  pilot  project  may  seIrve
            hospital  in   this   Area   and   discussed   the   prouram   with as a  model  for  other  communities  to  augment  rehabflita-
            administrators,  by  the  time  the   project   was   funded   many tion care where it is not available.
            changes  had  occurred  in  all   levels   of   personnel.   There- In  addition,  we   have   instituted   a   stroke   volunteer
            fore,  most  of  the  commitments  for  placing   staff   in   tile training  prooram.  Ten   volunteers   have   begun   a   two-
            training  program  were   no   longer   valid   when   the   course month  intensive  training  program  utilizing   a   carefully
            actually  began;  so,  again,  we  have  to  begin  a   recruitment selected  faculty  representing  all  disciplines  of   stroke
            program.  Moreover,  as  a  result  of   the   change   in   fiscal rehabilitation. These volunteers  will  function  in  a  capa-
            policy  in   Medical   and   Medicare   fundinc,,   there   was   acity to aid in  the  resocialization  of  the  stroke  patient
            marked  curtailment  of   available   monies   to   extended   care and, whenever pos sible, will assist him  in  his  rehabflita-
            facilities  in  our  Area.  This  not  only  makes   it   econon-d- tion program under  the  guidance  of  the  special  therapist
            cally  impossible  for  them  to  send  staff  for   an   extensive following the patient's discharge from the hospital.
            training  program,  but  also  limits  their  ability  to   provide In   1969   the   Collaborative   Community   Stroke    Survey
            optimum rehabilitation in their own facility.                     was  begun   in   seven   counties   throughout   the   United
            Although   the   team   concept   in    rehabilitation    is    not States in  an  attempt  to  gather  pertinent  epidemiological
            totally new to  the  field  of  medicine,  it  is  a  new  approach data   concerning   stroke   throughout   our   country    and
            in  many  of  the  hospitals  in  this   Area.   Because   of   the compare  various  separate   areas.   Orange   County   became
            emphasis  placed  on   the   active   involvement   of   all   team involved with this study and we  hope  to  use  this  data  to
            members  some  of   the   physicians   reacted   to   the   program help  us   evaluate   our   stroke   program   concepts.   The
            with  diffidence.  Also,   many   of   the   nurses   felt   uncom- mobile van team also will  be  recording  their  efforts  with
            fortable  about  suggesting  the  proper  level  of  care  to   the patients and comparing  them  with  a  control  group  to  see
            doctor as the patient's physical needs changed.                   if a coordinated  team  can  aid  and  improve  rehabilitation
            Analyzing  all  of  these   difficulties,   we   believe   we   now care in extended care facilities.
            have  some  practical  solutions.  First,   a   follow-up   faculty We  shall  begin  a  follow-up  study  on   stroke   patients
            is   being   organized   to   aid   and   supervise   the   already this  Fall  utilizing  a  form  which  was  developed   by   a
            trained   stroke   teams   in   their   own   institutions.    This committee  of  members  of  all  health  disciplines  involved
            follow-up  team  will  consist   of   a   nurse   and   appropriate in  the  delivery  of  comprehensive  stroke  care.   It   was
            therapists  to  aid  and  help  organize  the   individual   stroke designed  to  extract  the  following  kinds  of  information:
            teams within the hospitals. They will   remain    in    an    acute the patient's functional  condition,  types  of  medical  care
            hospital   for   approximately   two   to   three   months    until and  rehabilitation  being  rendered,  social   and   economic
            the   training   of   all   personnel   has   been    accomplished, conditions, special needs  of  the  patient  and  his  family.
            team    conferences    and    other    aspects    of    the    team Follow-up  visits  will  be  made  by  public  health   nurses
            approach    are    underway    and    the    total    team    feels from  the  Visiting  Nurse  Association   of   Orange   County
            confident  in  their  activities.  They  also  will   discuss   the from  a  random  sampling  of  stroke  patients   six   months
            entire  program  at  staff  meetings  to  orient   the   physicians after their  episodes,  then  again  at  twelve  and  eighteen
            in  the  new  rehabilitative  techniques.  In  this  way  we   hope months.
            to  stimulate  the  physicians  as  well  as  the   hospital   per-  It is our feeling that the  level  of  acute  care  to  the
            sonnel  to  institute  the  team  approach  to  stroke   care.   At stroke patient has improved in our Area as  a  result  of  the
            the   conferences,   which   will   be   on   a   weekly   or   bi- stroke  team  training.  However,  we  have  also  made   many
                                                                                                                                           193




                     mistakes in attempting the  introduction  of  the  stroke           cation,   Discharge   Planning   and   Follow-up,   Area
                     team as we have designed it. An analysis of our work  has           Resources   Development,    and    Public    Education.)
                     given us approaches to solving problems relating  to  the           2 .Development  of  a  Basic  Training  Course  for  Stroke
                     stroke team. The assessment also has helped us  seek  new           Teams  and  of  an  In-Service  Education  Program   for
                     and  innovative  methods  of  meeting  the   health   and           other  professional  health  personnel  of  the  conunu-
                     rehabilitation needs of the  stroke  patient  beyond  the           nity.
                     wafls of the acute care facility.                                   3. Development   of   an   Ih-Service   Training    Program
                                                                                         for  paramedical   personnel   to   make   them   knowl.
                                                                                         edgeable in rehabilitative techniques.
                     North Carolina Comprehensive Stroke Program                         4. Development   of   guidelines   (organizational,   medi-
                                 B. LIONEL TRUSCOTT, M.D.                                cal, nursing, and rehabilitative)
                                                                                         5 .Coordination  with  State  Board  of   Health   Physical
                     OBJECTIVE                                                           Therapy  Consultants  and  with  Medical   Centers   for
                                                                                         consultative support to the community-
                     To  offer  the  stroke  patient  increased  opportunities           6 .Development  of  a  system  to   identify   the   accom.
                     for early diagnosis and treatment,  early  hospital  dis-           plishments,  problems,   and   breakdowns.   (hospitali-
                     charge,  and  continued  follow-up  through  a  community           zation  forms,  discharge  planning   forms,   follow-up
                     stroke program.                                                     reports, etc.)
                                                                                         7. Computeri zation  of  appropriate  data  and   retrieval
                     Development of the Program                                          for feed-back to community health personnel.
                     Identi
                        .fication  of  Subobjectives.   The   objective   must           8 .Determination  of  qualifications  and  procedures   for
                     be reached as a result  of  accomplishing  subobjectives,           obtaining a local, part-time secretary.
                     and these must be (a) realistic within the limitations of
                     personnel and time  of  the  average  community  hospital           Program  Design  It  was  not  considered  feasible   to
                     and the area it serves, and (b) subject  to  measurement.   involve  each  community  in  the  planning  process   of   such
                     The major subobjectives thus identified were:               a   complex   program.   In   consultation    with    practicing
                     I .A    community    health    team    for     comprehensive physicians  and  resource  personnel  from   the   three   medi-
                     management  of  the  stroke   patient:   from   diagnosis   cal.  centers  and  the  State  Board  of  Health,  the  Project
                     through follow-up.                                          Staff  accomplished  the  above  activities.  To   ensure   that
                     2. Professional  health  personnel  knowledgeable   in   the all  necessary  steps  were  completed   in   correct   sequence
                     most   advanced   methods   of   diagnosis   and   treat-   for   maximum   efficiency,   a   time-sequential   work    plan
                     ment of stroke.                                             was   developed   according   to    the    Program    Evaluation
                     3. Increased   availability   of   manpower    trained    in Review Technique (PERT).
                     rehabilitative techniques.
                     4. Guidelines  for  high  quality,  uniform,  total  manage- Establishment of a Community Stroke Program
                     ment of the patient.
                     S. Consultative   support   for   communities   lacking   in        I .Community   Approval.   (a)   The   aims   and    proce-
                     specialized personnel.                                              dures  of  the  Program  are  explained  to  a  few  in-
                     6. An  evaluation  mechanism   to   determine   the   extent        terested physicians. (b)  The  interested  physician  or
                     to  which  the  subobjectives  and  activities  had  been           physicians  appoint  an  ad   hoc   Steering   Committee
                     achieved.                                                           representing all  deliverers  of  health  care;  Project
                     7. Feedback   of   data   to   community,   for    measuring        Staff describes  details  and  responsibilities  in  the
                     impact of program and identifying needs.                            local  program.  (c)  A  permanent  Local  Stroke   Pro-
                     8. Part-time Executive Secretary to administer all                  gram  Committee  is  formed,  and   chairmen   of   Sub-
                     activities.                                                         committees  appointed.   (d)   Members   of   In-Service
                     Activities. The activities  to  accomplish  each  of  the           Education    Subcommittee    ("Stroke     Team")     are
              above subobjectives were:                                                  selected by the Program Committee.
                     1. Development  of  an  organizational  framework  for  a     2.  Educatioti  and  Training.  (a)  Stroke  Team  attends   a
                     community   stroke   program,   with   clearly    defined           4-day Basic Training.  Course.  (b)  Project  Staff  and
                     areas   of   responsibility:   Local    Stroke    Program           Consultants    conduct    two    In-Service    Education
                     Committee   with    Subcommittees    (In-Ser-vice    Edu-           sessions  (2  hours  each)  for  community   physicians,

                   194






           and  nurses.  (c)  Project  Staff   and   Consultants   con-     Computerization and Retrieval of Data
           duct  5-6  practical  sessions   in   rehabilitative   tech-
           niques    (positioning,    transfer,     ambulation)     for     Feed-Back to Community
           nurses    and    physical    therapists.    (d)    Community        1. Periodic visits
           hospital  nursing  staff,  with   aid   of   training   aids        2. Annual Workshop
           loaned  by  Program   and   help   of   Project   Staff   as
           needed,  conduct  34   practical   sessions   for   licensed     Summary of Results
           practical  nurses,  aides   and   orderlies.   (e)   Project        Improvement  of,  and   accessibility   to   the   health   de-
           Staff   helps    plan    periodic    continuing    education
           sessions.                                                        livery  system  is  apparent   in   the   following   brief   sum-
                                                                            mary:
    Implementation of Community Stroke Program                                 1.   Community   Stroke   Progiams   presently    involve    22
       Admission   of    Patient                                                  hospitals  and  8  nursing  homes,   with   follow-up   con-
       1. Nurse notifies Secretary                                                ducted   by   19    county    health    departments.    Over
       2.  Secretary  notifies:   Project   Staff   and   State   Board           915,000  people  reside   in   the   counties   with   local
           of Health Physical Therapy Consultant.                                 stroke programs.
       Evaluation and initial orders                                           2.   Education,   training,   and   more   effective   use   of
       1.  Nurse  and  physician  record   admission   clinical   data,           manpower   participating   in   local    programs:
           consultation   and   laboratory   requests   on   form   lb.              M.    D.......................                125
                                                                                     R.    N......................                 390
       2. Physician writes Stroke Admission Orders.
                                                                                     P.  H.  N.               ...................  103
       Treatment       ofPatient
                                                                                     P.    T.               .....................    18
       1. Guidelines of Management followed.                                         L. P. N.s and Aides           .............   314
       Discharge Planning Conference                                                 Others                  ...................     55
       1.   Secretary   notifies   Project   Staff    and    Conference                    Total              ...................  1,005
           members   of    date.
       2.    Conference    held.                                               3.  Altered  and  improved   patterns   of   care   are   indi-
       3. Copy of Discharge Plan sent to Project Staff.                           cated   by   gradually   increasing   precision   and   com-
       PatientDischarged                                                          pleteness   of   clinical   and    laboratory    evaluation,
       1.  Secretary  notifies  Project   Staff   of   date   of   dis-           institution  of   early   rehabilitation,   more   organized
           charge and of first follow-up.                                         discharge    planning,    and    systematic    post-hospital
       2.  Forms  la   and   lb   completed   and   sent   to   Project           follow-up.  Some  pertinent  facts,   from   the   hospital-
           Staff.                                                                 ization   forms   used   in   this    program,    illustrate
       Follow-Up                                                                  changes  after  the  start  of  a  local   program:   (These
       1.  Project  Staff  and  physician   receive   follow-up   eval-           figures  are   based   on   122   pre-stroke   program   and
           uation reports.                                                        145 post-stroke program patients.)
    'ome Features of Evaluation                                                                                Pre-stroke      Post-stroke
                                                                                                               -pro,ram          program
       Basic Training Course: Evaluation by participants                       Patient evaluation                cohort            cohort
       In-Serpice      Education
       1. Evaluation by participants.                                          1. Blood pressure       ......    71%               96%
       2. Pre-and post-session testing.                                        2. Type and speed of
                                                                                     onset   ..........          7 6T,,            88%
       Hospitalization      Data                                               3. Side, severity of
                                                                                     weakness        ........    59%               72%
       1 . Date   of   admission                                               4. Functional ability     ....    46%               63%
       2.  Clinical and administrative data
           (Hospitalization Forms I a and I b)                                 Use of Multitests
       3.  Date of Discharge Planning
       4.  Discharge Plan                                                      1. Electrocardiogram      ....    27%               51%
       5.  Date   of   Discharge                                               2. F.B.S./2 hr. p.p.
                                                                                     sugar   ..........          39%               63%
       6.  Date of first follow-up                                             3. Other (skull x-ray,
       Follow-up Date.' Periodic follow-up reports                                   etc.)   ..........          18%               27%

                                                                                                                                           195





                                                                                             Measurement of Health Status (side and'se                 ty
                                                        Pre-stroke       Post-stroke                                                              veri      0
                                                         program          program         weakness,  functional   abilities,   etc.)   at   admission,   di'$
                         Patient evaluation               cohort            cohort        charge, and at 3-month intervals thereafter is prese
                                                                                          available on approximately 200 patients treated acc
                                                                                          ing   to   the   Guidelines   of   Management.   These   data   art
                                                                                          now being retrieved for evaluation.
                       Treatment                                                             Reduction    of    hospitalization    costs.    Comparison    of
                       1. Stroke admission                                                pre-stroke    program    cohorts    with    post-stroke     program
                              orders        ..........       7%             71%           cohorts  indicate  that  the  latter  have   a   reduced   hospital
                       2. Rehabflitation begun                                            stay  of  over  4   days   (approximately   $200   less   per   pa.
                              within 48 hrs. after                                        tient).
                              admission       ........       0%             22%

                       Mortality within  48  hours          24%             16%           FuturePlans
                       Discharge planning done   . . .      49%             61%              1.   Consolidating   gains   of    participating    communities.
                                                                                             2. Stroke Prevention and Surveillance.
                       Scheduled,  follow-up  care                                           3.   Training   additional    manpower    through    new    pro.
                          to date           ..........     None           loopts.               grams.



































               196









                                       EVALUATION OF CANCER REGISTRIES


                                                                 Participants

   Charles R. Key, M.D. - iNf oderator                                George Linden
   Assistant Director for Cancer                                      Chief, California Tumor Registry
   New Mexico Regional Medical Program                                State Department of Health


                                                                      Abraham Ringel
                                                                      Public Health Analyst, Operations
   Charles R. Smart, M.D.                                                   Research and Systems Analysis
   Director, Intermountain Tumor Registry                             Regional Medical Programs Service




            Use and Evaluation of Cancer Registries                   value.  Most  physicians  and  other  users  of  registry  data
                      ABRAHAM RINGEL                                  do  not  have  the  time  or  background  to  evaluate  statis-
                                                                      tical data.

                                                                            A subsidiary value of a cancer registry is its effect  in
      Service-oriented  cancer  registries  are   organized   and     the  preparation  of  complete  and  accurate  medical  charts.
   operated  primarily  to  assist  physicians  and  patients  in     One  way  to  measure  this  would  be  to  evaluate  the  com-
   the care of the latter. This  is  accomplished  most  directly     pleteness and accuracy  of  various  items  in  medical  charts
   with  periodic  letters  to  physicians,  and  sometimes  also     prior to the initiation of the registry,  with  medical  charts
   to patients (with the  physician's  consent)  to  ensure  rou-     completed  after  the  registry  was   organized.   Comparisons
   tine surveillance of the disease. Thus,  one  measure  of  the     of  the  information  recorded  concerning  diagnosis,   extent
   effectiveness of a registry is the  increasing  percentage  of     of disease, pathology,  and  therapy  for  the  same  sites  in
   successful  medical  follow-up  of  patients  over  time.  The     the  two  periods  might  show  significant  changes  for   the
   advantages of medical follow-up  are  also  reflected  in  the     better.
   increased  diagnosis  of   additional   primary   malignancies           Examples of  measures  to  detern-dne  the  effectiveness
   and recurrent cancers in the  early  stages  of  the  disease.
                                                                      of cancer registry programs are:
      Additional  services  may  take  the   form   of   periodic           I .Improvements  in  the  medical  follow-up   of   patients
   comparative  reports  to  physicians  to  evaluate  the  diag-           in each of the participating hospitals;
   nosis  and  management  of  cancer  in   the   community   and           2. Improvements  in   the   proportion   of   cases   micro-
   in  the  separate  hospitals.  Patient  information  by   age,           scopically  confirmed  in  the  participating  hospitals;
   race,  and  sex  by  cancer  -site  and  histologic  type,  by           3. Improvements  reflected  in  the  earlier  diagnosis   of
   extent  of  disease  (stage),  methods  of  diagnosis,  treat-           cases by anatomic site;
   ment  modalities,  and   survival   may   lead   to   improved           4. Changes  in  the  length  and/or  quality  of   survival,
   understanding  and  management   of   the   disease   in   the           by  age,  sex,  race,   and   socioeconomic   group   for
   community. For example,       'the  data  collected   by   the           each type of cancer;
   registry may be used to  determine  the  trend  in  the  diag-           S. Improvements  in  the  completeness   of   reporting   by
   nosis and survival of patients with various  sites  of,cancer.           participating hospitals;
   This information may  also  be  helpful  to  hospital  adn-dn-           6. Improvements   in   the   completeness    and    accuracy
   istrators  in  the  development  of  strategies  for   optimum           of abstracted cancer cases (quality control);
   operation of their  institutions,  as  well  as  to  community           7. The  schedule  of  participation  and   compliance   with
   planners  to  determine  priorities  and  the  allocation   of           agreed  upon  procedures  and  definitions  by   partici-
   resources for facilities, equipment, and manpower.                       pating hospitals;
                                                                            8. The utility and value  of  the  central   registry   in
      However,  it  must.  be  emphasized  that  statistical  re-           intramural and    community    programs    of     profes-
   ,,Ports  without  analysis  and  interpretation  have   little           sional and public education.

                                                                                                                                  197





                         Alternative Methodologies for Evaluation                   of cancer  patients  and  that  this  increased  survival  would'
                                         of Registries                              not  have  occurred  without  the  activities  of   the   central
                                     GEORGE LINDEN                                  cancer registry? I can assure you that  this  is  a  very  diffi.
                                                                                    cult hypothesis to prove directly and conclusively.
                   Let me  first  express  my  appreciation  for  being  invited       This   does   not   mean,,   however,   that   evaluation   is
                to  participate  in  this  Regional  Medical   Program   special    ii-npossible;   cancer   registries   can   be    evaluated    by
                session  on  evaluation  of  cancer  registries.  Let  me   also    dropping  down  to  a  lower  order  of  evaluation.  There   are
                make clear  that  I  am  not  in  any  way  an  expert  in  pro-    many  areas  in  our  work  and   personal   lives   where   com.
                gram  evaluation.  I  am  here  today  because   of   my   back-    plete  scientific  proof  of  a  given   hypothesis   cannot   be
                ground   and   experience   in   the   organization,   operation    obtained  but  where  the   preponderance   of   evidence   leads
                and  use  of  a  central  cancer  registry.  My  first  impulse,    us to what we regard as        a    reasonable'conclusion     and
                when  Mr.  Ringel  invited  me  to  participate,  was  to   back    action can be  taken.  on  the  basis  of  that  conclusion.  For
                off as fast as I could; all  I  could  think  of  was  "I  don't    example-one of the goals       of  most  central  cancer   regis.
                know  how  to  evaluate  cancer  registries."  But  Mr.   Ringel    tries is to provide data and   information  that  is  useful   to
                is  very  persuasive.  He  accepted  my   statement   and   then    the medical community in       its  cancer  educational   activi-
                went on from  there  to  convince  me  to  participate  in  this    ties.  These  data  provide  a  resource  for  the  physician  in
                session.                                                            describing  and  analyzing  his  experience  or  his   hospital's
                   Evaluation itself  is  not  new  to  me.  My  training  as  a    experience and can also be used  as  a  basis  for  clinical  and
                statistician and  my  position  as  Chief  of  California  Tumor    other  studies.  I  would  not  want  to  take  on  the  task  of
                Registry  for  more  than  fifteen  years  have  forced  me   to    proving conclusively and  scientifically  that  the  use  of  the
                be   continuously   aware   of   the   problem   of   evaluating    data  for  medical  education  did  in  fact  assist  the  cancer
                what  I  was  doing  or  attempting  to  do.  Most  of  it   has    patient.  Conversly,  however,  there  would   be   little   dis.
                been    informal-the    one     formal     evaluation     having    agreement   with   the    assumption    that    the    continuing
                occurred  when  I  first  joined   the   Registry   staff.   The    education  of  physicians  who  are   diagnosing   and   treating
                Registry,  which   had   been   operating   for   seven   years,    cancer patients  will  help  the  patients  with  cancer.  If  we
                underwent  a  thorough  evaluation  of  its   activities.   This    can  accept  this  as  a  reasonable  assumption,  we  can   then
                resulted  in   the   deletion   of   many   items   which   were    say that one of the goals of  a  cancer  registry  is  being  met
                originally  thought  to  be  "nice  to  know  about"  and   some    when  we  provide  physicians   with   these   data.   The   next
                which  were  important  but  not   obtainable   and   also   in-    step is to decide whether  the  registry  is  in  fact  providing
                volved  some  basic  changes  in  procedures  which   made   the    such  data,  and  here  we  are  on  much   flrmer   ground.   We
                Registry more efficient and  better  able  to  meet  its  goals.    can  review  the  activities  of  the  registry   and   determine
                   Our  purpose  here  is  to  discuss   means   of   evaluating    whether  the  registry  has  or  has  not  provided  such  infor-
                cancer  registries  which  have  been  developed  as   part   of    mation  for  the  use  of  the  medical  community.  We  can   go
                the  Regional  Medical  Program  activities.  Any   such   eval-    one  step  further  and  try  to  determine   whether   in   fact
                                                                                    these  data  are  being  used  and  how  they  are   being   used
                uation must,  of  course,  go  back  to  the  purposes  and  the
                goals for which the  registries  were  established.  These  will    by  the  medical,  hospital,   and   public   health   community.
                differ  among  the  various   operations   and   each   registry       Another   example   of   the   evaluation    of    a    cancer
                will  have  to  be  evaluated  in  terms  of  its  own   precise    registry on what  I  call  a  secondary  level  has  to  do  with
                purposes  and  goals.  There  is,   however,   a   common   goal    the  following:  Can   we   prove   conclusively   that   medical
                that underlies the activities of all cancer registries  and  all    follow-up  increases  the  survival  of   the   cancer   patient?
                cancer  programs  and  that  is  the  benefit  to   the   cancer    Obtaining  such  proof  may  be  possible  (it  certainly   would
                patient.  The  primary   question   therefore   becomes:   "What    be  difficult  to  do)  and  I've  heard  the   statement   chal-
                effect  does  the  registry  have  on  the   cancer   patients?"    lenged. I think I  am  a  reasonable  person.  I  think  it  rea-
                Since the survival of  cancer  patients  is  usually  the  focus    sonable  that  medical  follow-up   of   cancer   patients   will
                of  our  measurements,  the  question  can   be   narrowed   to:    result in longer survival  than  the  survival  of  patients  who
                "What  has  the  registry  done  to  improve  the  survival   of    receive  no  medical  follow-up  after  their  first  course   of
                cancer patients?"                                                   treatment.  I  am  willing  to   accept   this   assumption   and
                  It  is  precisely  this  question  which  led  to  my  initial    therefore  would   accept   an   increase   in   the   level   of
                reaction of pulling  back  and  saying  I  didn't  know  how  to    medical  follow-up  of  cancer  patients  as  evidence  that  the
                evaluate cancer  registries.  Can  we  prove  that  the  activi-    registry  activity   had'been   beneficial   (being   also   hard
                ties of the registry actually  led  to  the  increased  survival    headed,  I  would  want  to  see   evidence   showing   that   it

                198





    was the activity  of  the  registry  which  had  increased  the       Is the codiiic, and classification of the vario
                                                                                       C,                                 us   pieces
    level of medical follow-up).                                      of  information  entered  into  the   registry   system   accu-
       We  are  on  much  firmer  ground  when   we   consider   an   rate? Is there any check on  the  quality  of  coding?  Is  the
    item  of  evaluation  which  the  American  College   of   Sur-   data  processing  system  working  as  it   should?   Are   the
    geons is planning  to  introduce  as  a  requirement  in  their   data  being  processed  accurately  and  on   time?   Are   the
    cancer program-that is, the quality of  survival.  I  think  it   computer proalraiiis  for  processing  and  retrieval  of  data
    would be  easier  to  prove  and  certainly  easier  to  accept   functioning  properly?  Can  data  be  obtained   quickly   and
    the  hypothesis  that  continued  medical  follow-up  is  bene-   at minimum cost?
    ficial to the well-being of the cancer patient.                       How current are  the  data?  Are  the  hospitals  reporting
       The point of  these  remarks  then  is  that  while  we  may   cases  early  enough  or  are  they  lagging  behind  in  their
    not  be  able  to  prove  directly  and  conclusively  that   a   abstractiti,-?  How  good  is  the  follow-up  system?  Is   it
    cancer  registry  increases  the  survival  rates   of   cancer   workmen, as originally planned or  are  there  difficulties  in
    patients,  we  can  on  reasonable  grounds   show   that   the   carrying it out?  What  proportion  of  patients  are  actually
    successful operation of a  cancer  registry  does  benefit  the   followed?   What   proportion   of   patients   are    followed
    cancer patient.                                                   medically?
       The  Regional   Medical   Programs   are   a   comparatively       These  are  sonic  of  the  questions  which  you  will  be
    young  operation  and  the  cancer  registries   organized   as   concerned  with  in   evaluating   the   effectiveness   of   a
    part of this program  even  younger.  Most  of  the  registries   registry  operation  during  the  first  organizational  years.
    were  organized  during  the  last  couple  of  years.  It  may       Although  the   accumulation   of   information   on   sur.
    therefore  be  a  little  premature  to  attempt  to   evaluate   vival may take a number of years,  it  is  still  possible  for
    them  in  terms  of  the  final  goals  for  which  they   were   the  registry  to  feed  information  back  to  the  hospitals,
    established. It  may  instead  be  necessary  to  look  at  the   the medical  comnilinity,  and  the  individual  physician  (if
    registries  in  terms  of  their  developmental  goals   during   this is  part  of  the  reporting  process)  during  the  early
    their  early  organizational  years.  It   is   obviously   not   years of the operation.  It  can  fairly  early  provide  basic
    possible to  judge  a  registry  in  terms  of  publication  of   information  on  the   demographic   characteristics   of   the
    five-year end-results if it has  only  been  in  existence  for   cancer patients, their cancers (site, histologic  type,  stage,
    two  years.  The  factor  of   time   therefore   becomes   im-   etc.) and  treatment.  Information  on  stage  of  disease  can
    portant. How long has the  registry  been  established?  If  it   provide  an  estimate  of  the  level  of  early  diagnosis  of
    has  been  in  existence  for  only  one  year  or  two  years,   cancer.  This  can  be   used   to   support   a   program   to
    what goals  were  specified  for  completion  within  that  one   improve the level of early  diagnosis  and  bring  patients  to
    or two year period?                                               treatment earlier.
       Evaluation  will  therefore  probably   be   in   terms   of       I don't want to  exaggerate  the  output  that  a  registry
    technical goals. Is  the  registry  system   itself   organized   can  produce  in  its  early  years.  A  registry's  usefulness
    and  operating?  Does  the  registry  have   properly   trained   increases  with  time  and  the  early  years  are  a  time  of
    staff?  Have  the  details  of  the   operating   system   been   limited output. What  is  most  important  of  course  is  that
    worked  out?  Have  they  been  documented?   Is   the   system   the  community   and   especially   the   physicians   be   in-
    actually  working?  Are  the  various  parts  of  the   system,   formed of the progress of the  registry.  and  be  the  recipi-
    hospitals, physicians, etc., cooperating fully?                   ents of early output  of  information.  This  is  an  important
       Has  the  registry  developed  suitable  forms  for  obtain-   point  for  evaluation.   Has   the   registry   produced   any
    ing the original data  plus  a  handbook  of  instructions  for   data?  If  so,  has  it  been  disseminated  to  the   medical,
    those  who  are  charged   with   obtaining   the   data?   How   hospital and lay communities? How has it been used?
    good  are  the  data  being  entered  into  the  system?   Does       I  would  like  to  take  a  few  moments  to  stress   the
    the abstract or other  form  on  which  the  data  are  entered   documentation  of  activities  carried   out   by   a   central
    accurately reflect the  patient,  his  cancer  and  his  treat-   cancer registry. At  the  beginning  of  my  talk  I  mentioned
    ment?  What  educational  means  are  being  used   to   insure   the  informal   evaluation   which   occurs   almost   continu.
    that the  personnel  in  the  hospitals  abstracting  the  data   ously.   On   occasion   it   becomes   very   immediate    and
    are   trained   and   knowledgeable?   Are   workshops    being   important.  We  were  asked,  several  weeks  ago,  to  provide
    conducted  to  assist  these   people?   Has   a   program   of   documentation regarding  the  value  of  our  activity  to  the
    quality control  of  data  been  instituted?  Is  there  review   Department's  program.  I  was  told,  at  3:30:@P.M.   @on   a
    of  incoming  records  and   independent   abstracting   of   a   Thursday, that the documentation was
    sample of cases to insure a high quality of data?                 before noon on Friday. We have, during C




                of  operation,  developed  material  which  can   be   readily     A  few  words  about  what  one  writer  of  the  material   I
                used  for  documentation.  I  wrote  a  very  short  statement     received in  preparation  for  this  session  called  "dynamic
                regarding  our  program  and  attached  to  it  two   of   the     evaluation".  I  agree  wholeheartedly  that  the   evaluation
                documents  which  we  had  developed.  One  of  these  was   a     process should not be  static.  The  placing  of  a  value  on
                                                                                   any part (or all)  of  the  registry  system  should  be  fol-
                Progress  Report  which  is  compiled  every  six  months.  We
                originally started this to  keep  track  of  and  to  evaluate     lowed by the  inquiry,  "Does  the  evaluation  indicate  that
                our own activities. We have since  found  it  useful  in  many     changes are  necessary  to  improve  the  situation?"  If  so,
                other situations. A copy  of  this  report  is  available  for     the  evaluation  should  at  least  indicate   the   necessary
                observation on  the  table.  The  other  document  I  attached     changes and possibly initiate  action  to  make  the  changes.
                was a  list  of  publications  which  the  Registry  has  pro-     Maybe  we  should  propose  that  a   future   evaluation   be
                dliced. I believe this  kind  of  documentation  is  extremely     made  of  the  effects  of  the  present  evaluation.  Was  it
                important in evaluation of a cancer registry.                      worth the  time  and  effort?  Did  it  really  result  in  an
                 The   Progress   Report   for   January-June    1970    shows     improved registry programs
                the number of cases received during the       first  half   of     What  I  have  said  today  is  certainly  not  exhaustive  in
                1970 and the total as of June 30. It  also  shows  the  status
                of  current  follow-up  efforts,  including  the   number   of     terms of evaluation of cancer  registries,  but  I  hope  that
                cases  in  active  follow-up,  how  many  died,   the   number     the  combination  of   your   own   discussions   on   program
                actually  followed   and   how   many   were   medically   ex-     evaluation  and  our  presentations   and   discussions   here
                aniined. The  report  also  contains  a  detailed  description     will make it possible for you to evaluate  the  activities  of
                of  the  requests  for  data  which  were   completed   during     your own cancer registry program.
                this  six  month  period.  There  were  a  total  of  57  such
                requests and I believe  that  the  listing  of  the  individual
                requests constitutes  evidence  useful  in  evaluation  of  the         Methodologies for Evaluating Effectiveness
                activities of  the  registry.  The  annual  reports  which  we                     and Value of Registries
                provided  to  hospitals  during  this  period  are  also   de-                  CHARLES R. SMART, M.D.
                scribed. There  is  a  section  on  the  Hospital  Data  Books
                which  we  developed  this  year  for  each  of  the  partici-      Incidence   and   Epidemiological   Registries   study    the
                patiiig  hospitals.  The  Data   Books   provide   a   compre-     differences in   geographic,   racial,   religious,   environ-
                hensive  and  clearly  presented   account   of   the   cancer     mental,  social  and  economic  groups   seeking   etiological
                experience of  each  hospital  and  are  a  solid  example  of     factors leading to prevention.
                the usefulness of a central cancer registry.  A  copy  of  the      End  Results   Registries   study   survival   to   determine
                Data Book is also on  the  table.  The  ProIress  Report   also    national  baselines  and  to  monitor   change   in   survival
                                                          9
                includes a  description  of  a  number  of  studies  in  which     rates.
                we were involved during this period,  a  listing  of  two  new       The  Clinical  Cancer  Control  Registry  has  in  the  past
                publications,  a  description  of  the  future  plans  of  the     been  hospital  based  and  patient  oriented,  attempting  to
                Registry,  and  an  account  of  our   activities   with   the     control  cancer   through   encouraging   life-time   interval
                Regional Medical Programs in  California.  It  also  covers  a     follow-through  examinations  on  all  patients   having   had
                proposed  central  cancer  registry  in  Los  Angeles  County;     cancer.  Through  the  regular  follow-up  examination  it  is
                hospital  consultation,  training  and  lectures  carried  out     hoped  that  recurrences  and  second  primary  cancers   will
                by  the  Registry  staff;  the  activities  of   the   Alameda     be discovered  at  a  time  when  they  can  still  be  cured.
                County  population   based   Cancer   Registry-,   work   per-     This type of registry also seeks to  serve  as  a  self-evalu-
                formed  under  contract  with  the  National   Cancer   Insti-     atory  and  educational  mechanism  for  both   the   hospital
                tute's  End  Results  Group;   the   Third   National   Cancer     staff and individual physicians.
                Survey; and a list of  visitors  to  the  Registry.  The  Pro-       While  these  various   registries   are   emphasizing   one
                gress  Report  has  developed  from  very  modest   beginnings     phase of the  problem  or  another,  their  functions  greatly
                to a  very  useful  tool  for  orientation  and  for  documen-     overlap  and  their  goals  can  be   summarized   under   the
                tation of the activities of  the  California  Tumor  Registry.     headings of service, education or research.
                Other   evidence   and   documentation   of    the    Registry        In  October  1965,  Congress  passed  Public  Law   89-239,
                activity is available for your  review  on  the  table;  there     known as Title IX - Education, Research,      .Training,    and
                are also sign-up sheets if you  want  copies  of  any  of  the     demonstrations  in  the  fields  of  Heart  Disease,   Cancer,
                material.                                                          Stroke,   and   Related    Diseases,    encouraging    through

                200




   @rants   the   development   of   cooperative    arrangements        Cancer   Control   depends   upon   both    Physicians    and
   @mong  medical  groups  and  institutions  in  making  avail-     Patients. The  physician  is  busy  and  must  find  the  inter-
   Lble to their patients the latest advances in  the  diagnosis     action  with  the  Cancer  Control  helpful  and  satisfying   -
   ind treatment of these diseases. This bill has given rise  to     the   re-enforcenient    must    be    meaningful.    Knowledge,
   -nany new clinical cancer  control  types  of  registries  to     skills and  attitudes  developed  on  the  part  of  the  physi-
   assist in clarifying the  local  problem  with  solid  facts, .   cian will be transferred  to  better  patient  care.  Evaluation
   :hus  allowing  logical  planning  of  needed  programs   and     feedback   from   the   physician   and   from   patient    care
   assuring a greater continuity  of  re-examination  of  cancer     should be utflized  in  adjusting  the  prograi-n's  methods  or
   ,)atients.   We   shall   concentrate   upon   and   describe     objectives.
   -nethods of evaluation of this type of registry.                     By   modifying   the   evaluation   measurements   used    in
             OBJECTIVES AND EVALUATION                               educational   programs    (attendance,    opinion,    gain    in
                                                                     knowledge,  change   in   behavior),   one   can   develop   the
      Analysis  of  goals  and  objectives  must  precede  eval-     following   parameters   for   the    evaluation    of    cancer
   Liation.                                                          registries:
      I . Decide upon the goals  you  intend  to  reach  at  the        I    .Participation
          end of the program                                            2.   Opinion or Attitude of the physician
      2.  Select  the  procedure,  content,  and  methods  which        3.   Improvement   in   life-time   interval   follow-through
          are relevant to the objectives                                     examinations
      3.  Carry out the program                                         4.   Improvement in patient management
      4.  Measure  or  evaluate  the  performance  according  to        5    .Improvement in patient survival
          the objectives or goals originally selected.
                                                                             EXAMPLE OF PROGRAM EVALUATION
                     PROGRAM DEVELOPMENT                             Utah Tumor Registry
                                                                        In  October  1966,   the   tumor   registry   of   the,lnter-
                                                                     mountain   Regional   Medical   Program   was   formulated    on
                                                                     paper.  The  registry  was  an  integral  part  of   a   compre-
                                                                     hensive  cancer  control  program   involving   clinics,   semi-
                                                                     nars,  telephone  -  radio  -  TV  programs,  etc.  The  general
                                                                     concepts  are  depicted  in  the  diagrams  on   the   following
                                                                     page:
                                                                             The objectives of  the  Cancer  Control  Registry  were:
                                                                             1 .To survey and to establish local baselines.
                                                                             2. To  provide  local  practicing  physicians  with   accu-
                                                                             rate, meaningful feedback.
                                                                             3. To  save  lives  through  the  systematic  follow-up  of
                                                                             all cancer patients.
                                                                             4. To  identify   deficiencies   and   design   operational
             DYNAMICS OF PROGRAM EVALUATION                                  projects accordingly.
                                                                             5. To evaluate operational projects.
                                                                             At first the importance of  No.  3  was  not  completely
                                                                        appreciated. It now heads the list.

                                                                        Methods
                                                                             1. Gain the support  of  the  medical  profession,  hos-
                                                                        pitals, health  department,  cancer  society  and  other  in-
                                                                        terested health agencies.
                                                                             2. Enhance  presently  existing  hospital  tumor  regis-
                                                                        tries by providing:
                                                                             a. Meaningful listings of their patients' data
    Accentuate the positive          Eliminate the negative
     Eliminate the negative      but alone leads to stereotyping             b. Survival reports by site and stage
        (PROGRESSION)                       (STATIC)                                                                                201







                                                             Cancer                                                New          Approach                                            Cancer Education
                                                      Patient   Survival                                                      in                                                     Communication
                                                                                                                 Cancer        Education                                                                          IONAL
                                                            valuati                                                                                                                                               MEDICAL
                                                                                                                                                                                                                            PROGRAMS



                                    Resi                                         rigin  Ca.Pts
                                    Hospital                             T,eatment     Centers
                                    Physician                                Load  &   O.ality   PATIENT SURVIVAL                      EDUCATION
                                                                                       blk
                                    Delay/                             Education<p
                                                                                      P@,.ici.n  TUMOR REGISTRIES                      MANPO.Ell
                                                      <,E:,Iy                                                                                             SELF   YOU-ME   ME-YOU                REMOTE        YOU-@FELLOWMAN
                                    Site, Type, Stage                  Sumi,alCOMpari@n  CO.PUTERANALYSIS                              FACILITIES
                                                                                                      REPORTING                        EQUIP.ENT                                            RADIO TV   V   DEO    APE
                                                                        F-ilili-                                                                                     CONSULTATION
                                                                                                                                       COOPERATIVE
                                                                                                                                       ARRANGEMENT                   CORE TRAINING

                                                                                                                                                                     SEMINARS



                                 c.  Computer  written  follow-up   letters   to   their   physi-                                      hospitals which had registries, data were included going
                            cians  on  all  living  patients  not  reported  within   the   past                                       back to January 1, 1957.
                            year.
                                 d.   Automatic   updating   from   death    certificates,    read-                                              Tumor Registry Report ofaccomplishments
                            mission to other hospitals, etc.                                                                                                       April 1, 1969 - Sept  1,  1970.
                                 e.  Public   Health   Nurse   tracer   on   all   patients   who
                            could  not  be  found  by  the  hospital   tumor   registry   secre-                                                                                                        IRMP &            Six  state
                            tary.                                                                                                                                                                    Utah  registry   registry
                                 3.   Divide.   and   conquer   cancer    by    providing    each
                            physician  with  his  own   patients'   data   as   derived   frorfl                                       New cases                     ................                      7,707               19,134
                            multiple  hospital  registries,  enabling  him   to   evaluate   his                                       Follow up letters           ............                                               49,287
                            own   cancer   practice   -   patient   follow-up,   treatment   and                                       Cases followed up by public nurse                    ...               319
                                                                                                                                           Dead  ....                                        .  .             42
                            survival.  Listings  of  current  medical   references   and   state                                           Referred to physician 'care                .......                 176
                            and  national  survival   rates   are   included   on   the   physi-                                           Lost to  follow  up                  ..........                    73
                            cian's computer report.                                                                                        In 'process                  ..............                        28
                                 4.  Merge  and  analyze  the  data  from   the   entire   state,                                      Patient   listings   sent    to    physicians    .    .  .                                  2,445
                            allowing  planning,   evaluation,   education   and   lost   patient                                       Hospital      print      outs . . . .  .  .  .  .  .  .  .                                    230
                                                                                                                                       Outputs for special research               . .  .  .  .  .                                      61
                            follow-up on a regional basis.                                                                             Outputs for articles in Rocky Mountain
                                 a. Medical society articles                    are  published   in   the   Rocky                          Medical   Journal         ...........                                                       8
                            Mountain Medical Journal                                                                                   Training sessions for registry
                                 b.   Cancer   society   -   developed   rural   cancer    survey                                          workers                    ...............                                                  4
                                                                                                                                       Tumor      registry      handbooks       printed       and
                            clinics as a result                                                                                            distributed                                                                               650
                                 c.  American   College   of   Surgeons   Study   Comn-dttee   on
                            Cancer                                                                                                     Cumulative cases in registries               .  .  .  .  .            40,488               51,915
                                 d. The State Health Department
                                 e.  The  Regional  Medical   Program   -   for   evaluation   of                                          Since   in   evaluation   one   is   primarily   interested   in
                            operational projects.                                                                                      determining   whether    goals    and    objectives    have    been
                                                                                                                                       accomplished  we  will  look  at  the  results  rather   than   how
                                                                                                                                       they were achieved.
                            Evaluation                                                                                                     Objective   No.   1.   To   save   lives   through    systematic
                                 For  the  purposes  of  this-  subject   we   will   deal   only                                      follow-up  of  all  cancer   patients.
                            with   the   Utah   data,   although   the   entire   registry   now                                           1.  How  many   cases   have   been   registered?   Elin-dnating
                            serves  six  states   and   is   known   as   the   Rocky   Mountain                                       benign  tuinors  there  were  28,996  cancers  registered   as   Of
                            States   Cooperative   Tumor   Registry.   Many   of   the   present                                       September   9,   1970   in   the   Utah    Registry.    Eliminating
                            innovations  in  this   registry   were   contributions   from   the                                       further  those  registered  in  more  than   one   hospital,   etc.
                            other  five  states.  WMIE  the  Utah   Registry   did   not   offl-                                       (the  non-analytic  cases),  we  are  left   with   23,183   analyt                             'ic
                            ciafly  e@t   until   April   1967   when   it   received   fundinc,                                       cases  which  were  treated  by  774   physicians   in   44   host-
                            from  the  DRMP,  patient  data   from   all   44   hospitals   were                                       pitals.  This  is  about  2500  new  cases  per  year.  Based   oil
                            collected  back  to   January   1,   1966,   and   in   those   four                                       the  population  of  I  million   people   in   Utah,   one   would

                            202




      expect    2,850    new    cases    per     year,     or     12.3%     more     than      Field of practice                      No.               No.               No.
      were    registered.    No    attempt    has    been     made     to     pick     up                                        Doctors             patients        patientsl
      the     cases     diagnosed     and     treated     in     physicians'     offices,                                                                            Doctor
      which    make    up    a    high    percent    of    all     skin     malignancies.
      lbs    is    evidenced    from    the    fact    that     the     three     leading      Surgery                                138            7,748                55
      dermatologists    in    the    state     each     have     fewer     than     three      G.P.'s     ...........                 252            3,417                14
      cases     recorded.     By     checking     the     death     certificates      and      lnt.     Med ..........                122            3,059                25
      cross      checking       with       the       radiation       therapists       and      Radiation Rx               ......        5            2,676                535
      pathologists    of    the    state    and     considering     the     number     of      OB       GYN .........                 80             1,897                24
      patients   diagnosed    and    treated    out    of    our    state,    it    would      Urology      .........                 23             1,778                74
      appear    that    we    are    mis@-    15    -    20%    of    the    malignancies      Neurology     and      Surg .  .  .    17                563               33
                                                                                               ENT        . . . . . . . . . .  .      26                358               14
      -in   the    Utah    Registry.                                                           Pediatrics   . . . . . .  .  .  .      43                294               7
         2.    What     fields     of     medical     practice     treat     the     most      Orthopedics      . . . .  .  .  .      35                196               6
      cancer patients in the registry?                                                         Opthalmology         . .  .  .  .      26                127               5



      3.     Physicians     Non-Participation      in      Cancer      Patient      Follow-up      Program.

                                                    (Non-participation  =   over   10%   or   5   patients   not   reported   2   yrs.)

                                                                             By    County    of     Residence

                                         Non   /   total                Salt Lake Cty              Weber Cty             Utah Cty              Other

                            G    P's        8  257     (   3%)            2  77                    0      26             2    37               4      117
                            Surg.          14  138     (10%)            11   87                    1      20             0    10               2    21
                            OB-GYN         11  80      (147o)           10   54                    1      14             0     7               0     5
                            Urol,           2  23      (  87o)            0  15                    2      4              O/    2               O/    2
                            Ortho.          2  35      ( 5 -7o)           I  26                    1      5              0 /   3               0     I
                            Neuro.          2  17      (1 ITo)            2  14                    0      3              0 /   0               0     0
                            Opthal.         2  26      (7%)               1  1.3                   0      5              0 /   5               1     3
                            ENT             1  26      (3%)               1  19                    0      3              0 /   2               0     2
                            lnt. Med.       4  122     (3  7o)            2  89                    0      20             2 /   9               0     4
                            Radiation       0     5    (0%)               0     3                  0      I              0 /   0               0     1
                            Ped's           1  43      (2%)               0  25                    0      8              0 /   2               1     8

                                             471772    (7  7o)          30   422        7%)        5      109  5%)       4 /  77      5 'Yo)   8      164   5%)

           Since  the  above  data  depict  the  Non-Participation  Rate   the   overall   participation   rate   according   to   the   criteria   set   forth   is   94%.


        4.     A     telephone     interview     was     carried     out     with     102   the   tumor   registry   were    used    as    a    control;    yet    two    thirds
      physicians'   office   staff    to    try    to    determine    the    manner    in   of    their    secretaries    said    they    knew    nothing    about     it     or
      which    the    follow-up    letters    were    being    used    and    if     they   gave a negative response.
      were    helpful.    The    offices     were     chosen     according     to     the
      size    of    community    and    the    number    of     letters     they     were          S.   In   view   of    the    questionable    validity    of    the     above
      receiving.   23%   of   the    offices    felt    the    letters    were    helpful   study    a    survey    questionnaire     was     distributed     at     a    scien-
      as    a    reminder    of    those    patients    not     returning     for     re-   tific     meeting     of     the      Utah      State      Medical      Association.
      exarwnation.    85%    of     the     letters     were     completed     by     the   Fifty      physicians      were      in      attendance      (some      out-of-state
      physician,    II%    by    the     nurse     and     4%     unknown.     Only     8   guests).       Forty-one       questionnaires       were       completed       show-
      offices   had   a   system   that    would    contact    the    patient    if    he   ing   73%    of    the    physicians    felt    the    registry    had    been    of
      failed     to     keep     an     appointment,     and     most     offices     de-   value   or   assistance   to   them.   Of    the    27%    that    felt    it    had
      Pended    upon    the    patient    to    return    at    the    time    of     his   been   of.no    value    or    assistance,    three    did    not    treat    cancer
      appointment    but    have    no    fafl-safe     mechanism     built     in.     A   patients,    two    had    not    received    any    reports,    two     did     not
      Ilumber    of    physicians     known     to     be     strong     supporters     of  use    their    reports,    two    were    internists,     and     one     physician

                                                                                                                                                                          203







                                                                        Number of Letters per Office


                                                                     1-10            11-49                    50+


                                                               Yes     I             Yes     2           Yes     1
                                               3,000           No       8            No     9             No    3            4/24=17%


                                             3,000 to          Yes      2            Yes    3            Yes    6
                                   E
                                   E         10,000            No       6            No     7            No     5            5/24=21 %
                                   0


                                                               Yes      5            Yes    6            Yes    6
                                             10,000            No     16             No     17           No     9            17/54@31 %


                                                              8/38=21 %              11/40@27%          7/24@29%             24/102=23%
                 saw   no   advantage   in   the   program.   One   wanted   simplifi-    disease.  I  was   amazed   to   see the  large  percentage   of   my
                 cation and remuneration.                                                 patients who have died rather rapidly of this disease."
                     6. Are the individual      physician's   cancer   registry   list-      b. In assessing our  success  or  failure  in  any  venture  it
                 ings  of  his  personal  cases  of  any  value  in  the   saving   of    is  important   to   have   standards   with   which   to   compare
                 patient's lives?                                                         our  performance.  In  the  physician  listings,   medical   refer.
                     a.  These  listings  are  simply  a  compilation  of   the   most    ences   have   been   provided   pointing   out   current   thought
                 significant   data   which   the   registry   has   on   all   of   a    on  the  latest  in  the  diagnosis  and  treatment  of  that  par@
                 doctor's   patients.   It    suppl ements    the     tumor     registry  ticular  disease.  I  have  been  of   the   opinion   that   these
                 follow-up  letters   in   reminding   the   physician   of   patients    articles are little used and are  likely  a  waste  of  effort.  In
                 who   have   not   returned,   but   in   addition   summarizes   his    considering   discontinuing   them   and   utilizing   the    space
                 experience   with   specific   kinds   of   cancer.    A    prominent    for  a  summary  analysis  of  the  states  experience  with   that
                 surgeon  in  Salt   Lake   City   said,   "Before   I   received   my    site  or  for  use  in  communicating   educational   messages   to
                 personal   computer   listing   of   my   cases,    I    could    not    physicians,   a   cry   went   up   from   the    Wyoming    Cancer
                 remember  a   single   case   of   cancer   of   the   stomach   that    Registry.  They  found  that  their   reprint   service   increased
                 was  living.  I  found  that  I  have  five  cases  still  living.  1    300  fold  following  the  distribution  of   physician   listings.
                 operate  with  a  different  attitude   on   patients   with   cancer    The  last  time  we  alternated  every  other   site   with   surn-
                 of  the  stomach   now!"   One   of   the   busiest   urologists   in    maries  of   our   state's   experience   with   that   cancer.   A
                 our  city  said,   "Before   1'received   my   computer   listing   I    questionnaire  in  the  form  of   a   post   card   was   prepared
                 thought   carcinoma   of   the   prostate   was   a   pretty   benign    and  sent  on  September  9th   with   the   physician's   reports.

                                                                        Value Of Reports To Physicians
                                                                           (post card questionnaire)

                                                                            Questionnaire                                    Percent Reporting
                     Type of Practice               Nos. Sent           Nos.  Returned      Percent Returned        Some or Frequent Value           Little
                     GP's   .................            257                  S9                   23%                       74%                     20%
                     Surg   .    .................       138                  42                   31                        90                      11
                     Int.   Med .      .............     122                  38                   31                        57                      39
                     OB-GYN   ...............            80                   is                   19                        so                      20
                     Peds   ..................           43                   10                   23                        so                      40
                     Ortho  .      ...............       35                    4                   11                        100                      0
                     ENT    .................            26                   10                   38                        100                      0
                     Opthal .       ..............       26                    3                   11                        66                      33
                     Urol   .     ................       23                    4                   17                        75                      25
                     Neuro  .      ...............       17                    0                     0                         0                      0
                     Radiol .      ...............         5                   0                     0                         0                      0

                                             TOTAL       77                   185                  24%                       75%                     22%


                 204




       The  questionnaire  indicated  that  82  did  not   use   the     the registry was nothing but  a  bunch  of  busy  work,  until
    references, 58 used  them  once  or  twice,  39  three  or  four     I tried the experiment of  calling  the  patients  I  had  not
    times and 17  over  five  times.  106  desired  the  future  use     seen for  re-examination.  Many  of  those  I  thought  should
    of  both   references   and   site   summaries,   54   summaries     have  been  dead  were  alive  and   many   that   I   thought
    alone  and  8  references  alone,  16  indicated   they   wanted     should be alive had died,  and  some  patients  were  able  to
    neither.                                                             have  further  treatment  and  hopefully  would   be   cured."
       c. The  addition  of  State  and  National  Survival  Figures     He  said,  "The  importance  of  this  program  is  not   sta-
    is  essential  to  the  physicians'  report  in  order  that  he     tistics  but   GOOD   PATIENT   CARE!"   He   encouraged   all
    might have standards  for  comparison  as  well  as  for  devel-     present to try the same experiment.
    oping  a  better  understanding  of  the   natural   course   of       If we accept the  word  of  the  Office  secretaries  before
    that specific disease site and type.  The  latter  will  benefit     discussed,  then  23%  of  offices  receive  sot?ie  help   in
    him  in  future   decisionmaking   regarding   patient   manage-     follow-up from the registry.  If  we  accept  the  opinion  of
    ment. One of  the  problems  in  our  present  reports  is  that     the  physician  questionnaire  at  the  State  Medical   Meet-
    we are cortiparing  absolute  with  relative  survival  -  which     ing, then 73,7o  and  it'helpful.  If  we  look  at  the  per-
    will have to be corrected.                                           centage  of  doctors  participating  in  the  follow-up   pro-
       7.  In  attempting  to   evaluate   (measure)   whetlfer   we     gram then  94%  is  the  figure.  The  real  question  is  how
    are  saving  lives   through   systematic   follow-up   of   all     much do these  letters  help  or  how  much  could  they  help
    cancer  patients,  we  have  been   skirting   the   issue   and     in a diligent conscientious doctor's office?
    measuring  the  methods  rather  than   the   objective.   There       One  can  see  that  even  in  the  most  diligent  doctors'
    are  a  number  of  studies  showing  where   second   primaries     offices cancer patients will be  lost  unless  there  is  some
    are  nearly  as  curable  as  the  primary  and  where  isolated     type of  fafl-safe  technique  which  will  call  to  the  at-
    metastases  have  been  cured  in  as  high  as  25,7o  in  some     tention of the doctor that the patient has  not  been  in.  In
    selected  series   and   where   subsequent   therapy   to   re-     the best offices the number  of  patients  who  will  be  thus
    current  or  metastatic  disease  has  sometimes   resulted   in     called in for re-examination will be in the  range  of  10  to
    cures.  Rather  than  trying   to   document   each   of   these     35%.
    cases  in  the  registry,  it  should  suffice  to   demonstrate      Objective  #2  of  providing   practicing   physicians   with
    that because of  the  registry,  patients  are  now  being  fol-     accurate,  meaningful  feedback   has   been   discussed   and
    lowed   and   re-examined   who   otherwise   would   not    be.     evaluated  through  the  post  card  questionnaire   and   the
    The  Governor  of  the  American  College  of   Surgeons,   Utah     response  for  reprints,  etc.  In  addition  to  the   above,
    Chapter, is a man of  long  experience  in  the  surgery  field.     articles  are  being  published  every  other  month  in   the
    He  thought  the  tumor  registry   was   just   a   statistics-     Rocky   Mountain   Medical   Journal   by   various    medical
    gathering  tool,   for   which   he   had   neither   time   nor     societies.  These  articles  are  attempting  to  answer   the
    patience  and  persistently   threw   all   of   his   follow-up     questions:  where  are  we?  where  should  we  be?  how   can
    letters in  the  waste  basket  as  soon  as  they  arrived.  He     we get there? Sixty-one  requests  have  been  filled  in  the
    agreed to try the experiment  of  pulling  all  of  his  records     past year for special studies for  physicians  and  hospitals.
    and filling  out  the  reports  on  those  he  had  seen  within      Objective  #  3  should  now  be  the  development  of  local
    the  past  year  and  then  calling  the  others  in   for   re-     baselines.   This   has   been   accomplished   through    our
    "an-dnation.  Several  months  later   at   a   surgical   staff     computer  summaries  and  survival  curves   which   are   run
    meeting   he   presented   his   experience    with    carcinoid     every 3  to  6  months  on  the  entire  region  and  on  each
    tumors  of  the  small  bowel.  Following  this   some   figures     state. In addition a special matrix  run  will  summarize  all
    from  the  hospital  tumor  registry  were   referred   to   and     infon-nation  which  we   have   on   the   computer   for   a
    the  doctor  interrupted  the  meeting  to   say,   "I   thought     specific site or for all states grouped together.

                                                   Survey Of Four Physician's Practices

                    Phys.      To ta I Pts      Dead          A@e        Not rep. 2 years No. letters   Pt's called in


                    A   .....    2,588          1,461        1,077          26    2.4%         385        110  (28%)
                    B   .....      709           270           436           1    0.2%         125         12 (109'o)
                    C   ......     244            129          115           1    0.9%          28          6 (2 1 Olo)
                    D   .....      137            45            92           2    2.2%          43         15  (35%)


                                                                                                                                    205





                  Objective #4 of identifying deficiencies  and  designing    first and second most ftequent malignancies in t
                indicated operational projects is in  constant  action.  A    A special investigation of cancers of the lip is I
                cancer of the  head  and  neck  survey  was  conducted  in    being done due to the poor survival rates
                Price, Utah, because of  the  increased  number  of  these    through the registry.
                carcinomas in this area.  Also  a  special  study  by  the       Objective #5 of evaluating operational canc(
                American College of  Surgeons  study  comniittee  in  Utah    is under way to see if specific- malignancies a        ting
                is investigating carcinoma of the stomach in this area due    the best primary and  palliative  therapy.  The  Salt  Lake
                to  increased  incidence  as  identified  by   the   tumor    area is being compared  to  the  Ogden  area  and  both  of
                registry. The American  Cancer  Society  undertook  cancer    these are compared  with  the  Southern  Idaho  area  where
                of the  breast  and  cervix  surveys  (detection  clinics)    we have  put  on  continuous  monthly  cancer  clinics  and
                throughout the rural areas of Utah because these  are  the    seminars.





































                206






                                   EVALUATION OF REGIONAL ADVISORY GROUPS

                                                          Participants
         James P. Harkness, Ph.D. - Moderator                                 Stephen L. Maxwell
         Deputy Coordinator, New Jersey Regional                              Judge of the District Court
         Medical Program                                                      St. Paul, Minnesota and
                                                                              Chairman, NortMands Regional
         David A. Pearson, M.P.H.                                               Advisory Group
         School of Public Health
         Yale University

         Homer Hagedorn, Ph.D.
         Arthur D. Little, Inc.
                                                                              Rhoda Abrams
         Paul E, White, Ph.D.                                                 Assistant Branch Chief, Evaluation
         Division of Behavioral Sciences                                        Branch, Office of Program Planning
         Johns Hopkins School of Hygiene and                                    and Evaluation
         Public Health                                                        Regional Medical Programs Service


                 Regional Advisory Groups as a Factor                           The  second  term  needing   examination   is   the   word
                     in the Regionalization Process                           "factor." As used in the title, it implies that the  RAG  is
                      DAVID A. PEARSON, Ph.D.                                 but  one  of  various  elements  which  combine  to  promote
                                                                              or  retard  regionalization.  Obviously,  there  are   other
         It is important to  examine  three  key  phrases  used  in           factors of an  extrinsic  and  intrinsic  nature.  A  decade
         the title of this paper. The first is  "regional  advisory           ago,  Masur  wrote  about  the  regionalization  aspects  of
         groups." The  implication'  is  that  RAGs  have  a  major           the Hill-Burton program stating  that  "factors  of  medical
         policy-making role which allows them to  both  affect  and           economics,  civic   pride,   institutional   autonomy,   and
         effect regionalized activities. Such is the case, as  each           professional chauvinism have  retarded  the  initiation  and
                                                                              development   of   coordinated   hospital   systems."2   The
         RAG  is  charged  with  responsibility  for  planning  and           Hfll-Burton goal of coordinated  hospital  systems  and  the
         operating   each   Regional   Medical    Program    (RMP).
                                                                              RMP  goal  of  cooperative  arrangements  contain  many   of
         Further, each RAG has a specific role in  developing  pro-           the same concepts  and  principles  of  regionalization.  If
         gram objectives and for reviewing, guiding and  evaluating           there is objectivity, it will be found  that  Niasur's  com-
         ongoing  planning  and  operating  activities.  Concerning           ments  about   Hill-Burton's   regionalization   apply   un-
         regionalization, Regional Medical Program  Service  guide-           nervingly to  Regional  Medical  Programs.  Therefore,  each
         lines  require  that  "the  Advisory  Group,  through  the           RAG,  in  keeping  with  its  charge  to  produce   an   in-
         grantee, must submit to the Division of  Regional  Medical           dependent  evaluation  must  ask  such   probino   questions
         Programs  an  annual  statement  giving  its   independent           as:                                                 0
         evaluation of effectiveness of  the  regional  cooperative               I .what regionalization activities have  we  planned     .?
         arrangement   (regionalization)   established    by    the               2. how many have been implemented?
         Regional  Medical  Program."'   Clearly,   the   RAG   has               3. how are they progressing?
         responsibility in the  regionalization  process  and,  [as               4. of all our RMP activities how          .many   are   truly
         stated in the above quote] it  must  make  an  independent                  "regionalized"  as  opposed  to  "regional"?  To  this
         evaluation of regionalized activities. Logic, rather  than                  extent  the  term  "regional"  implies  simply   those
         official guidelines, dictates that RAG evaluation  is  in-                  RMP  activities  that  happen  to  take  place  in   a
         dependent of the RMP Coordinator or Ws staff.
           'USDHEW,  Health  Services  and   Mental   Health   Administra-        'Di. Jack Masur,  "Regional  Planning  Cannot  Remain  a  Paper
         tion, Guidelines, Regional Medical Programs 196 8, p. 9.              Pattern," Hospitals, 34, January 1, 1960, p. 48.

         1@l
                                                                                                                                              207


         I
         I
         I





                     geographic area  or  region,  and  the  term  "regional-     plemented  regioiialization  in  the  same  manner;'they   did
                     ized"   implies   very   definite   well-defined,    co-     not. Perhaps our present  inability  to  state  specifically,@
                     operative activities.                                        what reaionalization means in the health field comes,4@ -
                  These questions  lead  to  the  third  phrase  which  needs     from  these  varied  approaches.   At   least   three   models
                explanation-"the   regionafizatiori   process."   It    might     result from a historical analysis of  health  regionalization:
                have  been  better  to  use   the   phrase   "regionalization     (1) patient care; (2) planning and  coordination;  and  (3)  a
                concept"  rather  than  the  "regionafization  process,"   as     continuing  education  model.  The  patient  care   model   is
                44 concept"  means  an  idea  whereas   "process"   implies   an  more  or  less  a  composite;  however  it  preceded  the  de.
                ongoing,  continuous  activity  or  development.  Clearly  in     velopment  of  the  other  models.  The  Dawson  Report,   the
                the continental United  States  we  do  not  as  yet  have  a     report  of  the  Committee  on  the  Costs  of  Medical  Care,
                regionalized  approach  to  the  delivery  of  medical  care;     and  the  regionalized  health  service  program   in   Puerto
                nor do we truly  have  a  definite  formal  regionalized  ap-     Rico fall Nvithin this category. The  patient  care  model  is
                proach on a national basis to any co    mponent   of   health     characterized by such   program  operations  as:  (1)   direct
                services, be it continuing education  or  hospital  planning.     service  patient  care;  (2)  regional   planning;   (3)   co.
                At  present,  we  are  more  in   the   world   of   applying     ordination of services and facilities;  (4)  post-graduate  or
                regional concepts or ideas to the field of health.                continuing education;   and (5)  clinical  and  administrative
                  Many individuals are  familiar  with.  earlier  efforts  at     consultation.     The     regional     scheme     has     such
                applying  the  regional  concept  to  the  broad   field   of     characteristics  as:  (1)  a  coordinated  network   of   com-
                health  and  medical  care;  the  Bingham   Associates   Pro-     prehensive regional health  and  medical  care  services;  and
                gram,   the   Rochester   Regional   Health   and    Hospital     (2)  cooperative  relationships between  local, d      istrict,
                Council,   the   Albany   Regional   Hospital   Pro-ram    of     state  and  (frequently)  national  planning   agencies.   The
                Albany Medical  College,  and  the  regionalized  program  of     "coordinated  network"  results  from  a  process   of   inte-
                medical  care  in  Puerto  Rico  stand  out  as   "benchmark"     gratii2g  services  through  cooperative  efforts  which   are
                efforts. In  the  United  States,  these  programs  were  the     directed at  relating  spatially  separated  health  care  re-
                initial application of the regional concept.  They  were,  to     sources  and  activities  to  one  another  within  a  defined
                a  considerable  degree,  based  upon  an  earlier  phase  of     service area.
                conceptual  development,  a  phase  which  began  in  serious       The  planning  and  coordination  model  has  similar   pro-
                fashion in  England  when  the  Report  of  the  Consultative     gram   operations,   but   excludes   direct   patient    care
                Council  on  Medical  and  Allied  Services  (Dawson  Report)     services. Whereas the regional  scheme  of  the  patient  care
                was published in  1920.  This  report  contained  a  recoi-n-     i-nodel  is  formal  and  somewhat  rigid,  the  planning  and
                mendation  for  regionalizing  the   delivery   of   personal
                health services. The  characteristic  regional  format  of  a     coordination   model    contains    voluntary    relationships
                medical center as a base  facility,  a  comi-nunity  hospital     between local, district  and  state  planning  councils,  plus
                as a district facility and a health  center  in  an  outlying     voluntary relationships  between  facilities  within  a  given
                area  with  a  two-way  flow   of   service   and   education     service   area.   Another   distinction   between   the    two
                between the  institutions  had  its  modeni  day  ori2ins  in     regional  schemes  is  that  the  patient   care   model   has
                this 1920 report.                                                 relationships among all  health  services  resources,  whereas
                  Individuals,  public  and  private  committees   and   com-     the planning  and  coordination  model  Iii-nits  its  concern
                missions  have  been  influenced  by  this  report  and   its     primarily  to  facilities.  Examples  of  this  model  include
                format for regionalization. Similarities  are  found  in  the     various hospital planning  councils  and  the  report  of  the
                                                                                  Commission on Hospital Care.
                1932  final  report  of  the  Comniittee  on  the  Costs   of
                Medical  Care,  reports  by  the   Senate   Subcommittee   on        The  continuing  education  model  is  derived   from   the
                Wartime  Health  and  Education  in  the   mid   1940s,   the     numerous    programs    of    postgraduate    education    de-
                report of the Comrrdssion  on  Hospital  Care  in  1946,  the     veloped  and  administered  by   medical   schools   beginning
                Commission  on  Financing  Hospital   Care,   and   in   such     at the  conclusion  of  World  War  11.  Although  this  model
                Federal government  efforts  as  the  Ewing  Report  and  the     justifiably belongs  in  a  discussion  on  the  evolution  of
                Magnuson  Commission.  The  writings  of   such   individuals     health   regionalization,   such   programs   are   "regional"
                as   Graham   Davis,   Joseph   Mountin,   and   John   Grant     only  in  the  sense  that  relationships  exist   between   a
                contain  a  philosophy  similar  to  that   of   the   Dawson     medical  school  and  certain  hospitals  within   a   defined
                Report.                                                           area, or certain organizations  offer  educational  activities
                  One   must  not  gain  the  impression   that   all   these     for tile health  manpower  of  a  given  geographical  region.
                reports,   individuals   and   programs   defined   or    im-     The  only  program  activity  is  continuing  education   and,

                208




                                                     cteristics   of    a    Generated   "coiiii)lellcnsive   regionalizatioii."   He   states
       usually, it is difficult to Pill-Poilit cil2r@                        0
       regional scheme.                                                      that   "comprehensive   regiotializatioii   Would    provide    a
          If we in the health field are objecti%,e,   we   will    recall    mechanism  for  alloc@ititig  resources,  including  iiiatipowet-
       that application of regional concepts liaL]    been    done     to    and facilities, itiioii- all  health  institutions  in  a  region
       other  industries,   and   with   -reater   success,   before   we    and  it  would  link  the  region's   central   arid   peripheral
       "found"  this  approach.  Stich  being  the  case,   and   through    institutions in ordci-  to  facilitate  patient  referrals,  flow
       the study of  various  types  and  forms  of  application,  it  is    of   patient   records,   consultation   by   specialists   iiicl
       not difficult  to  develop  certain  essential  elzi-n-3nts  which    -eiieralists,  ind    continuing,  education.    "       Clearly.
       characterize  a  regional  scheme   that   can   apply   Generally    Bodeiiheiiiier'S  use  01'  the  term  "comprehensive   re-ional-
       to any activity.  The  recional  concept  is  not  static;  it  is    ization"  parallels  the  functional  elements  of  the   patient
       flexible  and  its  application   varies   depending,   upon   the    care  and  ideal  iiiodcls,  and  clearly  the  operatin-  RiNIPs
       type  of  activity.  Therefore,  in  this  atitlior's  mind,   the    and  their  RAGs   have   failed   to   implement   comprehensive
       essential  elements  of  regionauzation   in   the   broad   sense    regionilizatioii.
       can  be  divided  into   two   categories:   structural   elements     I-lave  RNIPs  had  any  successes,  arid  do  they  deserve   a
       and functional elements.                                              passin(,  grade  for  any  implementation  of   regiotialization?
          Structural  elements  are  common   aiiio,-izi   all   regional    Clearly, this  there,  the  answer  must  be  yes.  Although  the
       schemes, whereas functional elements               arid     relate    followili- information  is  based  on  1969  data,  the   Current
       specifically  to  the  activity  which  is  to  be   regionalized.    percentages are about   the  same.  Almost  a   year   ago,   the
       The three corru-non structural elei-neiiis are as follows:            Division  of Regional   Medical     Pro-rams     had     approved
          1. the  region  must  be  demarcated  so  that  the   area   is    536 projects@ of this  total   number   55   percent   were    in
             economically and spatially defined-.                            coiitinuiiic,   education   and   trainin-,   26   percent   were
          2. there  must  be  a  single  organizational  structure   that    demonstrations  of  patient  care,  I  I   percent   were   coti-
             involves   the   complete   region   wherein    adniinistra-    cerned  with   planning,   coordination   and   evaluation,   and
             tion   is   undertaken   by   a    single    a(,ency    with    8  percent  were  in  the  area  of  research  and   development.
             authority  to  undertake  its   responsibilities   througl-i     The  author  does  not  have   first-hand   knowledge   of   all
             centralization   of   policy   and    decentralization    of    536 projects-,  however  he  has  visited  various  RMPs  out  of
             activity or programs operations; and                            professional interest  and  as  a  consultant  for  the  Division
          3.  there  must  be  a   single   financin-   mechanism   which    of  Recional  Medical   Programs.   Further,   progress   reports
             supports completely the entire recional activity.               and  other  descriptive  materials  have  been   perused   on   a
       The   functional   elements   specific   to    health    regional-    number  of  funded   RMP   activities.   From   this   composite,
       ization are, again in this author's eyes:                             the  author  has  a  comfortable  feelin-   that   Hs   knowledge
          1. direct service patient care;                                    about  RMPs  and  their  activities   is   representative.   This
          2.    maintenance    and    improvement    of     professional,    being the case, it is a fair  judgement  to  state  the  RMPs  in
             tectuiical and administrative practice;                         general  have  addressed  themselves  to   various   aspects   of
          3.  rational  planning,   coordination   and   integration   of    re-ionalization,  or  "cooperative   arrangements,"   as   stated
             resources and services.                                         in the Federal guidelines.
       Unless  a  scheme  contains  these  six  essential  elements,   it      Over half  of  the  funded  RMP  activities  are  in  the  area
       falls short of the ideal.                                             of   continuing   education,   and   many   involve   cooperative
          Obviously,  Regional  Advisory  Groups   have   had   a   major    arrangements  among  institutions,   acencies   and   other   re-
       impact   in   determining   both   the   regional    scheme    and    sources   in   the   regions.   Cooperative   arrangements    are
       regional  activities  of  RMPS.  What   is   the   result,   then,    more    characteristic    of    patient    care     demonstration
       when  a   comparison   is   made   between   @NIP   regional   ac-    activities  where,  frequently,  there   exists   a   coordinated
       tivities,  the  three  models  of   health   reoionalization   and    effort  of  patient  referral  and  a  flow   of   patients   and
       the   ideal   model?   Comparing   RMP   activities    with    the    patient    services    between    institutions.    Examples    of
       patient  care  and  ideal  models   results,   generally,   in   a    regional  cooperation  are  not  difficult  to   find   among   8
       failing  grade  for  the   operating   @NIPS.   One   must   agree    percent of the  total  funded  activities  in  the  area  of  re-
       with   Bodenheimer's3    evaluation    that    RMPs    have    not    search  and  development.  Not   all   of   the   planning,   co-
                                                                             ordination  and  evaluation  activities  (1  I  percent  of   the

           3Tbomas  S.  Bodenheimer,  "Regional  Nledical  Programs:   No
          Road to Regiona@,ation,"  Medical  Care  Review,  26,  December
          1969, pp. 1125-1166.                                                   41bid., p. 1155.
                                                                                                                                           209




                total)    are    concerned    with    cooperative     arrangements    of performance or operations; each RAG plans
                because  many  such  projects  support   RMP   core   staffs.   On    evaluates.
                the  other  hand,  various  projects  can  be  isolated  in   this     Cooperative  arrangements  are  the  distingui
                area  where  there  are   definite   efforts   to   identify   the    characteristic of the Regional Medical Programs  a
                characteristics   (health,    economic,    social,    demographic,    proach  to  regionalization.   To   go   beyond   this   approach
                etc.)  of   a   region;   further,   numerous   projects   address    toward  the  patient  care  and   ideal   models   may   possibly
                themselves   to   coordination   of   facilities,   manpower   and    go  against  PL  89-239  Section  900c,   which   requires   that
                other resources in the regions.                                       RMP  activities  cannot  interfere  with  the  patterns  or   the
                  Federal   guidelines   explain   that   the   terms    "coopera-    methods   of   financing   patient    care    or    professional.
                tive   arrangements"    and    "regionalization"    are    synony-    practice or with the administration of hospitals.
                mous,  although  they  also   state   that   regionalization   can     Each  RAG,  as  one  of  the  contributing   elements   in   the
                connote   more    than    regional    cooperative    arrangements.    regionahzation   process,    must    deten-nine    whether    the
                In  support  of  a  more  broad   connotation   for   regionaliza-    patient  care  and  ideal  model  runs  afoul  of  this  require-
                tion   beyond   the.hmited   idea    of    cooperative    arrange-    ment. There appear  to  be  no  official  regulations  or  guide-
                ments,  the  guidelines  list  several   other   facets:   linking    lines  to   determine   how   far   an   RMP   can   go   without
                patient  care  to  research  and   education;   sharing   of   re-    interfering with existing patterns.
                sources;  coordination   among   and   between   public,   private      Each  RAG  will  undoubtedly  face  this   challenge   in   the
                and   voluntary   health   agencies   and   organizations.    This    near  future,  not  necessarily  from   Regional   Medical   Pro-
                broadening    of    the    term    approaches    the    functional    gram  Service  itself  but  from  outside  influences   such   as
                elements  of  the  patient  care  and   ideal   models   discussed    organized    community     and     consumer     groups,     tech.
                above.                                                                nological,   scientific,   and   organizational   accomplishments
                  If  this  distinction  is  only  partly  true  then   there   is    and  progress  in   the   health   field,   and   proposals   for
                confusion.   A   personal   opinion   holds   that   the   Federal    national.health insurance.
                guidelines  explain  the  broad  facets  of  the   regionalization      What    will    become    of    Regional    Medical    Programs
                concept   but   emphasize   only   one    segment    ("cooperative    when  one  or   a   composite   of   existing   legislation   and
                arrangements")   of   the   regional   process.   However,    "co-    proposals  for  national  health  insurance   is   enacted   into
                operative  arrangements"  can  exist,   and   many   did,   before    public  law?  Have  the  RAGs  made  any   evaluation   of   such
                PL   89-239.   Continuing   education   relationships   and    co-    existing  proposals  and  legislation?  Is  regionalization   and
                operative    arrangements    have    existed    between    medical    the  organization  of  health  services  included  within   these
                schools   and   conununity   hospitals   for   years;    similarly    activities?
                between   health    agencies    and    organizations    and    the       Currently,   there   are   about   10   to   15    legislative
                health   manpower   within   given   geographic   areas.    Formal    proposals  for   national   health   insurance.   A   number   of
                and   informal   cooperative   arrangements   for   patient   care    these  contain  specific  components  which   are   directed   at
                also  existed  prior  to  RMP.  As  must  be   obvious   by   now,    changing   the   organization,   delivery   and   financing    of
                regionauzation   means   more    than    developing    cooperation    health   services.   Health   legislation   during   the    1960s
                between  the  resources  of  a  given  area.  Further,  it   means    contained   wording   to   prevent    change.    Regardless    of
                much  more  than  undertaking  activities  for   persons,   insti-    personal   feelings   about   whether   the    nation's    health
                tutions  or  agencies   simply   because   they   happen   to   be    service  system  should  or  should  not  be   changed,   it   is
                located within a defined geographic area.                             obvious  that  the  authors  of  health   legislation   in   this
                   The delineation of a  region  is  one  of  the  easier  aspects    decade  see  things  far  differently  than  those  of  the  past
                of regionalization in  the  health  field.  Relatively  tried  and    decade.  The  boldness  of  the  current  proposals   to   change
                true    economic,    epidemiologic,    and    demographic    tech-    the  "system"  should  provide  some  evidence  of   the   neces-
                niques  can  be   applied   to   designate   a   particular   geo-    sity  to  alter  existing  practices.  The  prediction  is   made
                graphic area as a district, region or health service         area.    that  the  patterns  of  organizing  and  financing  health   and
                Given   a   qualified   and   capable   staff,   the    RAG    can    medical  care  services   and   programs   will   change   within
                delegate  delineation   responsibilities.   More   serious   ques-    the  next  five  years.  Therefore,  it  is  not  whether  change
                tions  and  problems   involving   policy   come   about   as   to    will  come,  but  rather  the   extent,   scope   and   type   of
                what   is   to   take   place   within   a   geographically    and    change.
                economically  delineated  area.  It  is   here   that   each   RAG       The   composition   of   each   RAG   represents   significant
                has not  only  significant  responsibility,  but  a  major  effort    leadership  in  each  of   the   regions.   Collectively   within
                which  transcends  mere   planning   to   include   an   appraisal    each region, and collect   ively  from  a  national   standpoint,

                210





      the RAGs can have  a  significant  impact  on  the  direction    set for our behavior, both  individually  and  collectively.
      of public policy in relation to health services.  The  nation    These priorities also take into account  the  likelihood  of
      is obviously  in  a  transitional  period  regarding  all  of    their being achieved and represent in effect a selection  of
      health   services-its   provision,   financing,    education,    ends from a  number  of  alternatives.  Research  may  serve
      research and planning. Now,  more  than  ever,  is  the  time    several purposes for us. It may indicate the various  alter-
      for objective and independent  evaluation  of  the  regional-    nativ@s open to us and it may  help  us  to  decide  on  the
      ization  process  in  each  of  the  planned  and   operating    feasibility of achieving them, but the  final  selection  of
      Regional Medical Programs.                                       ends is governed by  our  values,  i.e.,  what  we  feel  is
         The  underlying  reasons  for  the  generic  approach   to    desirable.  Some  organizations  are  fortunate  in   having
      regionalization involve  principles  of  optimal  allocation,    consensus anion(, their members'  values.  Others  are  less
      distribution and  use  of  resources,  maximization  of  out-    fortunate  and  are  not  able  to  decide  on   priorities.
      put, and the concept  of  providing  goods  and  services  to    Research cannot create values. It can,  however,  facilitate
      persons  within   defined   locales.   Application   of   the    their application.
      regional approach to any field or activity  is  done  because
                                                                          Once values are  explicit,  priorities  can  be  set  and
      it holds promise as a way of achieving balance.                  evaluation is possible. Evaluation is a  process  of  deter-
         Regionalism results  out  of  a  basic  need  or  require-    mining to what extent goals have been realized,  and,  at  a
      ment  for  a  more  structured  approach.  In  our   society,    secondary level, why a degree  of  success  or  failure  has
      this need is caused by  complexity  and  the  related  neces-    come about.
      sity  to  obtain  benefits  which  have  economic  or  social       Iwo caveats should guide  our  consideration  of  evalua-
      value; i.e. the need for greater efficiency which,  in  turn,    tion. One  is  that  the  common  practice  of  assessing  a
      is related to prudent allocation of such  resources  as  man-    chaotic situation in search of  a  measure  or  measures  to
      power,  equipment,  facilities,   money   and   natural   re-
      sources.  The  organized  approach  to  obtain  greater   ef-    justify a program is not,  by  our  definition,  evaluation.
      ficiencies is planning, and it is here that  the  process  of    This procedure is  more a  process  of  post  hoc  rational-
      planning  and   the   concept   of    regionalization    come    ization or of docui-nentation  to  provide  legitimation.  A
      together, This   is   why   Comprehensive   Health   Planning    second point is that  once  indices  of  success  have  been
      Programs  and   Regional   Medical   Programs   should   work    devised, they sometimes  can  be  the  tafl  that  wags  the
      together.                                                        dog, while, ironically, no longer  reflecting  the  achieve-
         In general,  regionalism  is  seen  as  the  natural  out-    ments  originally  desired.  The  measures  or  indices,  in
      growth of progress, and is  associated  with  better  service    effect, lose their validity. An example of  this  occurs  in
      to  the  consumer  and  maximization  of   output   for   the    the field  of  rehabilitating  the  handicapped,  where  the
      producer. The  regional  concept  is  viewed  as  a  tool  Of    measure of "number of  patients  processed"  has  often  led
      logic to reduce certain intangible  factors  to  more  under-    to the rejection of  persons  requiring  longterm  treatment
      standable  components.  Those  of  us  in  the  health  field    and the acceptance of  persons  with  negligible  handicaps.
      have  an  opportunity  to  apply  this   concept.   In   this    In  this  case,  the  type  of  "score"  has  displaced  the
      respect, regionalism provides the basis for scientific  plan-    direction of the program.
      ning and operation in the  health  field.  Herein  lies  both       One must therefore  periodically  reassess  the  validity
      challenge  and  opportunity  for   each   regional   advisory    of measures  one  chooses  and  also  must  not  allow  them
      group.                                                           to become the criteria for selecting  a  course  of  action.
                                                                       This is the problem of  the  means  becoming  the  ends  and
                                                                       of. the "rigged game."  In  such  situations,  the  criteria
                     Regional Advisory Group                           may lose any  meaningful  relationship  with  desired  ends.
                         gasis for Evaluation                             For  these  reasons,  evaluation  seldom  can  rest  upon
                PAUL E. WHITE and VAN HOVE                             single  measures.  The  meaning  of  each  measure  must  be
                                                                       ascertained periodically, and the validity of a  measure  or
         In considering  the  question  of  RAG  evaluation  it  is    index can be  ascertained  only  by  its  interrelationships
      necessary  to  distinguish   between   evaluation   and   re-    with other  indices.  For  example,  the  number  and  kinds
      search. Although  evaluation  may  involve  research,  it  is    of organizations represented on a RAG tells us  very  little
      essentially a process  of  comparing  scientific  aspects  of    about'what  we  really  wish  to  know  when  we  ask  about
      reality with  preconceived   norms.   The    norms    reflect    the  representativeness  of  the  RAG.  We   must,   through
      values and are usually expressed in terms  of  priorities  we    research,  determine  the  consequences  of  various   forms

                                                                                                                                211





               of  "representativeness"  on  the  activities   and   achieve-   ships among these  characteristics.  Because our
               merits of the regions.                                           collection was concluded only recently, the findin
                  The  'research   we   have   been   conducting    is    not   report are tentative and may not be borne out by a
               evaluative.  We   have   been   studying   the   behavior   of   complete analysis of the data. The tentative
               selected   RMPs   in   an   endeavor   to   understand    some   permit some interesting speculation, however.
               aspects of why they  behave  as  they  do,  not  whether  they       One tentative finding is that RAG membe
               behave well. In  the  course  of  our  research,  however,  we   to  reflect the target populations of the RM
               have   employed   methodological   techniques    and    gotten   of  grant  applications  approved  and  possibly  in  terms  of
               soi-ne results that have significance for evaluation.            core staff activities as  well.  In  areas  with  large  metro.
                  Let   us   now   consider   RAG   evaluation.    Evaluation   politan   areas,   those   RAGs   whose   membership    reflect
               generally is expensive  and  should  therefore  be  done  only   only  state  level  organizations  (with  the   exceptions   of
               with a clear purpose or purposes  in  mind.  In  the  case  of   hospitals)  tend  not  to  be  involved   with   urban   health
               the  RAG,  a  reasonable  purpose  would  be  to  reveal   its   problei-ns. One tentative explanation for this  is  that  state
               shortcomings in given respects and  to  correct  them.  It  is   level   organizations   overrepresent   rural    or    suburban
               assumed that someone  is  interested  in  and  has  the  power   interests.
               or  sanctions  to  correct  these  shortcomings  should   they       Beyond  membership  per   se   we   have   documented   at.
               exist.                                                           tendaiice  at  meetings  of  the  representatives  of   various
                  Intensive evaluation of a  single  local  RAG  is  probably   organizations  and  professions.  An  example  of  at.tendance
               not advisable for several  reasons.  One  is  that  the  work-   in one RAG  is  given  in  Table  1.  Inspection  of  a  number
               ings of  the  local  RAG  are  alreadv  well  known  to  local   of cases such as these will perrrdt  us  to  detect  trends  in
               participants.  Another  is  that  the  value  or   import   of   particular RMPs in  relation  to  such  variables  as  funding,
               possible  changes  in  RAG  structure   or   function   can.be   grant  awards,  core  staff  activities  and   RAG   functions.
               ascertained  only  by  coinparing  the   characteristics   and   (In  our  analyses,  state  and  local  organizations  will  be
               functions  of  a  number  of  RAGS.  Research  findings   from   separated.)
               the study of  a  number  of  RAGs  can  provide  us  with  the       A  provocative   finding   which   requires   further   in-
               means, i.e., methods and criteria  for  evaluating  particular   vestigation  before  being  interpreted   is   that   as   t.he
               RAGs without undue effort or expense.                            proportion  of  physicians  on   the   RAG   increases,   their
                  We  shall  focus  on  three  major  functions  which   RAGS   attendance  decreases..  Conversely,  as  the   proportion   of
               can perform and which  are  likely  to  be  valued.  We  shall   lay people increases their attendance increases.
               call these representation, legitimation  (witlgn  the  region)       We are at  present  considering  attendance  as  a  measure
               and  decision-making  on  two  levels:  one,  decision  making   of  i-nember's   interest   and   are   investigating   factors
               with regard to setting  explicit  policy  and,  two,  decision   conducive  to  greater  or  lesser   attendance.   One   factor
               making  with  regard  to  particular  tasks.   In   evaluating   lo-ically related to attendance or  interest  is  the  function
                                                                                    0
               these particular  functions,  baseline  data  are  not  neces-   0 f attendance for  the  member.  Several  of  these  functions
               sary,  for,  being  themselves  aspects  of  RAG   activities,   may   be   enumerated.    Narrow    organizational    interests
               they obviously did not exist  prior  to  the  organization  of   i-nay  be  furthered  by  procuring  money  for  the  organiza-
               the   RAG.   'Me   principle   task   in   evaluating    these   tion,  by  monitoring   the   distribution   of   funds   among
               functions is in determining, (1)  whether  or  not  they  are,   competitors, and  on  another  level,  by  keeping  abreast  of
               in  fact,  carried  out  and  to  what  extent,  and  (2)  the   cl-iaiiges in the inter-organizational field.  The  setting  of
               validity  of  the  indices  we   employ   to   measure   these   policy for achieving  collective  goals  is  another  function'
               functions. Essentially, the problei-n is one  of  whether  the   This policy may  range  in  scope  from  establishing  criteria
               indices measure what we believe they measure.                    for grant approval and core staff  activities  to  deciding  on
                  Representation is an  interestin-  aspect  of  the  RAG.  A   particular proposals as they arise  without  explicit  criteria
               frequently  used  measure  is  membership   of   various   or-   for their determination being stated.
               ganizations   and   professions   in   the   RAG.   Yet    the       Comparisons   of   attendance   with   the   dispersal   of
               i-neaning of representation  is  i-nore  complete  than  this.   funds  and   Mtli   the   explicitness   of   goals   and   the
               We  imply  by  representation,   not   only   membership   per   consistency or  departures  from  the  goals  of  fund  alloca-
               se, which may  be  token,  but  involvement  as  expressed  in   tions will allow us to  assess  various  kinds  of  involvement
               interest and n7eaiiiiigful role. In our  research,  therefore,   in  the  RAGS.  Tables  11  and   IV   illustrate   a   content
               we  have  compared  the  characteristics  and  activities   of   analysis  of   the   meetings   of   six   RAGS.   These   data
               several  RAGs  in  an  effort  to  understand  the   relation-   represent  a  preliminary  attempt  to   characterize   meeting


               212





             topics  on  a  scale  from  narrow  to  broad   interests.   We     rejection   of   applications   will   permit   us    to    make
             shall in our analysis try to relate  these  different  concerns     inferences  about  the  decision  making   process   and   about
             to other RMP characteristics.                                       the role of  the  RAG.  Core  staff  activities  will  similarly
                In  relationship  to  our  discussion  of  the  analysis  of     be  analyzed  in   terms   of   RAG   and   committee   involve-
             decision  making,  which  has  been  the  major  focus  of  our     ment in decisions affecting the staff activities.
             research, we should  like  to  consider  briefly  the  question       What  kinds  of  questions  'Might   be   answered   by   such
             of  RMP  legitimation.  The   RAG   may   play   an   important     analyses?  We  can  illustrate  with  our  finding  on  regional
             part  in  legitimation.  Interestingly,   while   legitimation,     responses  to  the  RMPS.  The  RMPS   has   two   major   means
             which we shall define as  "the  perception  of  the  activities     of  communicating   policy   to   the   regions:   (1)   through
             of an organization as  reasonable  and  useful,"  is  vital  to     directives and (2) by  its  dispersal  of  funds.  Our  analysis
             the survival of the  organizations,  the  bases  of  legitimacy     of project  applications  indicates  that  the  regions  respond
             are  generally  not  well  understood.   We   have   used   two     little to directives, while, on the  other  hand,  the  national
             measures   of   RMP   legitimacy,   (1)   consensus   on    RMP     level's  awarding  of  funds  in  particular  areas   stimulates
             functions  and  (2)  submission  of   proposals   for   grants.     the  submission  of  project  applications   in   those   areas.
             The  RAG  is   important   in   this   respect   for   we   are     The implications of  these  findings  for  policy  are  obvious.
             comparing   consensus   (a)   among   RAG    and    RMP    com-       Other  questions  to   which   we   hope   to   have   answers
             mittee  members  with   (b)   consensus   among   organizations     include,    "What    are     the     consequences     of     RAG
             involved  with  RMP,  and  (c)  among  organizations  that  are     membership  for   the   dispersal   of   funds   in   terms   of
             not   involved    with    RMP.    From    these    comparisons,     recipients  and  programs?  What  is  the  effect  of  frequency
             controlled statistically  for  other  RMP  characteristics,  we     of  meetings  on   RAG   attendance,   of   RAG   functions   on
             hope  to  learn  whether  and   under   what   conditions   RAG     programs?  Is,  for  example,  a  RAG  that   actively   screens
             membership contributes to RMP legitimacy.                           applications,  as  evidenced  in  the  selective  rejection   of
                Our  analysis  of  decision-making  is  perhaps   the   RAGs     project  applications,  related  to  a  consistent  regionaliza-
             most important function.  It  reflects  representation  in  its     tion policy?"
             fullest   sense   and   is   possibly    conducive    to    RMP       The   types   of   questions   we   are   posing   reflect   a
             legitimacy  in  the  community  (contingent,  of   course,   on     concern   with   understanding    the    unintended    or    un-
             the range  of  organizations  on  the  RAG).  We  have  several     anticipated    consequences    of     organizational     policy,
             measures of decision  making.  One  is  a  measure  of  who  is     structure,  and  activity.  .  This   paper   is   intended   to
             perceived  as  influential  in  decision  making.  (See  Tables     illustrate the use  of  measures  and  their  interrelationships
             V  and  VI.)  These   assessments   will   be   compared   with     to  discover   organizational   processes   and   outcomes.   We
             actual  decisions  made  with  respect  to  grant  applications     have  selected  for   consideration   three   aspects   of   the
             and core staff activities.                                          RAGS:    representation,     legitimation,     and     decision-
                Another  measure  is  provided  by  the  course   of   grant     making.  Although   our   research   is   not   evaluation,   we
             applications from  submission  to  (1)  rejection  or  (2)  ap-     trust  the  findings  Nvill  have  implications  for  evaluation
             proval  by  the  FMs.   Characteristics-   of   the   applicant     by  indicating  the  validity  of  given  measures  and   facil-
             organization,  the  proposal,   RMP   staff,   RAG   and   com-     itatiiig  corrective  action,  once  norms  and  desired   ideal
             mittee  composition  in   relation   to   the   acceptance   or     consequences have been decided upon.











                                                                                                                                              213






                                                 TableI.-RAG Composition andAttendance by Institution of Affiliation and
                                                 Profession of Members


                                                                                           RMP #6


                                                                                         Year                  Year                Year                  Year
                                                                                        67                       68                 69                    70
                                                                                      c1      A  2         c        A             c       A             c       A

                       Affiliation:

                          Federal      Agency             ...................     .03       1.00        .03         .00       .03         .75       .03
                          State/Local      Ag .            ...................    .06         .38       .06         .62       .06         .75       .08
                          Heart/Cancer Vol               ...................      .06       1.00        .08         .87       .08         .92       .08
                          Other   Vol.    Org .            ...................    .06         .25       .09         .25       .08         .50       .05
                          Phys. Org                 ........................      .03         .75       .03       1.00        .03         .75       .08
                          Other   ProL    Org ....................                .06       1.00        .09         .75       .08         .70       .13
                          Hospital                  ........................      .13         .94       .14         .65       .17         .87       .21
                          University                ........................      .06         .75       .06         .75       .08         .66       .08
                          Non-Affil .                .......................      .48         .60       .46         .45       .38         .48       .28
                             Total                  ........................      (31)        .69       (35)        .56       (36)        .65       (39)

                       Profession
                          Physician                  .......................      .38         .71       .34         .76       .38         .75       .39
                          Administrator     ....................                  .06         .88       .08         .50       .08         .66       .07
                          Nurse                   ..........................      .03       1.00        .05         .86       .05         .87       .07
                          Health Prof .                .....................      .03         .75       .03         .50       .03         .50       .05
                          Non-Health  .                .....................      .50         .59       .50         .48       .47         .53       .41
                             Total                  ........................      (34)        .68       (38)        .59       (40)        .64       (41)

                                 Composition:  Proportion  of  RAG  members   in   that   category
                                 Attendance,  Average  attendance  by  the  members  in   each   category






                                                             Table 2.-Scale of RAG Topics According to Narrowr
                                                            (1) or Broader (7) Conception of the Role of the RAG

                                                                                   Topic                                  Value


                                                           Structure   of   RMP              ......................        3
                                                           Application and review procedure      ...........               2
                                                           Specific   proposals              ......................        I
                                                           Staffing of RMP    .......................                      5
                                                           Staff  function  and  duties            .................       5
                                                           Budgets                    .............................        6
                                                           (Other) individual committee function
                                                           and/or activities  .......................                      7
                                                           Goals and priorities of RMP             ................        4
                                                           Housekeeping    .........................                       1
                                                           Relation w/Washington      ..................                   6








                   214





             TABLE 3.-Frequency Distributions of Matters Discussed in RAG Meetings

                     Matters Scaled According to a Narrower or Broader
                         Conception of the Role of the RAG


             Meetings 66-68                    Meetings 69-70



                                  50




                                  40



                                  30




                                  20



                                  10



                                  0
        2   3   4    5  6   7         1   2   3   4   5   6   7



                                  50



                                  40




                                  30



                                  20



                                  10



                                  0

                            7
        2   3                         1       3   4   5   6   7



                                  50



                                  40




                                  30



                                  20




                                  10


                                  0 L-"
        2   3   4    5  6   7         1   2   3   4   5   6   7


                                                                215





              I'ABLE 3.-Frequency Distribution s of Matters Disczissed in RAG Meetinus-Cont.

                         Matters Scaled According to a Narrower or Broader
                             Conception of the Role of the RAG


                 Meetings 66-68                    Meetings 69-70


                                       50



                                       40




                                       30



                                       20



                                       10



                                       0
             2   3       5   6   7         1   2   3   4       6   7


                                       50



                                       40



                                       30




                                       20




                                       10



                                       0
             2   3       5   6   7         1   2   3   4   5   6



                                       50




                                       40




                                       30




                                       20



                                       10



                                       0
             2   3  4    5   6   7         1    2   3   4   5   6   7





                                                                Table 4.-Meati Scores on Topics Scale


                                                                                                         66-68             69.70


                                                         I.                    .....................      3.5              3.5
                                                         2 .....................                          2.5              3.8
                                                         3.. .  - .  .   ..............                   3.6              3.0
                                                         4.                    .....................      3.8              3.4
                                                         5 .....................                          2.9              3.6
                                                         6.                    .....................      3.9              4.0


                                       Table   4.-Mentiotied    as    Influential    Part    of    RMP    Organization    by    Principal    Staff    and
                                       Committee Chairmen
                              RMP    No.    I            RAIP  Alo.  2           RjlfP    No.     3      Rjlfl'No.     4          RAIP   No.    5                Total
                            s          c    T        s       c       T       s        c           T    s       c       T        s       c       T       s        c       T
        Staff     ...       .66        .13  .36      .20     .22     .21     .42      .45         .43  .5 5    .83     .62      .50     .42     .46     .49      .38     .48
        RAG       ...                  13   .07      .40     .22     .28     .32      .09         .23  .20             .15      .25     .14     .'-) 0  .'-) 5   .12     .20
        Board               .33        .25  .28                              .05      .04         .07                                   .14     .07     .05      .10     .07
        Steering
              Corn.                    .25  .14                              .11      .09         .10                           .25     .14     .20     .07      .10     .08
        Categorl-
           cal    ...                                .40     .55     .50     .05      .27         .13                                   .14     .07,    .05      .21     .12
        Other     .                    .13  .07                                                        .25     .17     .23                              .09      .05     .07
        Outside                        .13  .07                              .05                  .03                                                   .00      .05     .02
           N     ....       (6)        (8)  (14)     (5)     (9)     (14)    (19)     (1 1)       (30) (20)    (6)     (26)     (8)     (7)     (15)    (57)     (42)    (97)

               S= Staff        responses
               C=     Committee     Chairmen     responses
               T=    Total     responses





                                       Table5.-CoordinatorandPAGChairtnanMentioiiedasInfluentialby@-.ncipalStaff
                                       and       Committee        Chairmen

                                                                          RMPNO.     I     RMPNO.     2     RAIPNO.     3     RMPNO.     4     RAIPAO.      5    Total
                                                                             S   C    T           S C  T       S   C   T        S   C   T       S   C   T        S   C   T
          Co-ord RMP                            ..........................   4        4           1 2  3       11    4    15    14   4    18    4   4   8        34   14   48
          RAG Chnnn         .........................                        1        1           1 1  2       2       2        1   1   2                        5  2    7
          #       Respondents .........................                      4   5    9           6  8  14     12 10   22       16   6    22    5   4   9        43 33     76

                  S=   Staff   responses
                  C=    Committee     Chairmen     responses
                  T=   Total   responses










                                                                                                                                                                         217






                                                MEDICAL CARE EVALUATION


                                                               Participants
     Glen E. Hastings, M.D. - Moderator                                 Lawrence Ni. Witte
     Coordinator, Nassau-Suffolk Regional                               Senior Assistant H@tli Services Officer
       Medical Program                                                  Office of Pro-.ram Planning and Evaluation
                                                                        Regional Medical Programs Service
     John W. Williamson, NI.D.
     Professor, Department of Medical Care
       and Hospitals
     Johns Hopkins School of Hygiene and
       Public Health



             ABCD Strategy on Patient Care Assessment*                    prevented by  current  medical  care.  This  paper  will  de-
                    JOHN W. WILLIAMSON, M.D.                              scribe  and  illustrate  such  an  approach  as  applied   to
                                                                          patient care research; it will be referred to  as  the  "ABCD
         One of the most important questions to        be   asked   of    Strategy.'.'
       any health care  system  is:  WHO  needs  to  learn  WHAT  to                            DESCRIPTION
       most  improve  health  status  of  the  population  receiving
       care?                                                                Those who have      faced the  task  of  evaluating  patient
         Previous  study  has   demonstrated   there   is   an   im-      care  and  have  contemplated  the  hundreds   of   variables
       portant relationship  between  patient  care  assessment  and      that  can  be  measured  have  probably  recognized   why   a
       education  that  might  provide  a  framework  for  answering      systematic  approach  is  needed.  The  ABCD   Strategy   was
       this  question.'  Systematic  investigation  is   needed   to      designed  to  help  identify  those  variables  which   might
       identify  educational  objectives  that  specify  the   indi.      have the highest probability  of  effecting  significant  im-
       vidual who needs  to  learn  as  well  as  the  goals  to  be      provement in the health status of a target population.
       achieved in the learning  process.  The  doctor,  nurse,  ad-        Figure 1 describes the elements  of  the  strategy.  Areas
       niinistrator, patient  or  general  public  might  each  con-      A and B  represent  the  outcomes  of  care;  C  and  D  rep-
       tribute  important  elements  of  change  to  achieve  needed      resent  the  processes  of  care  that  are  associated  with
       improvement.  After  "instruction,"  the  same   methods   of      those outcomes. It is important  to  note  that  these  areas
       inquiry used to identify the  problem  can  be  reapplied  to      are lettered in the order of their  priority  for  evaluation
       evaluate  the  impact  of  educational  effort   exerted   to      and  not  in   their   chronologic   sequence   in   clinical
       solve the problem.  Finally,  from  this  second  evaluation,      practice.
       new objectives can be identified  for  repeating  the  educa-
       tional cycle,  if  warranted,  to  achieve  further  improve-
       ment.
                                                                          FIGURE  I.-ABCD  Strategy   of   Patient   Care   Assessme?it
          Systen                                   oa     requires   a
       priority list of  health  problems  to  be  studied.  Methods
       for developing  such  a  list  to  encompass  conditions  in-                              OUTCOMES              PROCESS
       volving  the  most  preventable  (or  remediable)  impairment
       are described in a  subsequent  publication.'  Given  such  a
       health  problem  of  high  priority,  a  strategy   is   then      DIAGNOSIS                    A                     C
       needed  to  identify   preventable   impairment   not   being

          *Development of this paper was supported by  Research  Grant
       PH 43-69-948 from the National Center for Health  Services  Re-
                                                                           THERAPY
       search and  Development,  Health  Services  and  Mental  Health
       Administration and the  Milbank  Memorial  Faculty,  Fellowship
       Fund.

                                                                                                                                     219





                     -Area  A   represents   Diagnostic   Outcomes,.the   con-    estimates.  Our work  and  that  of investigator
                  ceptual  base  required  by  the  physician   to   formulate    Beverly  Payne  indicate  that  the latter source
                  therapy  and  prognosis.  It  could  be  as  simple   as   a    estimates) may offer the most practical basis for
                  single  symptom  or  laboratory  result  (e.g.  "cough"   or    standards.3.4 Naturally, all sources of inform
                  "hyperuricemia")  or  as  complex  as  a  disease  diagnosis    to be considered in developing a final synth
                  together with  i-najor  treatment  or  prognostic  consider-    formulation of such criteria should be pri
                  ations   (e.g.   Lobar   Pneumonia   due   to   pneu@ococcus    independent of outcome measurements activity.
                  in a non-compliant patient allergic to penicillin).                Outcome     measurement:     To      measure      diagnostic
                     -Area  B  represents  Therapeutic  Oiitcon7es,  the   ef-    outcomes represented in Area  A,  it  is  important  to  deter-
                  fect of treatment  on  the  health  status  of  the  patient    mine the  proportion  of  the  population  requiring  care  for
                  (e.g.  whether  the  patient   lived   or   died,   remained    a  given  health  problem  who  do  not   receive   it   (false
                  ambulatory  or  was  bedridden,  returned  to   "work,"   or    negatives)   and   similarly,   the   proportion    of    those
                  remained dependent, etc.).                                      receiving care  for  the  same  problem  who  do  not  need  it
                     -Area  C  represents  Diagnostic  Pi-ocess  or  the  pro-    (false  positives).  To  measure   the   therapeutic   outcomes
                  cedure  carried  out  to  formulate  the   conceptual   base    represented  in  Area  B,  follow-up  study  is  important   to
                  symbolized  in  "A"  (e.g.  history  taking,  physical   ex-    investigate the patient's resulting  functional  condition.  If
                  amination,  ordering  laboratory  tests,   analyzing   data,    the follow-up  interval  is  sufficiently  long  as  to  ensure
                  arriving at a diagnostic synthesis, etc.).                      stabilization of health  status,  each  patient  can  be  clas.
                     -Area  D   represents   the   Tlterapeutic   @ocess   in-    sified by  level  of  i-naximum  overall  impairment.  In  this
                  volving  planning,  implementing   and   evaluating   thera-    study, the following six levels were used:
                  py   (e.g.   prescribing,   operating,    instructing    the       1. No impairment
                  patient,  follow-up,  compliance  of  the  patient,   etc.).          Measurable   impairment   or    risk    (though    asymp-
                  The strategy depicted  in  this  fi-ure  suggests  that  the
                                                    0                                   tomatic)
              areas  most  important  for  assessment  are   Diagnostic   Out-       3. Symptomatic    (though     working)
              coi-nes  (Area   A)   and   Therapeutic   Outcomes   (Area   B).       4. Not  at  "work"  or   "major   life   activity"   (though
              The  data  obtained  from   these   assessments   may   indicate          ai-nbulatory)
              wheth,er  subsequent  effort  to  study  and   improve   patient       5. Bedridden
              care process (Areas  C  and  D)  is  warranted.  Note  that  the       6. Dead
              arrows leadijicy to C and D are  dotted;  this  is  to  indicate
              that  if  the  outcomes  are  within  previously   agreed   upon       Process   Study:   Compariilg   measured    findings    with
              standards,  further  study  of  care  process  can  be  deferred    established  criteria.  reveals   whether'detailed   study   of
              for that  particular  problem  in  favor  of  an  outcome  study    medical care process  is  indicated.  It  is  recommended  that
              of  the  health  P"roblem  with  the  next   highest   priority.    95%   confidence   intervals   about   measured   findings   b@
              Finally,  it  should  be   borne   in   mind   that   subsequent    used, especially if one's  sample  is  less  than  one  hundred
              process  study  might  possibly  reveal  the  outcome   criteria    patients.  (For  example,  if   a   maximum   acceptable   case
              or  standards  were  unrealistic  rather  than  indicating   re-    fatality  rate  were  set  at  5%,  a  measured  rate  of  10%,
              modiable deficiencies in patient care.                              with  confidence  limits  of  4  to  15%,  would  not  be  sig-
                  To implei-nent  this  strategy,  outcome  criteria  need  to    iiificantly  different  from   criteria   and   process   study
              be  developed  and  compared  to   outcome   t?ieaszii-ci?7eiits    would not  be  indicated.)  Finally,  the  specific  objectives
              to  determine  %vhether   process   stittv   is   required.   If    and  methods  of  process  study  will  vary  widely  according
              process study  is  indicated,  resiilti)ig  action  should  then    to  the  content  and  seriousness   of   the   outcome   defi-
              lead  either  to   revision   of   the   outcome   criteria   or    ciciicy leadiii- to such inquriy.
              eventually to improved care outcomes.                                  Resztlliiig action:  As  a  result  of  process  study,  the
                  Outcome  Ci@itei-ia.-  Many  sources  of   information   can    (lirc(,tioii and priorities  for  action  to  improve  outcomes
              be  used  in  synthesizing  such  criteria  or  standards.   For    should be  revealed.  If  improvei-neiit  were  not  found  in-
              instance,   to   determine   the   maximum    acceptable    one-    dicated  by  such  study,  then  the  outcome  criteria   would
              year case  fatality  rate  for  patients  with  a  given  health    seem to  require  modification.  If  the  criteria  proved  ac-
              problem,  data  from  any  of  a  number  of  sources  might  be    curate,  then   improvement   of   the   health   care   system
              used: general mortality or actuarial statistics; mortality          would be  necessary  to  correct  those  factors  found
              studies of populations similar to  one's  own;  previous            causally related to the deficient  outcomes.  Later,  a
              mortality  studies  of  one's  own.   population;   peer            repeat cycle of criteria, measurement, comparison,  and

              220




     possibly   another   process   study,   would   be   done   to    faculty  from  the  State  University  in  continuing   educa-
     evaluate the effect o'l' the preceding effort to improve.         tion  programs  related  to  the  diagnosis   of   UTI.   When
                                                                               C,
                                                                       subsequent  study  revealed  little  or  no   improvement   in
                         ILLUSTRATION                                  performance,  the  physician  staff,  nurses  and  administra-
     Assessing Diagtiostic Outcomes (Area A)                           tion  solved  the  problem  by  instituting   routine   bacte-
                                                                       riologic screening (by smears and cultures)  of  urine  ftoiii
       Example 1: Area A study of Urinary        Tract    Infection    all admissions  to  this  hospital.  This  procedure  includes
     rnanagemept in a conununity hospital       in   the    Midwest    follow-up verification  of  bacterial  infection  of  patients
     was  done   by   prospective   examination   of   over   6,000    with positive  screening  test  results,  thereby  effectively
     consecutive     admissions.     This     investigation     was    reducinc, both  false  ne@ative  and  false  positive  results
     conducted   in   collaboration   with   the   Department    of    well below diagnostic outcome criteria.
     Medicine   of   a   nearby   State   University   School    of      E.Yatiiple 2: Area A study  of  UTI  was  also  carried  out
     Medicine.   Criteiia   were   independently   established   by    prospectively  in  the  Medical  Out-Patient  Clinic  of   the
     group  judgment,  taking  into  account  the  sensitivity  and    same  State   University   mentioned   above.5   Critetia   of
     specificity  of  methods  for  detecting  urinary  tract   in-    "maximum   acceptable"   outcomes    established    for    the
     fections (UTI's) and the implications  to  the  patient  of  a    Comniunity   Hospital   study   were   applied   here:   false
     missed   diagnosis"   or   a   "misdiagnosis."   The   maximum    negative, 15%;  false  positive,  20%.  Measurement  of  diag-
     acceptable percentage of  false  negatives  was  set  at  15%.    nostic   outcome   was   accomplished   by   an    independent
     The  maximum  acceptable  number  of   false   positives   was    Study    Team    who    exarained    133    consecutive    new
     set  at  20%.  Measurement  revealed   that   265   of   6,145    patients  admitted  to  the  Medical  Clinic.  They   obtained
     consecutive  non-new   born   admissions   probably   required    from each  patient  a  detailed  history  and  urine  specimen
     management  of  UTI;   187   of   these   patients   did   not    for urinalysis  and  culture.  After  receiving  this  special
     receive this care from the regular  hospital  staff,  yielding    workup, patients were  admitted  to  the  Medical  Clinic  for
     a false negative rate of over 70%.  Of  I  10  patients  diag-    routine  management  by  the  regular   clinic   staff.   Over
     nosed by the hospital  staff  as  having  UTI,  32  had  urine    three  months  later,  the  patients'  charts  :were  examined
     test results negative for pyuria  and/or  bacteriuria,  yield-    and recorded  results  were  compared  with  the  findings  of
     ing a false positive rate  of  29%.  Process  Study  (Area  C)    the  Study  Team.   Of   18   patients   requring   management
     in  this  example  was  indicated  for   two   reasons   since    for  UTI,  according  to  the  Study  Team,  10  were   missed
     maximum  acceptable  criteria  for  both  false  negative  and    by  the  clinic  staff.   Although   there   were   no   false
     false   positive   diagnoses   were   exceeded   by   measured    positives, it is interesting that  the  upper  limits  of  the
     findings.  The  process  study  consisted   of   testing   the    95%  confidence  intervals  about  the  proportions   of   0/8
     physicians  with  a  series   of   brief   simulated   patient    and IO/  1  8  are  the  same  as  the  upper  limits  of  the
     vignettes  (describing  a  patient's  history,  physical   and    proportions  found  in  the  Community  Hospital.   In   other
     laboratory findings) requiring a  decision  to  treat  or  not    words, with 95% probability,  it  is  possible  that  in  both
     to treat for UTI with  antibiotics.  It  was  found  that  the    institutions,  there  could  be   as   many   as   77%   false
     doctors   tested   would   usually   prescribe   antibacterial    negatives and 36%  false  positives.  Process  Study  of  Area
     treatment  for  all  patients  who  complained  of   classical    C was clearly indicated  in  this  instance.  The  Study  Team
     UTI symptoms  regardless  of  the  results  of  urinalysis  or    identified  one  or  more  major  UTI  screening   indications
     urine-culture.  Chart  reviews  and   follow-up   study   con-    in 108  of  133  consecutive  new  patients  admitted  to  the
     firmed this finding  'in  actual  practice.  Clinical  studies    Medical  Clinic.  (Examples  of  major  screening  indications
     have  established  that  many  patients  with  UTI  will   not    are: previous  UTI'S,  recent  pregnancy,  history  of  pelvic
     present  with  classical  symptoms,  and  that  many  patients    surgery, catheterization, renal  calculi,  etc.)  The  Medical
     with  such  classical  symptoms  as  burning   and   frequency    Clinic staff  found  and  recorded  such  indications  in  the
     of urination will not  have  bacterial  infections.  It  seems    charts of only 69  patients  and  followed  through  with  the
     possible that a practice  of  diagnosing  only  patients  with    indicated  screening  in  only  31  of  these.  To   determine
     overt symptoms  could  account  for  many,  if  not  most,  of    whether  the  problem  was  a  matter  of  "not  knowing"   or
                                                                       41           11                                e
     the false negatives and false positives found.  It  was  clear      not  doing,  each  staff  member   complet   d   a   written
     the  improved  outcomes   would   depend   upon   the   physi-    examination  (similar  to  speciality  board  exams)  to  test
     cians learning to utflize urine test results in the  diagnosis    his  knowledge  of  urinary  tract  infection  diagnosis   and
     of these  infections.  Resulting  action,  in  this  instance,    treatment.   The   average   score   was    83Yo;    subscores
     included several  meetings  of  the  medical  staff  with  the    indicated  that  these  physicians  had   adequate   knowledge

                                                                                                                                  221






                                                                                                                    as not indicate
               of  UTI  screening   indications   and   diagnostic   require-   staff. Process study of Area C w                    d   since
               ments. The  educational  problem  identified  here  was  of  a   the  95%  confidence  interval  about  the  8%  included  the
               different  nature   than   that   found   in   the   Community   5%   maximum   acceptable   level.   Resulting   action   was
               Hospital,  but  had  equally  serious  implications  for   the   taken  in  this  example  because  the  medical   staff   was
               patients concerned. It  is  surmised  that  continuing  educa-   sufficiently  disturbed  by   the   finding   of   8%   false
               tion,  consisting  of   formal   courses,   information,   and   negatives.  They  carried  out  an  educational  program   on
               admonitions   to   improve,   would   probably   be   as   in-   heart  failure  diagnosis  for  the  Emergency  Room   staff.
               effective at  the  University  as  it  was  in  the  Community   This implies  that  the.maximum  acceptable  criteria  estab-
               Hospital  described  above.  Resulting  action  has  consisted   fished by the staff may have been too high.
               of discussion of the problem in  one  staff  meeting  of  tIhe     Having  illustrated  the  assessment  of  Area   A   (Diag.
               Department   of   Medicine.    Unfortunately,    unlike    die   nostic  Results),  using  methods  which   employ   the   in-
               persistent  and  untimately  successful  effort  in  the  Com.   dependent diagnostic  assessment  of  a  cohort  of  patients
               munity  Hospital,  to  our  knowledge,  no  action   has   yet   to identify percentage of false positive and  false  negative
               been taken to solve this problem at the University.              diagnoses,  attention  will  now  be  given  to  illustrative
                                                                                examples  concerning  assessment  of  Area   B   (Therapeutic
                  Example  3:   Area   A   study   of   heart   failure   was
               conducted at a City  Hospital  in  the  East  by  a  group  of   Relultl)-
               full time internists who  were  also  faculty  members  of  an
               internationally  recognized  private   school   of   medicine.   Assessing Therapeutic Outcomes (Area B)
               They  were  interested  in  applying  these  same   principles     Example  1:  Area  B  study  was  conducted   by   a   one-
               to the study of  patients  in  the  Emergency  Room  of  their   year  follow-up  of  75  patients  having  management   needs
               institution. The  first  cohort  they  investigated  consisted   related  to  heart  failure   among   the   113   consecutive
               of  113  consecutive  patients  suspected  of   having   acute   Emergency  Room  patients  in   the   above   City   Hospital
               coronary  artery  disease.  Criteiia  were   based   on   peer   suspected of having an  acute  coronary  occlusion.  Critelia
               judgment.  The  staff  identified  5,7o  as  the  maximum  ac-   of the maximum acceptable  case  fatality  rate  set  by  the
               ceptable level for both  false  negative  and  false  positive   medical  staff  by  peer  judgment   techniques   were   30%.
               diagnostic  results.  Measurement   of   diagnostic   outcomes   Measurement    of    therapeutic    outcome    on    one-year
               was based  on  retrospective  chart  review  since  sufficient   follow-up revealed that 23 (3 1 'Yo) had died. Process  study
               information about  critical  positive  and  negative  findings   of  Area  D  was  not  indicated  since  the  95%  confidence
               was available in  the  chart  of  nearly  every  patient.  The   lirrdts  about  the  measured   findings   did   not   exceed
               findings indicate a false negative rate  of  3%  and  a  false   criteria. Resulting action has not been necessary.
               positive rate of  OL7o.  Proc@ss  study  of  Area  C  was  not      Example  2:  Area  B  study  of  the  75  acute   coronary
               indicated  since  criteria  were   not   exceeded.   Resulting   suspects   having   "heart    faflure"    management    needs
               action has not been indicated or undertaken.                     included the follow-up of the  46  patients  who  were  alive
                  Example  4:  Area  A  study   of   diastolic   hypertensive   and  ambulatory  one  year  following  their  coronary   care
               patients  requiring  heart   failure   management   was   con-   unit admission  Oiteria  were  set  by  staff  judgment.  The
               ducted  prospectively   by   screening   approximately   2,000   maximum  acceptable   proportion   of   patients   ambulatory
               consecutive  medical  patients  who  visited  the  same   City   at the end of one year who were not back to th     eir  major
               Hospital's   Emergency   Room   Criteria   of   maximum    ac-   social activities,  e.g.  at  "work,"  was  20%.  Measurement
               ceptable  rates  of  diagnostic  outcomes   were   established   indicated that 17 (37,7o) of these 46, had  not  returned  to
               by the staff at 5%  for  false  negative  and  10%  for  false   "N@,ork."  Process  study  of   Area   D   seemed   warranted
               positive  diagnoses.   Measurement   of   diagnostic   outcome   because  of  the  marked  discrepancy  between  the  standard
               was  carried  out  on  each  medical  patient  found  to  have   established  and  the  measured  findings.   Study   revealed
               diastolic  pressure  of  110  mm  HG   or   greater   on   the   that 12 of  the  17  individuals  who  had  not  returned  to
               routine  blood  pressure  examination.  When  such  a  patient   "work"  had  not  had  a  coronary  occlusion  at  the   time
               was  discharged   from   Emergency   Room   care,   a   member   they  were  in  the  coronary  care  unit.  These   patients,
               of the Study  Team  took  him  to  an  adjacent  room  for  an   although  suspected  of  having   an   acute   coronary   oc-
               independent  workup  for  cardiac  failure,   which   included   clusion, proved    to  have  other  explanations  for   their
               a  history,  physical  examination,  EKG   and   chest   film.   symptoms. Of these 12, ten  had  been  leading  active  lives
               From  this  study,  98  patients   were   found   to   require   prior to admission  to  the  Coronary  Care  Unit.  The  fact
               management related to  heart  failure.  Only  8  of  these  98   that more than two out of  three  of  the  patients  who  had
               (or  89o)  were  niissed  by  the   regular   Emergency   Room   not returned to their previous  major  activity  and  had  no

               222




        discernible organic reason for that  level  of  impairment    their  physicians  but  were  not  taking   .dication   for
        seemed an important finding. Restilting  action  consisted    hypertension  or  heart  f;aflure.  Ille  reni3inder   were
        of staff review of this Problem and  the  conclusion  that    neither  under  a  pliysici&Ws  care  not   taking   needed
        "work" evaluation  studies  of  patients  discharged  from                                               estion  that   a
                                                                      medication,  There  seemed  to  be  no  qu  ,,c  of   these
        the Coronary Care Unit are indicated.                         serious problem had been Identified in  tile ere  appear   to
                                                                      patients. As %vitti the pi@ingtxsnioc  '  th
        Example  3:  Area  B  study  of  the  98   patients   with    be administrative and  educational  Implications  for  both
        heart  failure  management  needs  related  to   diastolic    patients and physicians, if not ik  entire  present  system
        hypertension was also based on a  one-year  follow-up.  In    of medical care,                           ot  respond   to
        this  example,  mortality  was  the  therapeutic   outcome                    which,  too  often.  does  n
        studied. Complete information  was  obtained  from  87  of    the paticilt's self-neglcct of his own  nledic3i  problems.
        these patients one year  following  their  Emergency  Room    Resulting  action,  since  the  tinx  of  this  study,  has
        visit. Nearly half of the group was below 50 years of  age    included inipro@-ed methods (,)t  t%@u2ting  and  following
        and most were active working-class  individuals.  Criteria    diastolic hypertensive patients  seen  it%  lilt  Emergency
        were es!2blished by first referring to standard  mortality    Room. These patients. as  v%,cU  si  t)tllcm  with  serious
        rate for Maryland, adjusted for age, sex, race and a diag-    chronic problems, 3tc "w  lt(tltcd  IL)  a  special  clinic
        nosis of hypertensive heart disease.7  These    statistics    which has responsibility (of  the  long-term  follow-up  Of
        indicated that of the 87  followed,  only  two  should  be    these individuals, ill oillet wow I( tile patient is to  be
        dead within a  year.  Individual  case-by-case  prognostic    managed  by  all  outside   ph)@n,   this   City   hospital
        analysis by the medical staff indicated as many  as  seven    clinic  will  still  maintain  ttV*niibdity  for   periodic
        might conceivably be dead within a year.  It  was  decided    monitoring  of  tile  lutient's  catt  and  condition.   BY
        that  nine  (10@o)  would  be  maximum   acceptable   case    concentrating responsibility in a  dtflnod  Interest  group
        fatality rate.  Meastirement  revealed  that  on  one-year    and stressing  follow-up  "ustion  of  care,  it  is  hoped
        follow-up, 18 (219'o) were found@ to  have  died.  Process    that subsequent study seat ttvcal ltnpto%vd  patient  out-.
        study of Area  D  was  definitely  indicated.  Preliminary    comes.
        analysis has revealed that of the 18 deaths, 1 1 may  have
        been preventable; all but three of these I  1  involved  a                 FINAL            CONSIDERATIONS
        cardiovascular death (stroke,  coronary  occlusion  and/or      This   approach   to   imict%t   care   awssment   raises
        heart faflure). Data obtained revealed that  only  one  of    concern  regarding  the  ttil3bdity  and  validity  of  the
        the 11 was receiving regular care  from  a  physician  and    criteria used to detern-Jne the need  (or  study  of  "care
        only two were taking antihypertensives  and/or  digitalis.    process." Since there Is Uttk outcome data In  the  litera-
        There  would  seem  to  be  serious   administrative   and    ture, medical  staff,  using  part  judgmnt  nwthods,  must
        educational implications for improving  the  follow-up  of    usually develop their own cdtads  and  standards.  To  test
        such patients. Restdting action taken at this  institution    the reliability of these to M  Cdtcds.  the  staff   members
        Will be described in the next example.                        in the  heart  failure  studlts  wet*  membled  three  dif-
         Example 4: Area B  study  in  this  instance  focused  on    ferent times, at three  nwnth  lnietyah,  to  obtain  their
        symptoms found  among  the  45  patients  back  to  "work"    estimates  of  "maximum   accepuble'   outcomes   for   the
        one  year  following  their  admission  to  the  Emergency    same  group  of  patients.   Altho*   them   was   moderate
        Room. Critetia established by  the  staff  would  indicate    variation of  individual  estimatm  the  nuximum  variation
        that, if more than 50% of the patients returning to  their    about  the  group  mean  was  ka  than  3%  comparing   the
        major life  activity  had  overt  cardiovascular  symptoms    three independent estimtes of six different  criteria.  The
        after  one  year,  further  inquiry  would  be   required.    staff  members  inferred  that   whether   or   not   their
        Measurement  revealed  that  38  (84,7o)  had  such  symp.    estimates were. valid, they at  least  Seemed  to  be  con.
        toms. Process study of Area  D  was  indicated  since,  as    sistent. To test validity of,these granules, the  team  was
        with  the  case  fatality  rate,  the  symptom  rate   far    assembled to provide individual pfognosb for  each  of  the
        exceeded staff criteria  of  maximum  acceptable  results.    100  patients  in  the  hypert@  cohort,  Again,  the   in-
        In this instance, investigation revealed that only  13  of    dividual  variation  was  wide,  but  the  group  estimates
        the  38  symptomatic  patients   who   had   returned   to    proved  specific   and   rneatdngfui   when   compared   to
        "work"  had  seen  a  physician  more  than  once  in  the    empirically    measured     follow-up     findings.     The
        preceding year and were  taking  needed  antihypertensives    comparisons indicated that    group  Prognostic   estimates
        and/or digitalis. Thirteen others were receiving care from    were surprisingly valid.





                   The overall value of the ABCD Strategy appears  to  be             continuing education programs.  It  is  hoped  that  sub-
               supported by three factors: 1) It requires that the  pro-              sequent use of this strategy may  facilitate  development
               viders of care focus on  prognostic  judgments,  probably              of practical methods for and renewed interest in  answer-
               the most critical element in clinical judgment  subsuming              ing the critical qu.-,@stion: WHO needs to learn WHAT  to
               both diagnosis and therapy. 2) It  focuses  attention  on              most improve the health status of the population  receiv-
               overall patient impairment and stimulates search for  any              ing care?
               of the multiple determinants (m6dical, social,  cultural,
               economic,  etc.)  of  such   impairment   that   may   be                                      Bibl;ography
               important. This approach is  in  contrast  to  the  usual
               preoccupation  with  correcting   only   pathophysiologic              1.  Williamson JW, Alexander  M,  Miller  GE:  Continuing  educa-
               causes   of   impairment.   3)   Since   this   strategy   focuses         tion  and  patient  care   research-physician   response   to
               continuing   educational   resources   on   solving   real   prob-         screening test results. JAMA 201:938-942, 1'967
               lems   in   medical   practice,   it   would   seem   to   enhance     2.  Williamson JW, Alexander M, Miller GE:  Priorities         in
                                                                                          patient-care  research  and  continuing  medical   education.
               educational  effectiveness   in   two   specific   ways:   a)   it         JAMA 204:303-308,1968
               identifies  learning  needs,  not  only  forthe   physician,   but     3.  Storey PB,  Williamson  JW,  Castle  CH:  Continuing  Medical
               other  health  care  personnel   and   patients   depending   upon         Education-A    New     Emphasis     (monograph).     Chicago,
               the problem; and,  b)  it  lends  itself  to  educational  assess-         Division of Scientific Activities, AMA, 1968
               ment in  terms  of  the  objectives  of  the  total  care  process     4.  Payne  BC  (ed):  Hospital  Utilization  Review  Manual.  Ann
                                                                                          Arbor, Univ. of Michigan, 1968
               -the improved health of those receiving care.                          5.  Gonelia JS, Goran MJ, Williamson JW, et  at:  The  evaluation
                  Finally,  if  we  focus  on.  the  ultimate  purpose   of   the         of patient care-a new approach. JAMA (in press)
               evaluation  of  a  health  service  system,  namely,   to   facil-     6.  Williamson  JW,  Mitchell  JH,  Kreider   SD:   Outcomes   of
               itate  improvement,  the   results   of   our   experience   would         medical  care:  a  study  of  heart  failure  management   in
               lead  us  to  infer  that  the   ABCD   Strategy   is   definitely         emergency room patients. (Mimeographed)
                                                                                      7.  Vital Statistics of the United States 1967, Vol. 11'  Mortal-
               feasible and  probably  practical  for  this  purpose.  It  offers         ity, Parts A  and  B,  Washington,  DC,  US  Dept.  'of  HEW,
               an   approach   that   may   prove   superior   to   the   present         Public fiealth Service, Health  Services  and  Mental  Health
               haphazard  method   of   planning   patient   care   studies   and         Administration, National Center for Health  Statistics,  1969





























             224




                            EVALUATION OF NEW CATEGORIES OF NIANPOWER

                                                              Participants
     Harriet Kitzman                                                     Elinor Walker
     Department of Pediatrics                                            Public Health Analyst, Health Services Manpower
     Strong Memorial Hospital                                            Special Research and Development Projects Division
     University of Rochester                                             Nation Center for Health Services Research and Development
                                                                         Veronica L. Conley, Ph.D.
     Sarah Mazelas                                                       Head, Allied Health Section, Continuing Education and  Training
     Evaluation Director                                                  Branch
     California Regional Medical Program-Area I                          Regional Medical Program Service


           Evaluation of New Categories of Manpower                      evaluative tool. Objective criteria for rating  the  inter-
                                                                         action has not as yet been developed.
                       HARRIET KITZMAN                                   At  the  completion  of  the  course  a  written   examina-
                                                                         tion was  given.  This  test  examined  the  components  of
        With the  stress  to  prepare  new  health  professionals  to    patient care process: problem definition, plan with  inter-
        assist in  meeting  the  growing  health  care  needs.  comes    vention and  evaluation.  The  test  aimed  at  identifying
        the challenge  to  evaluate  the  new  health  professionals'    process  components  and  rating  the  components   accord-
        contribution and their impact  on  the  health  care  system.    ingly.
        The program which I  will  discuss  today  is  the  Pediatric    Another  method   of   evaluating   the   competencies   of
        Nurse   Practitioner   Program   at   the    University    of    the graduating, pediatric nurse  practitioners  dealt  with
        Rochester.   This   program,   formally   begun   in    1968,    the graduates' perception of their abilities after reaching
        prepares registered nurses  (primarily  those  who  have  not    the  work  setting.  Approximately  one  month  after   the
        been  employed  during  child  rearing)  to  provide   direct    nurses began practicing in  their  new  roles,  a  research
        patient  care  in  ambulatory  health  care   settings.   The    assistant interviewed each nurse to determine  the  nurse's
        settings  most  frequently  have  been  the  private   pedia-    judgment of her  abilities.  The  questions  developed  for
        tricians' offices.                                               the interview were  based  on  the  expected  behaviors  as
                                                                         defined in the course objectives.
         During  this  session  evaluation  of  the  pediatric  nurse     To  evaluate  the  impact  of  pediatric   nurse   practi-
        practitioners'  competencies  at  the  end  of  their  educa-    tioners  on  the  health  care  system  two  settings  were
        tional program will be discussed, as well as the  effects  of    used-a rural setting (a  community  with  a  population  of
        the care on the population being served.                         15,000) with no pediatrician and a Rochester suburb.
          With  precise   definition   of   expected   behaviors   in     Two  nurses  who  lived  in  the  rural   community   were
        measurable terms, the  task  of  evaluation  is  well  under-    prepared  as  pediatric  nurse   practitioners   and   then
        way.  Expected  behaviors   of   the   graduating   pediatric    established  as  well-child  care  providers  in  the  com-
        nurse  practitioners  were  defined  prior  to   the   course.   munity. Effects of the  pediatric  nurse  practitioners  on
        These  behaviors  were  known  to  students  as  the   course    the level of preventive health care services to children in
        began. Periodic discussion assisted  the  students  in  self-    the  community   and   the   physicians'   acceptance   are
        evaluation during the  course  while  curriculum  flexibility    presently  under  study,  Baseline   community   data   was
        enabled  the  students  to  find  learning  experiences  that    obtained  by  interviewing  mothers  of  all  children  who
        met  their  needs.   Clinical   preceptorship   allowed   for    were born in the area six and seven  months  prior  to  the
        continuous  feedback  to  students  and  faculty  during  the    pediatric nurse practitioners'  arrival  to  determine  the
        course.  Video-taped  patient  visits  allowed  students  and    level of well-chfld care the infants had received in  their
        preceptors  to  critically  review   student-patient   inter-    first months of life. The interview  included  a  question-
        action. This was used both  as  a  learning  measure  and  an    naire which  was  pretested  in  a  private  pediatrician's
                                                                                                                                 225





              practice where records were available.  One  year  after  the    pediatricians.  The  care  given  by  pediatric
              nurses  began  practicing  in  the  community  (both   in   a    practitioner-pediatrician teams was studied to
              well-chfld center and in general practitioners' offices)  the    mine quality, quantity, cost and acceptance to con
              population  then  six  and  seven  months  of  age  will   be    and professionals. A control group of patients
              studied  by  the  same  method.  By   comparing   the   group    by  pediatricians alone was  used. Chart  rev           ai
              studied prior to the  availability  of  the  pediatric  nurse    telephone  call  sampling  showed  the  number  and  purpose
              practitioners with the  group  studied  after  the  pediatric    of patient  contacts  and  visits  which  provided  informa.
              nurse practitioners  were  established,  the  impact  of  the    tion on the quality and quantity  of  care.  Total  cost  of
              pediatric  nurse  practitioners  on  the   total   preventive    the care given was  determined  by  an  accountant.  Patient
              health  services  to  children  will  be  determined.  Physi-    questionnaires   provided   information   about   the    ac.
              cians' acceptance of pediatric nurse practitioners  is  being    ceptance of care  given  to  the  experimental  and  control
              studied  by  use  of  interview  questionnaire  both   before    groups.  Individual  interviews  of  the  pediatricians  and
              pediatric  nurse  practitioners  were  established  and   one    pediatric nurse  practitioners  involved  provided  informa.
              year after they began practicing in the community.               tion  regarding  the  acceptance  of  the   new   role   and
                 The  study  involving  utflization  of   pediatric   nurse    relationships.
              practitioners   in   suburban   Rochester   private    pedia-       I have briefly described  the  methods  which  have  been
              tricians'  offices  began  in  July,  1968.  Four   pediatric    used to evaluate the pediatric nurse practitioner's  contri-
              nurse  practitioners  were  prepared  and  placed   in   four    bution to the health care of children.  Data  will  soon  be
              pediatricians' offices sharing the well-chfld care  with  the    available.





























              226







                                              TRAINING FOR EVALUATORS

                                                               Participants
    George E. Mflier, NI.D. - Moderator                                 Marion E. Leach, Ph.D.
    Director for Study of Medical Education                             Head,  Educational  Sciences  Section  Continuing  Education   and
    University of Illinois                                                 Training Branch
                                                                         Regional Medical Programs Service
    Donald Pochyly, M.D.
    Chief of Training, Center for Study of Medical Education
    University of Illinois


                         Summary of Session                                 2.  The  Training  of  the  Evaluator  to  function  within
                    MARIAN E. LEACH, PH.D.                                      the  Regional  Medical   Program:
                                                                                a.  he  needs  to  understand   the   Regional   Medical
                                                                                   Program objectives;
        Dr.  George  Miller  and  Dr.  Donald  Pochyly   addressed              b. he needs to understand health professionals;
     themselves to issues relating to the functioning  of  evalua-              c. he needs  to  learn  how  to  provide  leadership  in
     tors vis-a-vis Regional  Medical  Programs.  In  providing  a                 that setting.
     framework  for  discussion,  Dr.  Miller  described   briefly           In  the  discussion   that   followed   the   participants
     the Roles of the Evaluator and his need for training:                 seemed to indicate  that  their  perception  was  that  they
                                                                           were supposed  to  recruit  and  hire  "evaluators"  without
        1. The Evaluator as Trainer:                                       knowing  what  their  functions  were.  In  the  absence  of
            a. to train the staff of  a  Regional  Medical  Program        articulating  their  own  problems   in   evaluation   they,
               to understand and to use the evaluator:                     therefore, could  not  specify  the  kinds  of  competencies
                                                                           or resource needed.  The  group  agreed  that  it  would  be
           b. to generate the conviction  in  staff  that  evalua-         more  useful  under  the  circumstances  to  deal  with  the
               tion is a process, therefore  evaluation  should  be        concept  of  "evaluation"  rather  than  a   person   "eval-
                                                                           uator."
               an  integral  part  of  planning   and   implementa-
               tion; e.g.,  planning  for  evaluation  begins  with           A number of inquiries were         made   by   participants
               the specification of objectives;                            about  training  opportunities        in   evaluation.    Some
                                                                           indicated that they had not been      aware  of  the  resource
            c. to train staff in the use of  data  (a  prerequisite        represented  in  the  Center  for  Educational   Development
               to the effective use of evaluation).                        headed by Miller.










                                                                                                                                         227





                                    ATTENDANCE BY ORGANIZATIONAL AFFILIATION
           REGIONAL                                       Warner, Roger J.                               LeBrun, Jeanne
           MEDICAL PROGRAMS                               Director of Planning & Evaluation              Associate Coordinator, Area III
                                                          Arkansas Regional Medical Program              California Regional Medical Program
           Alabanw
                                                          Bi-State                                       Lloyd, John S., Ph.D.
           Joiner, Charles L., Ph.D.                                                                     Assistant Coordinator for Evaluation
           Director, Bureau of Research and               Overman, Ralph, Ph.D.                          California Regional Medical Program -
           Community Service                              Planning Director                              Area V
           Alabama Regional Medical Program               Bi-State Regional Nledical Program
                                                                                                         Nlaples, Lila, R.N.
           Packard, John Ni., M.D.                        Stoneman, William, 111, Ph.D.                  California Regional Medical Program
           Director, Alabama Regional Medical             Coordinator, Bi-State Regional
           Program                                        Medical Program                                Mazelis, Sarah
                                                          Ulsomer, Dennis                                Education-Evaluator - Area I
           Patterson, Douglas                             Project Manager, Bi-State                      California Regional Medical Program
           Evaluation Coordinator, Alabama                Regional Medical Program
           Regional Medical Program                                                                      Mykytew, Marion, M.D.
                                                          California                                     Associate Director-Area VII
           Albany                                                                                        California Regional Medical Program
           Brading, Paul L.                               Andrew, Barbara, J., Ph.D. Area V              Roush, Robert E. Ed.D.
           Educational Psychologist                       Assistant Professor Medical Education          California Committee of Regional
           Albany Medical College                         Research                                       Medical Programs-Area V
                                                          Division of Reserach in Medical Education
           Forer, Raymond, Ph.D.                          University of Southern California              Stroessler, John H.
           Assistant Coordinator of Evaluation            School of Medicine                             Assistant Coordinator, Area VIII
           Albany Regional Medical Program                Barnhfll,BruceNi.  AreaV                       California Regional Medical Program
           Albany Medical College                         Research Associate
                                                          Division of Research in Medical Education      Tesman, Bertram L., M.D.
           Arizona                                        Univer@ity of Southern California              Assistant Coordinator, Stroke Program
                                                          School of Medicine                             California Regional Medical Program-
           Ewy, Gordon A., M.D.                           Berkowitz, Sheldon  Area I                     Area VIII
           Cardiologist                                   Associate Specialist Project Evaluator
           Arizona Regional Medical Program               University of California                       Thompson, Jack E.
                                                                                                         Coordinator of Evaluation
           Flynn, John                                    Butts, David, Ph.D.  Area V                    California Regional Medical Program
           Chief of Operations, Arizona                   University of Southern California
           Regional Medical Program                       School of Medicine                             Central New York
           Humphrey, Alan, Ph.D.                          Combs, Robert C., M.D.                         Curry, Walter
           Assistant Coordinator for Evaluation           Coordinator, Area VIII                         Teaching Coordinator
           Arizona Regional Medical Program               California Regional  Medical  Program          Central New York Regional Medical Program

           Ivey, William                                  Elaimy, Wadie, Dr. P.H.
           Associate Coordinator, Arizona                 Assistant Coordinator, Planning &              Parresi, Anthony
           Re 'onal Medical Program                                                                      Teaching Coordinator
             91                                           Evaluation
                                                          California Regional Medical Program -          Central New York Regional Medical Program
           Arkansas                                       Area IV
                                                                                                         Schneider, Robert
           Rensch, Edward,                                Fielder, John, Ph.D.                           Instruction@ Communications Coordinator
           Deputy Coordinator, Arkansas                   California Regional  Medical  Program          Central New York Regional Medical Program
           Regional Medical Program                       Horovitz, Lee                                              r-
           Walter, Jacquelyn S., R.N-                     Associate Coordinator, California              Sovie, Margaret
           Nursing Coordinator, Arkansas                  Regional Medical Program - Area IV             Nursing Coordinator
           Regional Medical Program                       U.C.L.A. Rehabilitation Center                 Central New York Regional Medical Program
                                                                                                                                                 229




                      ColoradoliVyoming                              Georgia                                          lo iva
                      Aldridge, M. Gene                              Trantow,     Donald                              Caldwell, Charles W.
                      Continuing Education Division                  Georgia   Regional   Medical   Program           Associate Coordinator
                      Colorado/Wyoming   Regional    Medical                                                          Iowa   Regional   Medical   Program
                      Program                                        Greater  Delaware   Valley                       Latessa, Philip
                      Barrett, James, Ph.D.                          Roberts,  Dean  W.,   M.D.                       Director of Health Statistics
                      Continuing Education Division                  Chief, Regional Medical Program Activities       Iowa   Regional   Medical   Program
                      Colorado/Wyoming   Regional    Medical         Hahnemann Medical College
                      Program                                                                                         Tracy, Roger
                                                                                                                      Director   of   Project   Development
                      Dyson, James E., Ph.D.                         Hawaii                                            and Evaluation
                      Associate   Director,   Colorado[Wyoming       Denney, Ruth                                     Iowa   Regional   Medical   Program
                      Regional   Medical   frogram
                      Continuing Education Division                  Chief of Planning and Research Services          Weinberg, Harry B., M.D.
                                                                     Hawaii   Regional   Medical    Program           Coordinator, Iowa Regional
                      Gough, Anne, R.N.                              Hasegawa,   Masato,   M.D.                        Medical Program
                      Chief of Nursing and Allied Health
                      Colorado/Wyoming   Regional    Medical         Program   Coordinator   and   Director
                      Program                                        Hawaii   Regional   Medical    Program           Kansas
                      Hastirigs, William 0.                          Tunks,   Omar    A.                              Adair, Charles, Ph.D.
                      Chief, Project Audit and Control               Chief of Operations                              Assistant  Coordinator,   Research   and
                      Colorado/Wyoming   Regional    Medical         Hawaii   Regional   Medical    Program            Evaluation Department
                      Program                                                                                         Kansas   Regional   Medical   Program
                                                                     fllinois
                      Stubblefield, Rex D.                                                                            Brown, Robert W., M.D.
                      Executive Assistant                            Auerbach,     Harry                              Coordinator, Kansas Regional
                      Colorado/Wyoming   Regional    Medical         Assistant Director for Research                   Medical Program
                      Program                                        Illinois  Regional   Medical   Program
                                                                                                                      Petre, Richard, PhJ).
                      Syner, James C., NI.D.                         Brown, Patricia R., R.N., Ph.D.                  Assistant Evaluation Director
                      Associate Director for Project                 Assistant Director                               Kansas   Regional   Medical   Program
                      Administration  &   Health   Information       Illinois  Regional   Medical   Program
                      Division                                                                                        Taliaferro, J. Dale, Ph.D.
                      Colorado/Wyoming   Regional    Medical         Creditor, Morton C., M.D.                        Director  of  Social  Systems   Research
                      Program                                        Executive Director                               Research and Evaluation Department
                                                                     Illinois  Regional   Medical   Program           Kansas   Regional   Medical   Program
                      Connecticut                                    Lovelace, Bryan, Jr.
                      Clark, Henry T., Jr., M.D.                     Assistant Executive Director                     Louisiana
                      Director  and  Coordinator,  Connecticut       Illinois  Regional   Medical   Program
                      Regional   Medical   Program                                                                    Baird, Beverly
                                                                                                                      Planning Staff Associate
                      Glasgow, John, PhJ).                           Indiana                                          Louisiana   Regional   Medical   Program
                      Associate Coordinator  for  Evaluation         Svan,  John,   Ph.D                              Walker, Karen, M.P.H.
                      Connecticut  Regional  Medical   Program       Director of Educational Services                 Staff Evaluator
                                                                     Indiana   Regional   Medical   Program           Louisiana   Regional   Medical   Program
                      D.C. Metropolitan
                                                                     Intermountain
                      F7orida                                                                                         Maine
                                                                     Schorow, Mitchell
                      Ausman, Robert K., M.D.                        Assistant  Coordinator  for   Planning           Doran, Peter, Ph.D.
                      Deputy Director, Florida Regional                & Evaluation                                   Associate Coordinator
                      Medical Program                                Intermountain Regional Medical Program           Maine   Regional   Medical    Program

                      Engebretson, Gordon R., Ph.D.                  Smart,  Charles  M.,  M.D.                       Jones, Janet H.
                      Associate Director,  Florida                   Director, Tumor Registry                         Director,  Grant   Program   Policy
                      Regional   Medical   Program                   Intermountain Regional Medical Program           Maine   Regional   Medical    Program

                      230






     Maryland                                          Lampton, T.D., NI.D.                             New Jersey
                                                       Coordinator for  Heart  Disease  and  Stroke
     Herbert,  Henry,   NI.D.                          Mississippi Regional Medical Program             Harkness, James P., Ph.D.
     Associate Coordinator for Continuing                                                               Deputy Coordinator
       Education                                       ivrissouri                                       New Jersey Regional Medical Program
     Maryland   Regional   Medical    Program
     McMurrin, Vern                                    Donnell, Denny, M.D.                             New Mexico
     Associate Coordinator for Evaluation              Chief, Program Methodology Unit
       and Planning                                    Nlissouri Regional Medical Program               Fitz, Reginald H., M.D.
     Maryland   Regional   Medical    Program                                                           Director, New Mexico Regional
                                                       Rikli, Arthur E., Ni.D.                           Medical Program
                                                       Coordinator, Missouri Regional
     Memphis                                            Medical Program                                 Griffith, Dudley
                                                                                                        Health Planning Technologist
     Amis,  Lewis  N.,  Ph.D.                          Sights, Warren P., IND.                          New Mexico Regional Medical Program
     Chief of Planning, Research and                   Director for OperationE
       Evaluation                                      Missouri Regional Nledical Program               Key, Charles R., M.D.
     Memphis    Regional    Media     Program                                                           Assistant Director for Cancer
                                                       Mountain States                                  New Mexico Regional Medical Program
     Culbertson,  James  W.,   M.D.
     Coordinator, Regional Medical Program             Smith, C. Eddar, Ph.D.                           New York Afetropolitan
                                                       Coordinator, Planning and Evaluation
     Miller, Edward                                    Mountain States Regional Medical                 Aronson, Jesse, M.D.
     Systems   Analyst,   Memphis    Regional           Program                                         Deputy Director, New York Metropolitan
       Medical Program                                                                                   Regional   Medical    Program
                                                       Nassau-Suffolk                                   Eller, John
     Michigan                                                                                           Evaluation Specialist, New York
                                                       Beckman, Robert                                   Metropolitan Regional Medical Program
     Ballinger, David                                  Director of Research
     Michigan   Heart   Association                    Nassau-Suffolk  Regional   Medical   Program
                                                                                                        North Carolina
     Brickner, Abraham                                 Hastings, Glen E., M.D.
     Executive Director                                Program Coordinator                              Hallman, Shannon P.
     Michigan   Heart   Association                    Nassau-Suffolk  Regional   Medical   Program     Associate Director, Social Research
                                                                                                         Section
     Larocque, Gaetane M., Ph.D.                       Owen, Harrison                                   University of North Carolina
     Associate Program Coordinator                     Assistant Coordinator for Project
     Michigan   Regional   Medical    Program           Development                                     Truscott, B. Lionel
                                                       Nassau-Suffolk  Regional   Medical   Program     Director, Stroke Program
     Lopushinsky, Theodore, Ph.D.                                                                       North Carolina Regional Medical Program
     Program Developer                                 Rothbell, Gladys
     Michigan   Regional   Medical    Program          Nassau-Suffolk  Regional   Medical   Program     Weaver, Ben F.
                                                                                                        Deputy Director, North Carolina
     Lyons, James                                      NebraskalSouth Dakota                             Regional   Medical    Program
     Michigan State University
     Welke,   Wm.   Graham,   Ph.D.                    Marcy, Deane S., NI.D.                           North Dakota
     Detroit, Michigan                                 Project Administrator, Coronary Care
                                                       Nebraska/South   Dakota   Regional   Medical     Dietz, Conrad
                                                        Program                                         Computer Service, North Dakota
     Mississippi                                                                                         Regional   Medical    Program
                                                       Morgan, Harold S., M.D.
     Bridgeforth,  Edwin   B.                          Program Coordinator, Nebraska/South              Parker, Lorraine
     Program Evaluator, Department of                   Dakota Regional Medical Program                 Assistant Director, Continuing
       Preventive    Medicine                                                                            Education for Allied Personnel
     Mississippi Regional Medical Program              Morris, George L., Jr.                           North Dakota Regional Medical Program
                                                       Project Administrator, Communications
     Campbell, Guy D., M.D.                             Facility                                        Wright, Willard A., M.D.
     Coordinator, Mississippi Regional                 Nebraska/South   Dakota   Regional   Medical     Program Director, North Dakota
       Medical Program                                  Program                                          Regional   Medical    Program

                                                                                                                                                 231




                 Northeaster.-i  Ohio                              Rin-.e, Robert, Ph.D.                            South Carolina
                                                                   Director of Communications
                 Johnson,   Bart                                   Ohio  State  Regional   Medical   Program        Bowman, C.W.
                 Director  of  Education                                                                            Associate Coordinator for Medical
                 Northeast Ohio Regional Medical Program                                                             Districts Program Planning
                                                                   Ohio Valley                                      South Carolina Regional Medical Program
                 Lifson,  Arthur
                 Assistant Director, Heart Disease                 Cook,Anne                                        Coleman, J. Walker, III
                 Northeast Ohio Regional Medical Program           Research Associate                               Director, Field Services, Planning
                                                                   Ohio  Valley  Regional  Medical   Program         and Evaluation
                 Meloy,  Richard   C.                                                                               South Carolina Regional Medical Program
                 Director, Administration                          McBeath, William H., M.D.
                 Northeast Ohio Regional Medical Program           Director, Ohio Valley Regional
                                                                   Medical Program                                  Susquehanna Valley
                 Northern New England                              Moss, Judson, Ph.D.
                 Danielson,    Donald                              Educational    Consultant                        Hoffman, John D.
                 Associate Director, Northern New                  Program       Development                        Chief of Internal Operations
                                                                                                                    Susquehanna Valley Regional Medical
                 England Regional Medical Program                  Ohio  Valley  Regional  Medical   Program         Program
                 University of Vermont College of Medicine
                                                                                                                    NicKenzie, Richard B.
                 Stewart,  Caryl                                   Oklahonia                                        Director, Susquehanna Valley Regional
                 Assistant Director for Program                    Bexfield, Frank W.                                Medical Program
                 Development                                       Associate  for  Evaluation  and  Planning
                 Northern New England Regional                     Oklahoma   Regional    Medical    Program        Taylor, David
                 Medical      Program                                                                               Coordinator of Research and Evaluation
                                                                   Cameron, Charles M., M.D.                        Susquehanna Valley Regional Medical
                 Wennberg,  John,   M.D.                           Professor  and   Chairman                         Program
                 Director, Northern New England                    Department   of   Health   Administration
                 Regional Medical Program
                 Northlands                                        Oregon                                           TennesseelMid-South
                 Hill, Russell N., Ph. D.                          Kole,  Delbert  M.,  M.D.                        Metcalfe, Robert M., M.D.
                 Evaluation   Officer                              Coordinator for Community Organization           Associate Director, Tennessee/Mid-Soutn
                 Northlands  Regional   Medical   Program          Oregon Regional Medical Program                   Regional Medical Program
                 Miller, Winston R., M.D.                          Yagi, Kan, Ph.D.                                 Zubkoff, Michael, Ph.D.
                 Program Director and Coordinator                  Consultant for Education and Evaluation          Head, Medical Economics Section
                 Nortlflands  Regional  Medical   Program          Oregon Regional Medical Program                  Tennessee/Mid-South Regional
                                                                                                                     Medical Program

                 Northwestern    Ohio
                                                                   Puerto Rico
                 Jenkins,  Keith                                                                                    Texas
                 Northwestern   Ohio   Regional   Medical          Mirando, Luis, M.D.
                 Program                                           Puerto  Rico  Regional  Medical   Program        Ferguson, David K.
                                                                                                                    Operations Officer
                 Tittle, C. Robert, Jr., M.D.                      Mullan, Bryan                                    Texas Regional Medical Program
                 Program Coordinator, Northwestern Ohio            Puerto  Rico  Regional  Medical   Program
                 Regional Medical Program                                                                           Humble, Robert 0.
                                                                                                                    Chief of Planning and Evaluation
                 Ohio State                                        Rochester                                        Texas Regional Medical Program
                 D'Costa,  Ayres,  Fh.D.                           Haynie, Gloria                                   McCall, Charles B., M.D.
                 Director, Fellowship Program                      Statistical  and  Evaluation  Unit  Staff        Coordinator, Texas Regional
                 Ohio  State  Regional  Medical   Program          Rochester   Regional   Medical    Program         Medical Program
                 Pace, William G., 111, M.D.                       Jacobs, Arthur, M.D., M.P.H.                     Reese, Hubert D.
                 Program     Coordinator                           Director, Statistical and Evaluation Unit        Data Acquisition Specialist
                 Ohio  State  Regional  Medical   Program          Rochester   Regional   Medical    Program        Texas Regional Medical Program

                 232





      Tri-State                                              l@'estern IVew York                                    Lange, No
                                                                                                                    Nursing c
      Colt, Avery                                            lnpll, John R. F., ?Y[.D.                              Wisconsin
      Tri-State    Regional    Medical    Program            Pro-.ram      Director
                                                             @@'estern New York Regional Medical                    I-eiiikc,  CbitICS   W.
      Farrisey, Claire                                        Program                                               Directoroft@,.v3lu&lion
      Special Projects Coordinator                                                                                  Wisconsin Rcgional Mojic
      Tri-State   Regional    Ntedical    Program            Kaye, Leonard
                                                             N@'estern New York Regional Nledical                   Slicele)" %%'jlli3m  J.
      Kcairnes, Harold, M.D.                                  Program                                               Assistant Coordinstat
      Coordinator for Evaluation                                                                                    Wisconsin RegionA Nlod"
      Tri-State    Regional    Medical    Program            Kellber-., Elsa R.
                                                             Associate for Assessment and Research
      Mott, Ruth                                             N@'estern New York Regional Medical                    NATIONAL
      Research Associate for Data Collection                  Program                                               R.NIP        REVIEW         COMMITM
      Tri-State    Regional    Medical    Program
                                                             l@'estern Pennsylvania
      Peterson, Osler L., NI.D.                                                                                     lzevie%%. Cofiilnittcc %leriber
      Associate Director for Data Collection                 Carpenter,   Robert   R.,    M.D.                      Los Angeles. Ctitfoinii
      Tri-State    Regional    Medical    Program            Director, Western Pennsylvania
      Department    of    Preventive     Medicine             Regional     Medical     Program                      Leiiion, Henry M., SI.D.
      Harvard Medical School                                                                                        Professor of,%Wi,.irK
                                                             Lapenas,       CoraLee                                 Del).utnient of lntetnAl Medidne
      Siebert, Dean J., NI.D.                                Research Assistant, Western Pennsylvania               'nic University of Ncbrxsks Nledicgl Center
      Associate Coordinator for Dartmouth                     Regional     Medical     Program                      Omaha, Nebrask3
       Medical School
      Tri-State    Regional    Medical    Program                                                                   Lewis, Edmund J., NI.D.
                                                             Reed, David E., M.D.
                                                             Associate Director for Evaluation                      Chief, Naval Divi@n
      VirgIinia                                              Western Pennsylvania Regional Medical                  Tliorndike Nlcnioriii Uborstofy
                                                              Program                                               Boston, Nfa&,;achusctts
      Mason, Jack L., Ph.D.                                                                                         Sclimidt,AlexaMerNi..&I.D.
      Education Sciences Officer                             Scheuer, Ruth
                                                             Assistant to Director of Evaluation                    Dean, Abraham Lincoln School of Medicine
      Virginia Regional Medical Program                      Western Pennsylvania Regional Medical                  University of Illinois
                                                                                                                    Chicago, Illinois
      Proctor, Margaret W.                                    Program
      Survey Officer                                                                                                Sister Ann Josephine, C.&C., PhD.
      Virginia Regional Medical Program                      lVest Virginia                                         Administrator
                                                                                                                    Holy Cross Hospital
      WashingtonlAlaska                                      Costello, Joseph                                       Salt Lakc City, Utah
                                                             Biostatistician, West Virginia
      Dohner, Charles W., Ph.D.                               Regional     Medical     Program                      NATIONAL
      Washington/Alaska Regional Medical School                                                                     RMP        ADVISORY         COUNCIL
      University   of   Washington   School    of            Gallina, Peter P.
       Medicine                                              Coordinator of Field Services
                                                             West Virginia Regional Medical Program                 Cannon, Bland, M.D.
      Duren, Gay                                                                                                    Niempliis, Tennessee
      Assistant Director for Evaluation                      Holland, Charles D.
      Washington/Alaska Regional Medical                     Director, West Virginia Regional                       Wykcoff,       Florence
       Program                                                Medical       Program                                 Watsonville, California

      Johnson, William R., Ed.D.                                                                                    REGIONAL
      Washington/Alaska Regional Medical                     lvisconsin                                             MEDICAL PROGRAMS SERVICE
       Program                                                                                                      ROCKVILLE,                 MARYLAND
      University   of   Washington   School    of            Hirschboeck,   John   S.,    M.D.
       Medicine                                              Coordinator, Wisconsin Regional
                                                              Medical       Program                                 Abrams, Rhoda
      Larter, Mariella, M.S.                                                                                        Assistant Branch Chief
      Coordinator, Coronary Care Unit Training               Kraegel, Janet, R.N.                                   Evaluation       Branch
      @Vashington/Alaska Regional Medical                    Nurse Utilization Project Director                     office of Program Planning and
       Program                                               Milwaukee, Wisconsin                                        Evaluation

                                                                                                                                                                 233




                 Conley I Veronica L., Ph.D.                      Leach, Marian E.,  Ph.D.                           Schoen, Teresa
                 Head, Allied Health Section                      Head, Educational Sciences Section                 Special Assistant
                                                                                                 nin-     Branch     Office of Program Planning and Evaluation
                 Continuing Education and Trainin@ Branch         Continuing Education and Trai
                 Conrath, Cecilia C.                              Nlargulies, Harold, M.D.                           Witte, Lawrence M.
                 Chief, Continuing Education and                  Acting Director                                    Program Planner
                 Training Branch                                                                                     Office of Program Planning and Evaluation
                                                                  Mark, Frank R., M.D.
                 Dana, Mary L.                                    Chief, Operations Research and Systems
                 Secretary                                        Analysis Branch                                    OTHER
                 Office of Program Planning and Evaluation                                                           FEDERAL AGENCIES
                                                                  McDonald, Carol B.
                 de la Puente, Joseph                             Secretary                                          Askew, Cornelius D.P.H.
                 Chief, Program Studies Branch                    Office  of  Program  Planning  and  Evaluation     Director, Health Services Evaluation
                 Kidney Disease Control Branch                                                                       Branch
                                                                  Mullins, Patty S.                                  National Center for Health  Services
                 Dickenson, EMBY                                  Program Analyst                                    Research and Development, IISMHA
                 Travel Clerk                                     Office  of  Program  Planning  and  Evaluation
                 Division of Travel                                                                                  Blaxell, Martha
                                                                  Munier, William B., M.D.                           Budget Examiner
                 Dunning, Herbert P.                              Staff Assistant                                    Officer of  Management  and  Bud-,et
                 Program Management Officer                       Continuing  Education  and   Training   Branch
                 Division of Professional and  Technical                                                             Brandwein, Raymond A.
                 Development                                      Nelson, Elsa J.                                    Program Analyst
                                                                  Health Services  Officer                           National Center for Health Statistics,
                 Ensor, Joan E.                                   Continuing  Education  and   Training   Branch     HSMHA
                 Program Analyst, Office of Program               O'Flaherty,  Harold   F.                           Campana, Edward
                 Planning and Evaluation                          Program Analyst                                    Program Analyst
                 Frampton, Rita P.                                Office  of  Program  Planning  and  Evaluation     Comprehensive Health Planning, HSMHA
                 Public Information Specialist                    Peterson, Roland L.                                Featherstone, Frederick V., M@D.
                 Office  of  Communications  and  Public          Assistant Director                                 Special Assistant to the Deputy
                 Information                                      office  of  Program  Planning  and  Evaluation     Administrator
                                                                                                                     HSMHA-Office of theadministrator
                 Friedlander, Edward M.                           Price, Kathy
                 Assistant Director                               Secretary                                          Fox, Peter, Ph.D.
                 Office  of  Communications  and  Public          Office of Administrative Management                Senior Economist
                  Information                                                                                        Office  of  Management  and   Budget
                                                                  Quave, Robert L.
                 Green, Dorothy E., Ph.D.                         Administrative   Officer                           Gorzkiewicz, Marjorie
                 Chief, Program Research Section                  Officer  of   Management
                 National Clearinghouse for Smoking                                                                  Project Manager, Health Services
                                                                                                                     Evaluation Branch
                 and Health                                       Resnick, Leah                                      National Center for Health Services
                                                                  Program Analyst                                    Research    and    Development,     HSMHA
                 Ichniowski, Francis C.                           Office  of  Program  Planning  and  Evaluation
                 Acting   Director,   Systems   Management   Office                                                  Logsdon, Donald N., M.D.
                                                                  Ringel, Abraham                                    National Center for Health Services
                 Kaplan, Alan S., M.D.                            Public'Health    Analyst                           Research    and    Development,     HSMHA
                 Deputy  Chief,   Continuing   Education          Operations  Research  and   Systems   Analysis
                 and Training Branch                               Branch                                            Walker,Elinor
                                                                                                                     Public Health  Analyst,  Health  Services
                 Klieger, Philip A., M.D.                         Robins, Morton                                     Manpower
                 Assistant Director-Stroke and                    Acting Chief                                       National Center for Health Services
                 Rehabilitation                                   Study Design and Analysis Staff                    Research    and    Development,     HSMHS
                 Office of organizational Liaison
                                                                  Says, Luther J., Jr.                               Yordy, Karl D.
                 Larson, Carol M.                                 Public  Health   Advisor                           Assistant Administrator,  Office  of
                 Allied Health Specialist                         Grants   Review   Branch                           Program Planning and Evaluation-HSMHA
                 Continuing Education and Training Branch


                 234





        OTIIER                                            Jeffers, James  R.,  Ph.D.                         Pochyly, Donald, M.D.
                                                          Director, Medical Economics                        Chief of Training
        Bauer, Katherine G.                                Research      Center                              Center for Study of Medical Education
        Research Associate, Harvard Center                Department    of    Economics                      Office of Research in Medical Education
        for Community Health and Medical Care             University  of   Iowa                              University of Illinois Medical Center

        Berg, Robert, M.D.                                Kelman,   Howard,    Ph.D.                         Rockwell, M.A., M.D.
        Professor and chairman                            Department of Preventive Nledicine                 Director, Rand Health Program
        Department of Preventive Medicine                  and  Public   Health                              Rand Corporation
        and Community Health                              New  York   Medical   College
        School of Medicine                                                                                   Schon, Donald A., Ph.D.
        University of Rochester                           Kimmel,     Wayne                                  President, Organization for Social and
                                                          Department    of    Economics                      Technological Innovation
        Blamphin, John M.                                 Public   Service   Laboratory                      Seipp, Conrad, Ph.D.
        Assistant Bureau Chief-Washington Office          Georgetown University Hospital
        Aledical IVorld News                                                                                 Associate Professor, Health
                                                          Krinneruch, Margery                                Services Research Center
        Crawford, William R., Ed.D.                       Cancer    Program                                  University  of  North   Carolina
        Associate, Evaluation Studies Section             American College of Surgeons                       Shapiro, Sam
        Center for Educational Development                                                                   Director of Research and Statistics
        College of Medicine                               Kitzman, Harriet                                   Health Insurance Plan of Greater New York
        University of Illinois                            University  of   Rochester                         Shepherd, Rodger M., M.b.
        Deverin, Daniele                                  Lewis, Charles  E.,  NI.D.                         Assistant Program Director
        Cybern Education, Inc.                            Professor  and   Head                              Continuing Education
                                                          Health Administration Division                     Presbyterian Medical Center
        Dunl ap, James Ph.D.                              School  of  Public  Health
        Arthur D. Little, Inc.                            University  of  California                         Tobis, Jerome, M.D.
                                                                                                             Chairman, Department of Physical
        Engbretson, William, Ph.D.                        Linden,    George                                  Medicine   and    Rehabilitation
                                                                                                             University of California
        President, Governors State University             Chief, California Tumor Registry                   Irvine College of Medicine
                                                          Department of Public Health
        Fleisher,Daniel, M.D.                                                                                Webber, Willard W.
        Director  of  Health  Professions                 Maxwell, Stephen L,, The Honorable                 Cancer Program Advisor
        Temple University                                 Jud-e of the  District  Court                      American  College  of   Surgeons
        Forsyth, Jeannette, Ph.D.                         St.  Paul,  Minnesota                              NVeckwerth, Vernon, Ph.D.
        Project Administrator                                                                                St. Paul, Minnesota
        Information Support System                        Mayer, Andrew, M.D.                                (Niember, Northlands Regional Advisory
                                                          Cancer    Program                                  Group)
        Gavett, J. William, Ph.D.                         American College of Surgeons                       White, Paul, Ph.D.
        Associate Professor,  College  of                                                                    Division of Behavioral Sciences
        IManagement                                       Metzner,  Charles  A.,  Ph.D.                      Johns Hopkins University
        Department of Preventive Medicine                 Professor, Department of Medical
        and Community Health                              Care     Organization                              Wflbur,.Richard S., M.D.
        School of Medicine                                School  of  Public  Health                         Assistant Executive Vice President
        University of Rochester                           University   of   Michigan                         American   Medical   Association
        Gustafson, David H., Ph.D.                        Miller,  George  E.,  M.D.                         Williamson, John W., M.D.
                                                                                                             Professor, Department of Medical Care
        Assistant Professor                               Director, Center for Study of Medical              and Hospital School of Hygiene and
        Industrial  Engineering  Division                 Education                                          Public Health
        University of Wisconsin                           Office of Research in Medical Education            Johns Hopkins University
                                                          University of Illinois Medical Center
        Hagedorn, Homer, Ph.D.                                                                               Wylie, Charles M., M.D.
        Arthur D. Little, Inc.                            Miller, Irving,  M.D.                              Professor, Public Health Administration
                                                          Evaluation    Technologist                         University of Michigan
        Henderson, Maureen, M.D.                          Albuquerque,    New    Mexico
        Pro'fessor,   Preventive   Medicine                                                                  Youngennan, Robert A.
        Department of Preventive Medicine                 Peterson, Sara J.                                  Interregional    Representative-
        and Rehabilitation                                Biostatistician                                    Georgia Regional Medical Program
        University of Maryland                            Fabius,   New    York                              Medical Association of Georgia

           U. S. GOVERNMENT PRINTING OFFICE: 1972 0 - 416-831                                                                                        235








































                                 DHEW Publication No. HSM-71-7010