Newsgroups: bit.listserv.stat-l
Date:         Tue, 18 Aug 1992 08:44:35 -0400
Reply-To:     pgill@kean.ucs.mun.ca
From:         pgill@KEAN.UCS.MUN.CA
Subject:      Is this a good agreement - summary

   Thanks to many people who responded to my question regarding
agreement among a panel of experts.  I am posting all the
responses along with my original query.  I believe there was an
error in the researcher's computation of Cohen's Kappa.  I will
pass all the comments to the researcher.

P. S. Gill
Memorial Univ of Newfoundland
St. John's, Canada

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

ORIGINAL QUESTIONS

   A researcher has developed a questionaire containing 25 questions.
Each question refers to an action in a given situation.  The
respondent would answer whether the action is appropriate (1)
or not (0).  Before sending the questionaire to the field, the
researcher wanted to test the questionaire on a panel of 5 experts
including herself.  The following are the responses of the experts,
expert 1 being our researcher :

Expert \        R E S P O N S E S  TO  Q U E S T I O N S
_____________________________________________________________________

  1        0 0 1 0 1  1 0 1 1 0  1 1 0 0 1  1 0 1 0 0  0 1 1 0 1

  2        0 0 1 0 1  1 1 1 1 0  0 1 0 0 1  1 0 1 0 0  0 1 1 0 1

  3        0 0 1 1 1  1 0 1 1 0  1 1 0 0 0  1 1 1 0 0  0 1 1 0 1

  4        0 0 1 0 1  1 0 0 1 0  1 1 0 0 1  1 0 1 0 0  0 1 1 0 1

  5        0 0 1 0 1  1 0 1 1 0  1 1 0 0 1  1 0 1 0 0  0 1 1 0 1
_____________________________________________________________________

   The following table the number of times (out of max 25) a pair of
experts agree on the appropriateness of the action :

  EXPERT          1       2      3      4     5
  --------------------------------------------------------
    1             -      23     22     24     25
    2                     -     20     21     23
    3                            -     21     22
    4                                   -     24
  --------------------------------------------------------

   It is reasonable to assume
   Prob{ two given experts agree on a given question by chance} = 0.5.
   The Binomial probablity of more than or equal to 20 (pure chance)
   agreements between two given experts is (approximatley) 0.0026.

  QUESTION 1. Does there seem to be a good agreement among the five
  experts ?  Moreover, all five experts agree 19 out of 25 times.
  The Binomial probablity that all five experts agree by chance on
  a given question is (.5)**5 = 0.03125.  Therefore, the probablity
  that all five experts agree (by pure chance) 19 or more times out
  of 25 times is almost 0.

  QUESTION 2.  The researcher tried to use Cohen's kappa as an
  indicator of agreement among the experts and found it to be
  negative.  Is Cohen's kappa approprite for binary agreement ?

 Any comments are welcome.

P. S. Gill
Dept of Maths & Stats
Memorial Univ of Newfoundland
St. John's   CANADA

THE COMMENTS FROM STAT-L-ERS

From:   MX%"UPHILG@UNC.BITNET" 27-JUL-1992 15:07:54.94


Dear Prof. Gill,
    You bring up good points (of course, that means "points that
    I, also, have worried about for some time).  I respond not with
    answers, but, rather, with amplifications of your questions.

>   QUESTION 1. Does there seem to be a good agreement among the five
>   experts ?
        Well, yes, of course there does.  It is very difficult to
        imagine any fair set of reasonable questions to which N
        competent answerers would respond without good agreement.
        (I omit from this discussion questions dreamed up by some
        psychologist with the intent of being deliberately ambiguous
        and unanswerable - questions like that can usually be weeded
        out by techniques simpler than Cohen's kappa, i.e., common
        sense.)  (Also omitted are studies in which the answerers are
        known to be not competent to answer the questions.)  Especially
        if (for more-than-binary responses) some account is taken of
        the "nearly agreed" answers.  For binary answers I always refer
        to Dana Quade's work on concordant/discordant pairs - I haven't
        thought this one out yet, but I bet that some augmentations
        of Kendall's tau (like Quade's Matched Pairs concept) could
        give a reasonable analysis.  However, my main point is that,
        given data such as you present, no thinking person should
        believe a test that said these data are no better related
        than by chance.  What is really wanted is something that gives
        a good assessment of how strong the relationship is (not
        just a p-value.)
        Needless to say, I guess neither of us is very happy with
        the binomial probability approach, because of the lack of
        independence.  Or have you figured out a good rationalization
        to satsify the naysayers?

>   QUESTION 2.  The researcher tried to use Cohen's kappa as an
>   indicator of agreement among the experts and found it to be
>   negative.  Is Cohen's kappa approprite for binary agreement ?

        Well, probably, but one of the difficulties of Cohen's
        kappa is that it doesn't have nice descriptive
        properties like going smoothly from -1 to 0 to +1.  Kant
        Bangdiwala has published a nice discussion this in the
        paper that presents his reliability statistic.  (I don't
        have any problem with a negative Cohen's kappa, but I wish
        you would ask the researcher to check the calculations -
        I have a sick feeling that something has gone badly wrong
        with the calculations - I think they should all be quite
        positive.  And with a gut the size of mine, my gut feelings
        usually carry a great deal of weight.)

    To sum up:
    1.  These data are obviously "reliable" - they pass the Gary
        Koch IOT test (InterOcular Trauma test - they hit you
        right between the eyes).  We note that Cohen's kappa
        is designed for two raters only, and that the lack
        of independence (given the five raters) most likely louses
        up any kind of inference one might have made using it.
        Some kind of concordant/discordant pair analysis that
        takes into account the lack of independence in the
        data is called for.

    2.  Without having seen what the researcher did, it is more than
        a little presumptuous of me to suggest that whatever was
        done in the attempt to get a kappa was wrong, but I fear
        that is exactly the case.  The correlations look strongly
        positive to the eye, and, besides, the textbook kappa doesn't
        allow for N>2 raters. (I wish I had the SPSS manual handy, maybe
        the RELIABILITY write up documents something much newer than
        my ancient textbooks;  if so, perhaps there is no surprise
        about the negative values, because what is reported would
        clearly be something quite different from the twenty year
        old Cohen's kappa for two raters.)


        If you get any earth-shatteringly good answers on this, please
        let me know.  I also have a paper with five raters that I am
        trying to write.

                             Phil Gallagher
                             UPHILG@UNC




~L

        Regarding P. S. Gill's "good agreement" question, I have some
        non-statistical questions:

        As stated, the point of submitting the questionnaire to 5
        experts was to "test" the questionnaire.  But, I'm wondering,
        *IF* the objective of the questionnaire is attitude assessment,
        what could be inferred from agreement OR disagreement among N=5?
        These 5 people may very well tend to have the same assessments
        as to appropriateness of the actions presented insofar as they
        are, at best, a (non-random?) sample of the population of
        experts.  Alternatively, if they disagree, does this imply that
        the questionnaire somehow fails to pass this "test" or merely
        that the questionnaire is very good at presenting situations for
        which no wide consensus exists?

        Kernon Gibes
        Internet: gibes@swirl.monsanto.com
~L
From:   MX%"FDANE@uga.cc.uga.edu" 27-JUL-1992 16:19:46.36

IMHO, it would be easier to use correlations (tetrachoric) to assess
inter-expert reliability.  With N = 5, pay more attention to the
absolute value of the coefficient than to its _p_ value.

Frank Dane, a.k.a
Dr. Francis C. Dane, Chair
Department of Psychology
Mercer University
Macon, GA  31207-0001  USA
FDANE@UGA or FDANE@UGA.CC.UGA.EDU
voice: (912) 752-2972
FAX:  (912) 752-2108
~L
From: KAPLON@TOE.TOWSON.EDU

Did the researcher try Kendall's Coefficient of Concordance?

Howard

-----------------------------------------------------------------------
| Howard S. Kaplon                    |  Mathematics Department       |
| BITNET:  Kaplon-H@TOWSONVX.BITNET   |  Towson State University      |
| Internet:  Kaplon-H@TOE.TOWSON.EDU  |  Towson, Maryland  21209-7097 |
| Phone:  (410) 830-3087              |  FAX: (410) 830-2604          |
-----------------------------------------------------------------------

~L
From: "Hoi K. Suen (814) 865-2235" <HKS1@PSUVM.PSU.EDU>
 Educational Psychology, 230 CEDAR Bldg.
 The Pennsylvania State University
 University Park, PA  16802

Cohen's kappa is particularly appropriate for binary and polytomous data,
but not ordinal data.  How did your colleague compute kappa?  I quickly
ran the responses from experts 1 and 2 through kappa and found it to be
0.84.  Given the high level of apparent agreements across the 5 experts
and further given that the marginals are close to 0.5, I would expect
all kappas will be high positive.  Perhaps the problem is in how the
contingency table was put together for the computation of kappa?
~L
From: MAJ@waikato.ac.nz

I have a feeling that information is lost in this kind of data
compression.


I would model the data using finite mixtures. The 25 items are assumed
to be from a mixture of two populations, conditional of the
subpopulation [damn! *on* the subpopulation] the responses of the
5 experts are independent. This is latent class analysis. The
mixing proportion and the conditional response probabilities seem to
me to be more meaningful parameters than any index of association.
For much larger sets of data more than two components may be
included in the model. Estimation is by EM.

--
Murray A. Jorgensen [ maj@waikato.ac.nz ]    University of Waikato
Department of Mathematics and Statistics     Hamilton, New Zealand
__________________________________________________________________
~L
From : dds@leicester.ac.uk
 David D Stretch
 Lecturer in Psychology, Dept. of Psychiatry,
 Clinical Sciences Building, Leicester Royal Infirmary,
 PO Box 65, Leicester, LE2 7LX, United Kingdom.


Brian Everitt has done work on this. I don't have the exact reference to hand,
but it is to do with the agreement and disagreement of observers about
diagnistic signs. He works out measures of agreement and disagreement and
derives large-sample (alas) tests of significance, although he gives
enough information for one to perform exact tests on samll sample sizes if
one is prpared to do a little work (I think it involves manipulation of
the hypergeometric distribution, though I could be wrong). The paper
appeared in the British Journal of Mathematical and Statistical Psychology,
and (again I may be wrong) I think it came out sometime in the 1970s.

> Is Cohen's Kappa applicable for binary responses ?

Well, it is. I have certainly used it for binary data, though the question
to ask here is whether anything better can be used. I think the Everitt
approach may be a slightly better attack here, or even griiting your teeth
and using some of the techniques developed in a paper by Landis and Koch.
(again, I'm in the throes of moving offices, and can't put my hand to the
exact references).


One comment I'd like to make is that what you've written seems to suggest that
you are prepared to believe (or not test), whether there are systematic
differences between the items rated in terms of the agreement the observers
display. Wouldn't it be at all useful for you to think of some way of
determining whether all the items rated could be reasonably assumed to be
homogeneous in the degrees of agreement noted? If you would like to do this,
I have a feeling that the Landis and Kock approach may be best, though I
suspect that you will have the VERY common problem here of too small a
sample size of observers. (This, in my experience often happens).

Finally, this may be a little naughty for me to make this comment, but
shouldn't these issues have been better sorted out more BEFORE the
observations were collected?

David Stretch

  +-----------------------+---------------------------------------------+
  |  Dr. David D Stretch  | I was gratified to be able to answer        |
  |  Dept. of Psychiatry  | promptly, and I did. I said I don't know.   |
  |  dds@leicester.ac.uk  |                                  Mark Twain |
  +-----------------------+---------------------------------------------+

The full reference for the Everitt paper is as follows:

Everitt, B.S. (1977). Some properties of statistics used for measuring
observer agreement in the recording of signs. British journal of
mathematical and statistical psychology, 30, 2, 227-233.
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU