X-Mailer: ELM [version 2.3 PL11]
Newsgroups: bit.listserv.stat-l
Date:         Tue, 15 Dec 1992 08:57:58 CST
From:         Mike Richman <richman@REEPICHEEP.GCN.UOKNOR.EDU>
Subject:      Re: Publishing Factor Analyses
              "A.C.Downing@newcastle.ac.uk" at Dec 15, 92 8:48 am

Quoting A.C.Downing@newcastle.ac.uk
>
> Steve Blinkhorn's posting on the subject of factor analysis, with his
> wonderful imaginary dialogue with an imaginary graduate student, raises at
> the end a subsidiary but still important furhter worry about factor analysis.
> "Then you publish the rotated  factor matrix" says inquisitor Steve
> (this after deciding no' of factors on the basis of variance accounted
> for in terms of standardized variables, and much else.)
>
> To be open to proper academic scrutiny, surely a paper in which a factor
> analysis is crucial needs to show the original correlation matrix, the
> unrotated initial factor matrix, the communalities and eigenvalues, the
> nature of the rotations, together with the final rotated matrix.  I cannot
> imagine a journal allowing such a spread, yet how else the readers get
> the information they need?  The necessity to be concise seems here to
> conflict with proper scientific disclosure.  Should now there be an agreed
> code of practice whereby fuller statistical details must be submitted with
> the paper and lodged in some kind of database such as list servers can
> now provide, so that they do not clutter up the journals but are vetted by
> the referees and are readily accessible to the readers?  It is rare
> indeed to see a factor analysis reported in a psychological journal in
> such a way that the reader can make any informed judgement about it at a
> technical level, or entertain alternative interpretations, etc.
>

Oh, I have seen this many times.  Given some of the large eigenanalyses
performed today (ie, I have a 30,000 by 1977 matrix to be decomposed on
a CRAY), and the nature of mapping loadings for visualization, this is
not always practical.  However, it does not excuse one from being held
accountable for their analysis and the methodological decisions.

> Of course this does not meet the more radical objections to factor
> analysis which have been voiced so eloquently, but isn't it a problem in
> its own right? - And one not confined to factor analysis.  I'm sure this
> suggestion cannot be novel, but isn't it time something of this sort became
> standard practice, so that a paper making claims based on statistical
> procedures producing voluminous output, could not be published unless such
> output were easily available for public scrutiny?
>

These same arguments have been voiced for over 20 years now with little
change.  Here is an excerpt from R.J. Rummel's (1970) textbook "Applied
Factor Analysis" (Chapter 23 "Publication") [SBN# 8101-0254-4]. That
chapter should be required reading for all of us who publish results
based upon common factor analysis, principal component analysis,
structural equations or, for that matter, any complicated statistical
technique (with suitable changes in the terminology and specific
methods).  Some of Rummel's book is dated, but I consider the list
below useful in evaluating my own work.  Note: this is highly abridged
and I've indicated a few personal comments in square brackets [].

"Publication is in effect a bid for nomination of the research results
for entry into the corpus of science.  This corpus contains laws and
substantive propositions in which science has confidence and which serve
as a foundation for continual scientific exploration along the periphery
of the unknown.  Propositions are only conditionally accepted into this
corpus, however, and may be ejected at any time that contractions are
found with other propositions in which science has more confidence.

Before research results can be nominated, they must be evaluated.
Evaluation is the weighing by the interested scientific community of the
adequacy of the data and the research design that links research
question to findings.  In order for this evaluation to be made, it is
crucial that sufficient information about the design be included...

If the design and findings are favorably evaluated, they stand in
nomination for membership in the corpus of science.  The crucial step
between initial evaluation of results and membership is replication...

Evaluation and replication are the most important factors governing a
report's contents and controlling its contribution to knowledge.  Other
considerations, however, should not be forgotten.  The heuristic value
of the design and findings are significant in domains that have not had
the benefit of much multivariate research or the particular design of
study....


A Research Report Checklist

With regard to the technical and substantive audiences, there are a
number of elements that should be included in every report.  This is the
information that is essential in order to evaluate, compare, and
replicate the study...

Model

 1. Indicate the research question...
 2. Give the factor model underlying the analysis...

Data

 3. Describe the data slice...
 4. Give the criteria for inclusion and exclusion of variables and cases...
 5. Describe the measurement characteristics of the data...
 6. Describe the data collection methods...
 7. Give your assessment of random and systematic error...
 8. Describe the amount of missing data and mode of estimation, if such
    estimates are used...

Preliminary Computation

 9. List the distributional transformations and criteria for their
    selection...
10. Give the matrix transformation [ie, covariance, correlation,...] and
    rationale for its choice...
11. Give the communality estimate, if one is made... they are important
    in themselves...

The Factor Computation

12. The factor technique should be indicated... the computing program
    employed might be indicated...
13. Give the number of factors criteria...
14. If rotation is done, it should be indicated.  If a rotated solution
    is published, indicate whether it is orthogonal or oblique...
15. If analytic rotation is done, the rotation technique must be
    indicated...
16. If other than simple structure is done, the rotating criteria should
    be indicated...

Factor Matrices

17. The unrotated and final solution should be given.  For the unrotated
    solution, the communalities and per cent of variance (or eigenvalues)
    should be indicated with the table...
18. Give the factor intercorrelation matrix for oblique factors.  If
    there are a small number of factors, these correlations may be included
    with the oblique factor tables.
19. The transformation matrix for the oblique factors should be included
    for studies with a large number of variables and many factors.  This
    enables others to adjust the published factors to new positions without
    the expense of starting the rotation anew. [Comment - might also be
    useful for orthogonal designs, but not necessary, given the original
    loadings, one can easity back transform the solution to reanalyze]

Factor Scores

20. The mode of computing factor scores should be indicated...
21. Give the multiple correlation coefficient of the regression factor
    score estimates of the original data.  This indicates the uniqueness of
    the factor scores for the data.
22. For some data...the cases will be of intrinsic interest.  Then the
    factor score matrix should be shown or be available...

Factor Comparison

23. The factor comparison technique used should be indicated.
24. A description of the two sets of findings in terms of cases and
    variable similarity is important for evaluating the comparison...

Distances

25. A distance matrix itself may be too large to include, but summary
    statistics or the mean distance and standard deviation for the matrix
    can be given... [Comment: there are better methods available since
    1970].
26. Whether distances are computed on weighted or unweighted factors and
    precisely what factors are used should be indicated..."

---
Mike Richman                        !CIMMS/The University of Oklahoma
richman@reepicheep.gcn.uoknor.edu   !100 East Boyd, Room 1110
405-325-3460 (office) 325-7614 (fax)!Norman, OK  73019-0628  USA
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU