CHI 2009 ~ Scientometric Analysis of the CHI Proceedings

April 7th, 2009 ~ Boston, MA, USA

Scientometric Analysis Of The CHI Proceedings
Christoph Bartneck, Jun Hu
Eindhoven University of Technology, Department of Industrial Design
Den Dolech 2, 5600MB Eindhoven, Netherlandsc.bartneck@tue.nl, j.hu@tue.nl
ABSTRACT

analysis of the CHI conference. We therefore use theconference’s proceedings as the basis for our analysis, since
it is the official record. What has really changed over thelast 25 years and who has contributed to the conference’s
success? How strong is the obvious US supremacy inreality? The CHI conference as it is today has not always
existed. One danger of not knowing the history of the CHI
conference is that what happens at the conference begins toappear as timeless and natural. History allows us to begin to
see the kinds of choices that have been made and theparticular biases and possibilities that resulted from thesechoices.
Two indicators might help to shed light on the distributionof papers across countries and organizations: the Pareto
Principle and Lotka’s law. The Pareto Principle, named
after the Italian sociologist, economist, and philosopher
Vilfredo Federico Damaso Pareto, is also known as the ruleof 80/20. It states that in many cases 80 percent of theeffects come from 20 percent of the causes. Garfieldsuggested that this principle also applies to citations in that
20 percent of articles may account for 80 percent of thecitations [5]. Does the Pareto Principle also apply to thecountries, organizations, and papers in the CHI
proceedings? Do 80 percent of the papers come from only
20 percent of the organizations?
Lotka [6] described a frequency distribution of scientificproductivity in which the number of authors making ncontributions is about 1 / na of those making one
contribution, where a nearly always equals two. The
general formula says: y = c / xn, where x is the number ofpublications, y the relative frequency of authors with x
publications, and n and c are constants depending on thespecific field (n is usually 2). Of course we cannot apply
Loka’s law to the authors in the CHI proceedings, becauseeven the most successful authors within the CHI conference
have only a handful of papers accepted. More interesting iswhether Lotka’s law applies to countries and organizations
involved in the CHI conference?
Besides these two indicators for the distribution of papers,
we are interested in the judgments made of the quality of
papers and organizations. During our own visits to theconference, we were sometimes surprised how certain
papers had made it into the conference and how certain
excellent papers received only little attention from theaudience.
In a multidisciplinary field such as this, assessing thequality of a manuscript for the CHI conference is difficult
since the different groups operate under different paradigms[7, 8]. The peer-review process tries to judge the quality
before the actual publication, but it is well known that the

The CHI conference has grown rapidly over the last 26
years. We present a quantitative analysis on the countries
and organizations that contribute to its success. Only 7.8
percent of the countries are responsible for 80 percent of thepapers in the CHI proceedings, and the USA is clearly thecountry with most papers. But the success of a country ororganization does not depend only on the number of
accepted papers, but also on their quality. We present aranking of countries and organizations based on the
h-index, an indicator that tries to balance the quantity andquality of scientific output based on a bibliometric analysis.
The bibliometric analysis also allowed us to demonstrate
the difficulty of judging quality. The papers acknowledgedby the best paper award committee were not cited more
often than a random sample of papers from the same years.
The merit of the award is therefore unclear, and it might beworthwhile to allow the visitor to the conference to vote forthe best paper.
Author Keywords

quality, bibliometrics, chi, history, h-index, g-index
ACM Classification Keywords

K2. History Of: Theory.
INTRODUCTION

The SIGCHI conference on human factors in computing
systems (CHI) is one of the most important conferences in
the area of human–computer interaction (HCI). It wasstarted in 1982, although a preliminary conference took
place a year earlier, and at the 2008 conference in Florence
two invited sessions looked back at the history of theconference. John Karat and Manfred Tscheligi hosted thesession “From Usability To User Experience: What Has
Really Changed In Practice In The Last 25 Years”, andBonnie John chaired the “Celebrating The Psychology Of
Human-Computer Interaction” session. Two previous
studies investigated the changes in evaluation methods usedin studies that are published in the CHI conference [1, 2]
and two studies tried to cluster CHI papers into categories[3, 4]. These studies inspired us to go beyond an anecdotal
account of the history of CHI, and present a quantitative
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee.
CHI 2009, April 4–9, 2009, Boston, Massachusetts, USA.
Copyright 2009 ACM 978-1-60558-246-7/09/04...$5.00.

699

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings
process is not without problems [9-12]. In a controversial
study, Peters and Ceci [13] altered published articles to
disguise their origin, and resubmitted them to the journals
that had originally published the manuscripts. Most alteredpapers were not recognized and were rejected on supposed"scientific grounds." The original article is only eight pages
long, but it evoked a discussion in the open peercommentary of 59 pages. The discussion illustrates that thepeer-review system is of high relevance to the researchcommunity and that it is a heated topic.
Several attempts have been made to evaluate the quality of
the review process, but one conceptual problem remains.
For a complete analysis it is necessary to also consider therejected papers. This information is usually not available
from the ACM, and hence we are not able to compare thesuccess of the accepted papers with the success of therejected papers.
However, the best paper award allows us to analyze
nominees, winners, and non-winners. It is not known
whether the best paper award committee is following thestrict blind peer review process, but our guess is that this isunlikely. Still, they make an honest effort to select“excellent” papers and they probably take the results of thereview process into account. Since 2004, the nominees andaward winners are indicated in the conference brochure, andhence we focus on the years 2004-2008.
The judgments made of the quality of papers does not stopwith their publication. Increasingly, bibliometric indicators,
such as the impact factor, are being used to judge thequality of researchers, journals, and organizations [14, 15].
Thomson Reuters, on the basis of the groundbreaking workof Eugene Garfield and his company, Institute for Scientific
Information (ISI), publishes the annual Science Citation
Index (SCI), as well as the Social Sciences Citation Index(SSCI) and the Arts and Humanities Citation Index (AHCI).
They list the impact factor of leading scientific journals,
based on a bibliometric analysis. The impact factor brings a
useful measure of objectivity into the evaluation process at
relatively low financial cost [16]. These indexes have beenused to judge the quality of the journals, but the databasehas also been used for other bibliometric analyses.
The performance of researchers and their organizations in
the United Kingdom will be evaluated officially in the 2008
Research Assessment Exercise (REA) with the help of
bibliometric indicators. ACM also started in 2008 to list
citation counts for the articles in its digital library. But
bibliometric analysis is also facing criticism. The interested
reader may consult MacRoberts & MacRoberts [17], whoreviewed common objections to bibliometric analysis, andgrouped them into: (in)formal influences are not cited,
citations are biased, self citations, different types ofcitations, variation in citation rate with type of publication,nationality, time period, size and type of specialty, andtechnical limitations. An example of a citation bias is that
not every citation is made to positively acknowledge
previous work. Lawrence [18] pointed out that bibliometricindicators are now so widely adopted that they force

April 7th, 2009 ~ Boston, MA, USA

scientists to change their primary goal from making
discoveries to publishing as many papers as possible in highimpact factor journals. As a result the utility, quality andobjectivity of articles has deteriorated.
The debate between supporters and critics of bibliometricanalysis continues. Meanwhile, improvements to themethodology of bibliometrics are being developed. It can beconcluded that, despite certain conceptual andmethodological problems, bibliometric analysis is a usefultool. The representative body for the executive heads of UK
universities (UUK) considers bibliometric indicators “…
probably the most useful of a number of variables that
could feasibly be used to create a metric of some aspect of
research performance” [19].
A fundamental question of all bibliometric analysis is onwhat data the analysis is based. Eugene Garfield
spearheaded the analysis with his Institute for Scientific
Information (ISI), and remained unchallenged for many
years. In 1997, Katz and Hicks proclaimed the arrival of
Desktop Bibliometrics [20]. Technological advances, partly
based on ISI’s Web of Science (WOS), would move
bibliometric studies “from the realm of the privileged few
with access to mainframe and minicomputers to the desktopof researchers equipped with personal computers”. The
development of Elsevier’s Scopus and Google Scholar (GS)surpassed their predication. Moreover, GS provides freeaccess to bibliometric data that may lead to moretransparency in tenure reviews, funding, and other science
policy issues, as it allows citation counts, and analyses
based thereon, to be performed and duplicated by anyone[21]. Today, everybody can use software, such as “Publish
or Perish”, to perform a simple bibliometric analysis withinminutes on the basis of data attained from GS.
However, GS has also received criticism [22-24]. In
response to Jacso’s studies, Harzing and Wal tried toreproduce most of the GS failures that Jacso had reported,
but were not able to do so reliably [25]. We may conclude
that some of the issues raised by Jacso may have been valid
at that time, but that GS continues to improve its service,
and many practical problems have been overcome by now.
There are also other reasons that make GS the preferreddata source for a bibliometric analysis of the CHI
proceedings. GS far surpasses WOS and Scopus in its
coverage of conference proceedings as well as ofinternational, non-English language journals [26].
Conference proceedings are an essential publication
channel for HCI research, and hence GS provides thenecessary coverage for a bibliometric analysis of the CHI
proceedings. To illustrate the power of GS, we performed asmall test. We searched for the relatively unknown
publications of “C Bartneck”, since he has a unique name.
WOS found 7 publications, Scopus 23 and GS 67. The GS
results are closest to the accurate value of 81. A similar
search for “DA Norman” resulted in 32 publications listed
in WOS, 41 in Scopus, and 658 in GS. Clearly, GS offers
the coverage we require to analyze the HCI community.

700

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings

April 7th, 2009 ~ Boston, MA, USA

when we sort them by the number of accepted papers, theh-index or the g-index?
In summary, besides presenting a historical perspective andrankings, we focus on a bibliometric analysis. The latter
offers a method, although not undisputed, for evaluating thequality of papers and organizations. In particular we areinterested in which countries and organizations are mostsuccessful in the CHI conference and what indicators might
be most useful for the ranking. We are also interested
whether the best paper award truly acknowledges
excellence.

An even stronger argument for GS is that neither WOS nor
Scopus fully index the CHI proceedings at present. Scopus
only includes the CHI proceedings from 1992 onwards, andfocuses on the main proceedings. The extended abstracts
are not included for all years. Thomson Reuter reacted to its
lack of coverage of conference proceedings by releasing its
ISI Proceedings service, but this indexes only the years
2002 and 2005 of the CHI proceedings.
However, a bibliometric analysis does allow us toreconsider the judgment of the supremacy of countries andorganizations. Clearly, not only the sheer number of
accepted papers is important, but also their quality,
expressed through their citation count. Jorge E. Hirsch
proposed the h-index to characterize the scientific output ofa researcher or organization [27]. It balances the quantityand quality of papers. The calculation of the h-index canbest be described by a concrete example. A researcher has
an h-index of eight if he/she has at least eight publications
that each has been cited at least eight times. The h-index
thereby takes the quantity and quality of papers into
account at the same time.
There is a catch in calculating the h-index for countries andorganizations. The authors of a certain paper can belong todifferent organizations and countries. Let us assume thatcountry x has 0.5 credits for papers that have exactly 1
citation, 1.33 credits for papers that have exactly 2 citations,and 2 credits for papers that have exactly 3 citations. We
then calculate
s1 =  2 + 1.33 + 0.5  = 3,
s2 = 1.33 + 0.55  = 1
and s3 = 0.55 = 0.
It follows that s1 > s2 > s3. Country x has at least one paperthat has at least one citation ((s1=3)>1). However, it does
not have at least two papers that have at least two citations((s2=1)<2). The h-index for country x is therefore 1. We
used the same principle to calculate the h-index fororganizations.
The popularity of the h-index may stem from its simplicity
and its robustness against large numbers of barely citedarticles. However, in the same way that it is robust against
the long tail of non-cited papers, it is also immune to theeffects of highly cited papers. To use the above example, itdoes not matter anymore if the eight papers that are above
the threshold of eight are being cited 8, 80, 800, or even8000 times. The g-index maintains the h-index robustness
against infrequently cited papers, while taking into accounthighly cited papers [28]. Egghe defines the g-index as the(unique) largest number such that the top g articles received(together) at least g2 citations. Let us use the example ofcountry x again. Country x has at least one paper that has atleast 12 citations ((s1=3)>1). However, it does not have atleast two papers that have at least 22 citations ((s2=1)<4).
The g-index for country x is therefore 1.
We are interested in comparing these two indices with thebare quantitative indicator of the number of accepted
papers. Does the ranking order for organizations change

DATA PROCESSING

The ACM made the complete meta-data of the CHI
proceedings available to us in an XML format. A firstreview of the XML data revealed that the structures ofvarious XML files were inconsistent. We used the Map
Force software to import the different XML files into one
consistent relational database. Notice that there was no CHI
conference in the year 1984, and hence no data is available
for that year.
Authors

A closer inspection of the data revealed that theidentification numbers of authors were inconsistent. The
same author could appear several times with different
identification numbers. Moreover, the authors themselves
used inconsistent names. William Buxton, for example, isalso listed as “W. Buxton” or “Bill Buxton”. To make thesituation even more confusing, he also worked for different
organizations. We therefore used the FuzzyDupes software
to group the aliases of authors together and to give themtruly unique identification numbers. However, this process
can never be perfect. Persons with the same name anddifferent affiliations could indeed be two different authors.
But the same author might have moved from one
organization to the other, or be working for bothorganizations at the same time. It is also possible that two
authors with exactly the same name work for the sameorganization. This problem might be difficult to overcome,and the market leader for bibliometric analyses, Thomson
Reuter (Web Of Science, WOS), recently introduced the“Researcher ID” to help in identifying unique authors. The
ACM also introduced the “Author Profile Pages” with thevery same intention. Both systems use normalization
algorithms to identify unique authors, but both systems also
rely on the contributions of the researchers themselves. The
problem remains difficult, and we therefore focus our
statistics on the organization and country level, and less on
the individual author.
Affiliations

We discovered that the information about the affiliations of
the authors was messy. 705 records of the 17,610 (4%) did
not include any affiliation information. A closer analysis of
the missing affiliations revealed that almost all affiliations
for the main proceedings in the years 1981, 1982, and 1992
were missing. It is unclear why this information is notavailable in the XML files provided by the ACM or in the701

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings
actual ACM Digital Library. The PDF themselves include
this information.
We also discovered that all the information about theauthors’ affiliations has been stored in one field. This
contains the name and the address of the organization. A
direct analysis was therefore not possible. Moreover,
misspelled names were common, even from authors
associated to “Standford”. The names of the affiliations
could also occur in different forms, in particular for nonEnglish organizations. For example, the Eindhoven
University of Technology appears in at least six versions:
Eindhoven University of Technology, Technical Universityof Eindhoven, Technische Universiteit Eindhoven, TU
Eindhoven, TUE, and TU/e. Sometimes authors even
included more than one affiliation. In that case, we used thefirst affiliation mentioned.
FuzzyDupes was not able to group the affiliations by itself,
since the affiliation field was far too inconsistent. We
therefore resorted to a semi-automatic procedure. We firstcompiled a list of 3000 acknowledged universities andinstitutes. We then performed a precise match between the
known names of organizations and the affiliation field.
Approximately half of the records could be matched in thisway.
We then increased the number of matches by including
variations of the known names, and by adding 34
universities, 36 institutes, and 111 companies to our list of
known organizations. We paid extra attention to records
that included a university affiliation, and were able to match
all of them in this way. The remaining 1247 records
contained affiliations to minor companies and institutes. We
manually classified 84 records as institutes, based on thefact that their affiliation contained the word “institute”. We
did not include these institute names in the list of known
organizations. The remaining 1163 records were classified
as companies. We would like to point out that the list of
known organizations distinguishes between sub sections oforganizations. For example, the two research IBM researchlaboratories, Almaden and Watson, have been included
separately. The quality of sub sections might differ
considerably and we respected the view of the authors. In
most cases, the members of the sub sections deliberately
chose to mention their sub section.
Lastly, we manually matched the 1247 unidentified records
to the known list of countries. Overall, we noticed that thevast majority of authors from the USA did not include theircountry in the affiliation. At best, they included their state.
Authors from others countries usually included theircountry. We have to point out that the country of theorganization does not necessarily match the nationality of
the author. For practical reasons it was impossible for use todetermine the nationality of each CHI author and webelieve that statistics on the organizational issue are ofhigher value than on the individual level. In summary, we
have been able to match 93 percent of the records to a list of
known organizations, and the remaining unmatched records
are mainly from minor companies and institutes that occur

April 7th, 2009 ~ Boston, MA, USA

only occasionally. All records with affiliation information
were matched to a country.
Before continuing, we performed a quality check of thematching procedure. We randomly selected 800 of the15600 records (5.1 percent) and manually checked whether
the affiliation was matched correctly. We found 2
mismatches (0.25 percent), which gives us sufficientconfidence in the quality of the matching to continue ouranalysis.
Citations

The bibliometric analysis is based on citations gatheredfrom Google Scholar (GS). We developed software that
automatically queried the website and stored the data backinto our database. In the search query we included the titleof the paper, its authors, and the year of publication. The
process was conducted on August 22-23, 2008. Besides thecitation count for every paper, we also collected the title ofthe publication from Google’s result page. We compared
the title and the authors returned by GS with the original
data in our database, to assess whether Google had foundthe correct paper. GS was not able to find 71 papers. Thesepapers were excluded from the further analysis.
One of widely known measure for the similarity of two text
strings is the Levenshtein distance (LD). The LD is definedas the number of operations necessary to transform one text
string into another, where operations include insertion,
deletion, or substitution of a single character. The average
LD between all the titles of papers in our database and theresulting titles from GS was 2.79 (median=2). This number
is still compromised by several factors. For one thing, theLD considers special characters such as spaces andpunctuation marks, which can cause a considerable increase
without being a meaningful difference. Moreover, GS does
not return titles longer than about 100 characters. When welimit the calculation of LD to titles with no more than 100
characters, we receive a distance of 1.98. A mean of 1.98
can be considered an extremely good match between thetitle stored in the ACM database and GS because the LD issensitive to the slightest difference in spaces, spelling, andpunctuation marks.
Next, we calculated the relativity of the paper title incomparison to the returned title from GS. Given two text
strings, the relativity normalizes the Levenshtein distancewith respect to the lengths of the strings [29]. This
calculation provides better results than the Levenshtein
distance alone. The mean relativity for papers with title
names shorter than 100 characters was 0.93. This result
suggests sufficient quality in our queries to GS.
In addition, we randomly selected 100 papers, and checked
the correctness of the GS result manually. From these 100
papers, GS recognized only three papers incorrectly. All
three mismatches were due to the fact that authors had
published multiple articles with exactly the same title in one
year. This problem points towards a more general issue.
Authors sometimes publish papers with exactly the same
title at different venues, even within the CHI conference.
Papers can appear as a full paper, but also as a late-breaking

702

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings
result. In the early days of the CHI conference, papers, inparticular the results of panel discussions, were also
republished as SIGCHI Bulletins. GS is not yet able todistinguish between these papers, and by default presents
the paper with the highest citation count first.
We conclude that, although the citation counts from GSmight not be perfect, there is no practical alternative and webelieve that the quality of the citation counts is goodenough to allow for valuable insights into the CHI
proceedings. Before moving on to the results, we need tomake a few definitions:Credit: One paper equals one credit. Each author receives
an equal share of the credit. For example, for a paper
written by four authors, each author receives 0.25 credits.
Continents: The definition of continents is disputable, and acomplete list of our mapping of countries to continents isavailable in the appendix.
English speaking country: developed countries in which
English is a traditional language, such as USA, UK, Ireland,
Canada, Australia, New Zealand.

April 7th, 2009 ~ Boston, MA, USA

increased number of authors. In such a case, the overall
productivity would have remained the same.
600

extended abstracts
500

main proceedings
number of papers

400

300

200

100

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

1989

1988

1987

1986

1985

1983

1982

1981

0

Figure 1: papers per year in the main proceedings and in theextended abstracts (accumulated)

When we look at the ratio of articles per unique author andat the ratio of pages per unique author, we see that the there
is no difference in the ratio of articles per author between
the main proceedings and the extended abstracts, and that
this ratio has remained fairly stable (see Figure 3).
However, the ratio of pages per author for the extended
abstracts is approaching the level of the main proceedings.
This means that by now the unique authors in the extended
abstracts write almost the same number of pages as uniqueauthors in the main proceedings.

RESULTS

The page limitation does not allow us to present thecomplete results. For example, we simply cannot show thecomplete list of the more than 700 organizations that havecontributed to the CHI proceedings. The interested reader
may consult the appendix to see the complete lists andrankings. We treated incomplete data, such as the 705
absent affiliations, as missing data in our analyses. We shallnow present the results of our study by moving from thegeneral proceedings, down to the countries, organizations,and papers.

4000

extended abstracts
3500

main proceedings

number of pages

3000

Proceedings in general

The CHI conference has experienced strong growth during
its 28 years of existence. The number of papers in the mainproceedings has roughly doubled (see Figure 1), and theextended abstracts have outgrown the main proceedings interms of numbers of authors, and also in terms of pages,
since they were introduced in 1992 (see Figure 2). On
average, for every paper in the main proceedings, we find2.63 papers in the extended abstracts. For every page in themain proceedings we have 0.85 pages in the extended
abstracts. However, if we focus on the last five years, wehave a ratio of 1:1.2 pages. The number of authors in themain proceedings is on average half of the number ofauthors in the extended abstracts.
The length of the papers has also increased. For the period
until 1993, a paper in the main proceedings had around five
pages, which increased to around eight. The papers in theextended abstracts grew from about two pages in 2003 to an
average of six pages in 2008. The papers in the extended
abstracts today have reached the same length as the papers
in the main proceedings before 1993. This effect can largely
be explained by the page limitations given by theconference. Most authors write as many pages as allowed.
However, one may speculate on whether this increase inlength of the papers might be counterbalanced by an

2500

2000

1500

1000

500

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

1989

1988

1987

1986

1985

1983

1982

1981

0

Figure 2: pages per year in the main proceedings and in theextended abstracts (accumulated)

The bibliometric analysis shows that the papers in the mainproceedings (mean = 40.167) are significantly(F(1,6295)=308.253,p<0.001) more cited than the papers in
the extended abstracts (mean = 11.917).
Geography
Table 1 shows the sum of credits per continent and country.
To provide a clear overview, we grouped the countries withfewer than five credits into the category “others”. Morethan 62% of the credits go to the USA, and the top four
countries (USA, UK, Canada, and Japan) alone accumulate
82 percent of all credits. This means that the distribution of
credits across countries goes even beyond Pareto’s 20/80
principle. 7.8 percent of the countries are alreadyresponsible for more than 82 percent of the credits.

703

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings
Moreover, 50 percent of the countries have fewer than 10
credits. It is also worthwhile mentioning that nearly 80
percent of all credits go to traditionally English speaking
countries (USA, UK, Canada, Ireland, Australia, New
Zealand).

Organizations

The majority of credits for the CHI proceedings go touniversities (see Table 2). They received more than twice as
many credits as companies, which receive more than twice
as many credits as institutes. However, the proportion of
credits in the main proceedings is similar to that in theextended abstracts. Universities, companies, and institutes
receive approximately double the number of credits for theextended abstracts as for the main proceedings.

4.00

3.50

country

2.50

2.00

mean articles per unique author (main)mean pages per unique author (main)
1.50

n. america

number of articles and pages

3.00

mean articles per unique author (extended)mean pages per unique author (extended)

1.00

0.50

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

1989

1988

1987

1986

1985

1983

1982

1981

0.00

Figure 3: mean articles per unique author and mean pages per
unique author across the main proceedings and the extended
abstracts.

300

europeCanada shows a remarkable performance. It has roughly 20
percent fewer credits than the UK, in particular in theextended abstracts, but receives the same h/g-index as the
UK. When we relate the total number of credits to the
populations of the countries [taken from: 30], Canada evenexceeds the USA and is only lower than Sweden and
Finland.
usa
usa (main)usa (extended)250

united kingdom
canadajapan

200

germany
swedencredits

April 7th, 2009 ~ Boston, MA, USA

netherlands

150

francefinlanddenmark

100

50

asia

*2008

2007

2006

2005

*2004

2003

2002

2001

*2000

1999

1998

1997

1996

1995

1994

*1993

1992

1991

1990

1989

1988

1987

1986

1985

1983

1982

1981

0

Figure 4: credits per country over time. An * marks the years
in which the conference took place in Europe.

africas.america

australia

When we look at the geographic development of the CHI
proceedings over time, we notice a sharp increase in thecontributions from the USA for the extended abstracts inthe year 1993-1994 (see Figure 4). This may be explainedby the introduction of the extended abstracts in the year
1992. While the USA contributions remained relatively
stable over the years, the extended abstracts contributions of
the USA dropped dramatically in the year 2000, which
could be connected to the Internet Bubble crisis and the factthat the conference took place in the Netherlands. Many US
companies usually publish in the extended abstracts (seeTable 3), and we speculate that US companies might nothave had the resources to send their employees overseas inthat year.

main
credits
1576.781387.03extendedcredits
2589.26
2358.19

Cred./mill. pop.
12.25

h

g

total
usa

sum
credits
4166.043745.22

137

27

canadamexicoother

406.4812.33
2.00

188.741.00
0.00

217.74
11.33
2.00

12.36
0.12
-

52
2
-

12
2
-

total
uk

1392.95
508.95386.95182.461006.00
326.48

8.38

52

12

germany
swedennetherlands

177.18142.00118.56

39.74
28.39
17.17

137.44113.61101.40

2.15
15.57
7.22

19
22
16

7
7
6

francefinlanddenmark

82.66
79.55
60.09

27.03
30.59
25.74

55.63
48.96
34.35

1.34
15.07
11.04

19
16
18

6
5
5

austria
italy

46.81
33.51

6.42
3.03

40.39
30.48

5.60
0.57

8
6

4
3

switzerland
irelandbelgium

25.84
19.85
18.57

6.60
7.85
1.00

19.24
12.00
17.57

3.45
4.62
1.78

8
6
6

3
2
3

israelportugalrussia

14.09
10.67
10.00

3.00
0.00
0.00

11.09
10.67
10.00

2.03
1.00
0.07

5
3
4

2
2
2

spain
norway

9.34
9.08

0.33
2.83

9.00
6.25

0.21
1.93

3
3

1
2

greecepoland
other

7.00
6.33
12.87

3.00
0.00
1.75

4.00
6.33
11.12

0.63
0.17
-

4
1
-

2
1
-

total
japan
south korea

335.95260.48
33.03

74.42
57.66
4.42

261.53202.82
28.62

2.04
0.68

35
6

9
3

india
singapore

12.48
12.02

0.85
2.17

11.63
9.86

0.01
2.71

2
4

1
2

china
other
total

7.08
10.87
67.85

5.33
4.00
19.66

1.75
6.87
48.19

0.01
-

3
-

2
-

australia
new zealand
total

49.38
18.48
18.12

11.83
7.83
1.25

37.55
10.64
16.87

2.38
4.42
-

9
9
-

4
3
-

brazil
other

10.78
7.33

0.00
1.25

10.78
6.08

0.06
-

3
-

1
-

total
south africa
Other

13.61
12.75
0.86

0.73
0.00
0.73

12.88
12.75
0.13

0.26
-

5
-

2
-

Table 1: credits, h-index and g-index per continent and country(sorted by continent and h-index)

When we split up the organization types across the top
twenty countries, we observe that on average 21 percent of704

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings
the credits go to companies, 16 percent to institutes and 62
percent to universities (see Table 3). Notable deviations arevisible for Canada, Denmark, and the UK. All of them havea stronger emphasis on Universities. 48 percent of Japan’s
credits come from companies, which is the highest value of
all countries.

450.00350.00extended2449.421005.46461.43300.00

Table 2: credits across types of organizationcompany
creditpercent
1120.2
0.30
62.5
0.13
30.0
0.07
123.9
0.48
29.3
0.17
12.3
0.09
15.3
0.13
16.9
0.20
24.9
0.31
10.2
0.17
10.0
0.20
5.4
0.11
6.9
0.21
6.7
0.20
2.2
0.08
3.5
0.18
5.7
0.30
0.5
0.03
1.5
0.11
10.3
0.82
3.8
0.31

institutecreditpercent
486.9
0.13
0.3
0.00
8.4
0.02
19.5
0.08
42.7
0.24
64.1
0.45
25.1
0.21
51.5
0.62
2.0
0.03
0.8
0.01
8.1
0.16
5.6
0.12
10.9
0.33
6.7
0.20
1.5
0.06
8.0
0.40
0.0
0.00
0.0
0.00
0.0
0.00
2.2
0.18
0.0
0.00

250.00200.00150.00The historic view provides a more detailed insight. Whilethe contributions of the companies and institutes haveremained fairly stable for the last 15 years, we see that thecontributions from universities have more than quadrupledin the same period (see Figure 5). Notice that the data forthis graph is slightly different from that for Figure 4. As
mentioned in the data processing section above, we wereable to match all authors that mentioned an affiliation witha country, but we were not able to match them all to anorganization type. In particular the non-university and noninstitute affiliations remain less well-matched.
countryusa
uk
canadajapan
germany
swedennetherlands
francefinlanddenmark
australia
austriaitaly
s. koreaswitzerland
irelandbelgium
n. zealandisraelindia
Mexicouniversity
university main
company
company main
instituteinstitute main

400.00100.00
50.00

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

1996

1995

1994

1993

1992

1991

1990

1989

1988

1987

1986

1983

0.00

1985

main
1232.14513.51
297.60proceedings are often also the organizations that are highly
cited. The appendix includes a similar table that is limited
to the last ten years, which represents current trends better.

credits

university
company
institute

total
3681.57
1518.97759.03

April 7th, 2009 ~ Boston, MA, USA

Figure 5: credits over time by organization typeA scatter plot showed a very steep decline in organizations
with increasing credits. 83 percent of the organizations havefewer than 10 credits. A non-linear curve estimation
analysis for Lotka’s Law revealed that the field-specificconstant c should be 1612, resulting in the formula
y=1612/x2 (R2=0.991). We also observe that mostorganizations in the top twenty are from the USA.

university
creditpercent
2132.6
0.57
420.6
0.87
368.1
0.91
115.4
0.45
105.1
0.59
65.6
0.46
78.2
0.66
14.2
0.17
52.7
0.66
49.1
0.82
31.3
0.63
35.9
0.77
15.7
0.47
19.6
0.59
22.1
0.86
8.4
0.42
12.9
0.70
18.0
0.97
12.6
0.89
0.0
0.00
8.5
0.69

organization
palo alto res. ctr.
mitcarnegie mellon uni.
uni. toronto
georgia techmicrosoft
uni. calif. berkeley
uni. michigan
stanford uni.
uni. colorado boul.
ibm res. watson
uni. maryland
uni. calgary
xerox
uni. washington
ibm res. almaden
hewlett packard labs
virginia techapple computer
bellcoresum
130.88262.95298.51118.04156.97
153.7597.17
120.33110.02
92.12
154.25
88.98
34.47
23.64
82.00
36.31
50.07
47.83
42.06
36.63

main
76.56
73.78
126.09
73.55
48.76
86.91
33.34
43.04
37.73
46.93
68.04
30.71
13.88
18.28
31.94
16.92
24.07
17.93
18.92
13.41

ext.
54.32
189.16172.4244.48
108.21
66.83
63.83
77.29
72.28
45.18
86.21
58.27
20.58
5.37
50.05
19.39
26.00
29.89
23.14
23.22

h
57
43
41
35
34
34
28
27
27
27
26
25
19
18
17
16
15
15
15
15

g
9
11
11
8
8
8
7
7
7
7
8
6
5
4
5
4
5
5
5
4

nom.
1.90
1.25
8.53
3.67
3.03
6.33
1.50
0.48
2.25
1.00
3.92
1.70
2.67
0
2.10
3.92
0
1.25
0.25
0

awar.
0
0
2.00
1.83
0
1.42
0
0
0
0
0
0
0
0
1.83
0
1.00
0
0
0

Table 3: credits across countries and organization types

Table 4: top twenty organizations in h-index order

When we focus our attention on the individualorganizations, we observe that 22 percent of theorganizations are responsible for 80 percent of the credits.
This proportion is very close to the Pareto Principle. Table
4 shows the top twenty organizations sorted by theirh-index. The Spearman rank correlation between the total
numberofcredits,
h-index,andg-indexis=0.986;n=758;p<0.001.
The order of the organizations does not change significantly
if they are sorted by the credits, the h-index or the g-index.
We speculate that the fine differences between the h-index
and the g-index might not matter at the level oforganizations that have many publications. The
organizations that publish many articles in the CHI

We shall now focus on the level of individual papers. We
observe that the top 18 percent of papers are responsible for
80 percent of the citations, which is close to the Pareto
Principle and Garfield’s observation [5]. A scatter plot
revealed a steep decline in the frequency of papers withincreasing citations. A non-linear curve estimation analysis
for Lotka’s Law revealed that the field specific constant cshould be 49500, resulting in the formula y=49500/x2(R2=0.946).
Best Paper Award

Table 4 also includes columns that list the best paper
awards and nominations. We notice that 62 percent of thenominations were given to the USA, followed by 12 percent
to Canada and 9 percent to the UK. The papers published in

705

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings

between the percentage of nominations and the percentage
of awards (2=24.626;df=15;p=0.055).

2008 are still too recent to have already received citations.
We therefore compared the number of citations that thenominees (n=64), and award winners (n=12) from the years
2004-2007 have acquired against a random sample that was
not nominated (normal) from the same years (n=76). An
Analysis of Variance (ANOVA) was performed in which
the status of the paper was the independent factor (normal,
nominee, winner) and the number of citations was thedependent variable. There was no significant difference(F(1,149)=1.103, p=0.335) between the three groups ofpapers. In other words, the number of citations received by
the random sample of not nominated papers did not differ
significantly from that received by the nominated or award
winning papers. Figure 6 shows that the high variance andthe large number of outliers may be responsible for the nonsignificant results. There is a great deal of debate in theliterature as to the definition of outliers and as to what to dowith them. A thorough review of the various arguments is
not possible here. Figure 6 shows that there are not only one
or two outliers, but a considerable number of them. We
argue that the exceptionally highly cited papers are an
important part of the CHI conference and should not be
removed. The data are more likely to be representative of
the population as a whole if the outliers are included [31].

rank
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

winnercountry
nomin. awards
total % aw. & nom.
% cred.
USA
52.91
13.58
66.49
0.604
0.625
Canada
12.53
3.08
15.62
0.142
0.068
UK
7.98
2.00
9.98
0.091
0.085
Germany
3.58
0.00
3.58
0.033
0.030
Sweden
4.17
0.00
4.17
0.038
0.024
Finland
1.00
0.00
1.00
0.009
0.013
Japan
1.00
0.00
1.00
0.009
0.043
NewZealand
1.83
0.00
1.83
0.017
0.003
Israel
0.50
0.00
0.50
0.005
0.002
Italy
0.50
0.00
0.50
0.005
0.006
Denmark
1.50
0.00
1.50
0.014
0.010
France
1.00
0.00
1.00
0.009
0.014
Ireland
1.00
0.00
1.00
0.009
0.003
Angola
0.00
0.33
0.33
0.003
0.000
Austria
1.00
0.00
1.00
0.009
0.008
Switzerland
0.50
0.00
0.50
0.005
0.004
Table 5: Total number and percentage of nominations
and credits.

DISCUSSIONS AND CONCLUSIONSThe CHI conference is one of the most important
conferences for the HCI community. The results of thisstudy show that the conference has grown rapidly in the last
five years. The number of papers and authors has roughly
doubled, while the number of pages has tripled. The
increase in papers can be attributed, in particular, touniversities. Their contributions have quadrupled in the last
15 years. When we focus on the organizations, we observethat the order in the ranking of the organizations does notchange significantly if they are sorted by number of credits,
h-index, or g-index. We speculate that the productivity ofan organization is related to the quality of its papers.
Organizations that publish many papers at the CHI
conference also receive many citations. The raw sum of
accepted papers is a blunt, but still effective, method of
judging the success of researchers and their organizations[32].
Two other factors might have contributed to the rapid
growth of the conference. Firstly, the introduction of theextended abstracts in 1992 had a massive effect on theproceedings, and by now their size in terms of papers andpages exceeds that of the main proceedings. The papers in
the extended abstracts receive only about one quarter of thecitations. Still, the extended abstracts have become an
indispensable part of the conference, and we would like toencourage ACM and the conference organizers to include
the extended abstracts in the calculation of the conference’s
acceptance rate.
Secondly, the abandoning of the printed proceedings infavor of electronic proceedings on CD-ROM and in the
ACM digital library has practically eliminated constraints
on the number of pages. It does not make any significant
financial difference anymore whether the PDF files are fouror eight pages long. Instead, the limiting factor is theavailable time of the reviewers who have to read all thosepages. Their available time will naturally limit the size ofcitationsnominees

April 7th, 2009 ~ Boston, MA, USA

normalFigure 6: Mean number of citations of the nominated,awarded, and normal papers (2004-2007).

One may wonder if the dominance of the USA in the CHI
conferences may also have an impact on the nominations
for the best paper award. When we compare the percentage
of nominations to the percentage of credits, we notice that
the USA is approximately evenly represented in thenominations and the credits (see Table 5). Canada,
however, has only six percent of the credits, but achieves 12
percent of the nominations.
A Chi-Square test among the 16 countries that received
either a nomination or an award revealed that there was no
significant correlation between the percentage of credits andthe percentage of combined awards and nominations(2=13.220;df=15;p=0.585). A second Chi-Square test
revealed that there was also no significant correlation

706

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings
papers, both for the extended abstracts and the mainproceedings.
The main contributors to the conference are a relatively
small group of organizations from the USA, Canada, andthe UK. They have the largest number of credits andcitations. Already 7.8 percent of the countries account for
80 percent of the credits. The USA, with 62 percent of thepapers, is the largest contributor to the CHI proceedings.
Given that the conference is organized by an Americanorganization and that it had been a dedicated US conference
since 1992, this might not be too surprising. Canada takes a
special position, since it focuses on publishing highly cited
papers in the main proceedings. It exceeds the much more
populous USA with this strategy.
Canada has also received an over-proportionate number of
nominations for the best paper award. We notice that nopaper from the extended abstracts has received this award,
and that the relationship between the number of credits,nominations, and awards is weak. Some countries receive
more nominations than the distribution of credits would
suggest, and other countries receive fewer. Authors fromsixteen different countries have been nominated for the best
paper award, but only authors from the USA, Canada, andthe UK have received the actual award, with the exceptionof one unusual co-authorship from Angola.
Judging quality remains a difficult task for the initial
reviewers, but also for the best paper award committee.
Despite its honest efforts, the best paper award committee
has not selected papers that are cited more often than otherpapers. In other words, the best paper award committee did
not perform better than random chance. From thisviewpoint, Desney Tan’s claim that “The Best of CHI
awards represent the top one percent of researchsubmissions to CHI” appears too optimistic [33].
Program committee members nominate papers for the best
paper award and based on these nominations, the program
chairs pick the winners. The program chairs are often
acknowledged experts with senior status. What remains
difficult for them might be even more difficult for the morejunior members of the program committees. We may
speculate whether the task of selecting papers that will behighly cited in the future is simply too difficult, if not
impossible. In any case, we do have to ask ourselves what
the purpose of the award is if it does not correlate with theviews of the HCI community. We speculate if it might beworthwhile considering whether the conference attendees
should be allowed to vote for the best paper.
The statistics presented in this study are certainly
informative. We hope that the ranking of countries andorganizations will help students and employees to makeinformed decisions when seeking education or employment
within the CHI community. It may also help organizations
to find effective partners in their specific region. But we do
not want to conclude this paper without giving some
practical recommendations to authors. We encountered
several problems while systematically searching for articles
in GS, and we would like to share our insights in order toApril 7th, 2009 ~ Boston, MA, USA

guide authors towards higher bibliometric accuracy andthereby a better online visibility. The arrival of the Internet
changed the landscape of scientific publication. More andmore researchers rely on online search engines to findrelevant literature. The online visibility of articles therefore
becomes increasingly important, and the 2008 RAE
demonstrates how important bibliometric analysis has
become, even for the progress of scientific careers.
First of all, we would like to point out that the ranking orderof GS results depends on the citation count. The more often
an article has been cited, the higher it will appear in the
results page. This ranking system is likely to strengthen the
Mathews Effect, which states that highly cited papers will
get even more citations, while rarely cited papers willremain rarely cited. It is therefore of utmost importance togain at least a few citations for each publication, because
many users of GS do not browse through more than one or
two result pages before they adjust their search query.
Being in the top-ten is key. The following practical tips
might appear to many as obvious, but our experiences withthe analysis of the CHI proceedings indicate that manyothers are still not aware of them. At the same time wewould like to encourage Google to improve its algorithm bytaking the problems listed below into account.
GS has problems with special characters, such as nameswith diacritics “ä, é, ç”, apostrophes “O'Reilly”, andligatures “æ, fi, ß” [25]. We can fully support theirobservation, and would like to add that authors should use
one consistent version of their name, ideally with all theirinitials. This increases the chances of a correct
identification.
The same holds true for the titles of papers. We observedthat GS has problems with extremely long titles, the umlaut,
and typographic stylization such as superscript, subscript,
italics, and bold. Punctuation marks can also bechallenging. The em dash “—“ is certainly moreaesthetically pleasing than an ordinary hyphen, but it causestrouble with bibliometric analyses. It is sad, but we have torecommend refraining from any stylistic ambitions in thetitles of papers.
Needless to say, it is never a good idea to publish multiple
articles with exactly the same name, even for an article thathas matured from a workshop paper to a journal article.
From the perspective of the h/g-index, it does not pay topublish the ideas of an article repeatedly. The citations will
simply be split over several papers instead of focusing on
one. To receive a high h/g-index, it is necessary to makeevery paper count. It is also advisable to choose a title thatis specific. A paper entitled “Web Usability” will have tocompete for attention with about 1700 papers. It might beworthwhile to first check the tentative title with GS toensure that it is sufficiently infrequent.
Lastly, we would like to recommend that authors post theirpapers in their organizations’ repositories or on their own
home pages. The ACM explicitly allows authors to “postauthor-prepared versions of the work covered by ACM
copyright in a personal collection on their own Home Page

707

CHI 2009 ~ Scientometric Analysis of the CHI Proceedings
and on a publicly accessible server of their employer”(ACM Copyright Policy, Version 4 Revised 11/01/02,
Section 2.5 “Rights Retained by Authors and Original
Copyright Holders”). It has been shown that such an Open
Access strategy improves the impact of articles [34]. The
combination of authors publishing their articles in open
repositories and the power of GS in indexing them may
hopefully lead to an efficient and effective Open Accesslibrary. Since not every organization has the financial
resources to access the ACM Digital Library, this istherefore not only a good strategy for increasing online
visibility but also an active means for lessening the gap
between the developing and the developed world.

April 7th, 2009 ~ Boston, MA, USA

submitted again. Behavioral and Brain Sciences, 5, 2 1982),
187-195.
[14] Holden, G., Rosenberg, G. and Barker, K. Bibliometrics insocial work. Haworth Social Work Practice Press,
Binghamton, 2005.
[15] Moed, H. F. Citation analysis in research evaluation.
Springer, Dordrecht, 2005.
[16] Garfield, E. Is citation analysis a legitimate evaluation tool?
Scientometrics, 1, 4 1979), 359-375.
[17] MacRoberts, M. and MacRoberts, B. Problems of citation
analysis. Scientometrics, 36, 3 1996), 435-444.
[18] Lawrence, P. A. Lost in publication: how measurement harms
science. Ethics in Science and Environmental Politics, 8, 1
(June 03, 2008 2008), 9-11.
[19] Universities UK The use of bibliometrics to measure research
quality in UK higher education institutions. Universities UK,
2007.
[20] Katz, J. and Hicks, D. Desktop scientometrics.
Scientometrics, 38, 1 1997), 141-153.
[21] Pauly, D. and Stergiou, K. I. Equivalence of results from two
citation analyses: Thomson ISI’s Citation Index and
Google’s Scholar service. Ethics in Science andEnvironmental Politics, 20052005), 33-35.
[22] Jacso, P. As we may search–comparison of major features ofthe Web of Science, Scopus, and Google Scholar citationbased and citation-enhanced databases. Current Science, 89,
9 2005), 1537-1547.
[23] Jacso, P. Dubious hit counts and cuckoo's eggs. Online
Information Review, 30, 2 2006), 188-193.
[24] Jacso, P. Deflated, inflated and phantom citation counts.
Online Information Review, 30, 3 2006), 297-309.
[25] Harzing, A. W. K. and van der Wal, R. Google Scholar as a
new source for citation analysis. Ethics In Science And
Environmental Politics, 82008), 61-73.
[26] Meho, L. I. and Yang, K. Impact of data sources on citation
counts and rankings of LIS faculty: Web of science versus
scopus and google scholar. Journal of the American Society
for Information Science and Technology, 58, 13 2007), 21052125.
[27] Hirsch, J. E. An index to quantify an individual's scientificresearch output. Proceedings of the National Academy of
Sciences of the United States of America, 102, 46 (November
15, 2005 2005), 16569-16572.
[28] Egghe, L. Theory and practise of the g-index. Scientometrics,
69, 1 2006), 131-152.
[29] Marzal, A. and Vidal, E. Computation of normalized edit
distance and applications. IEEE Transactions on PatternAnalysis and Machine Intelligence, 15, 9 1993), 926-932.
[30] United Nations Department of Economic and Social Affairs.
World Population Prospects. New York, 2006.
[31] Orr, J. M., Sackett, P. R. and Dubois, C. L. Z. Outlier
Detection And Treatment In I/O Psychology: A Survey Of
Researcher Beliefs And An Empirical Illustration. Personnel
Psychology, 44, 3 1991), 473-486.
[32] Green, R. G. The Paradox of Faculty Publications in
Professional Journals. Haworth Press, City, 2005.
[33] Gold, V. SIGCHI Announces Best of CHI 2008 Award
Winners: Conference Honors Research that Addresses
Problems of Accessibility, Homelessness, Healthcare,
Emerging Markets. ACM, City, 2008.
[34] Lawrence, S. Free online availability substantially increases apaper's impact. Nature, 411, 6837 2001), 521-521.

Acknowledgements

We thank Bernard Rous and Ronald Boisvert of the ACM,
who made the metadata of the CHI proceedings available tous. Razvan Cristescu and Matthias Rauterberg contributed
to this study with their valued feedback.
Appendix

The full lists and rankings are available at:
http://www.bartneck.de/projects/research/chi2009/
REFERENCES[1] Greenberg, S. and Buxton, B. Usability evaluation consideredharmful (some of the time). ACM, City, 2008.
[2] Barkhuus, L. and Rode, J. From Mice to Men–24 Years of
Evaluation in CHI. Twenty-fith annual SIGCHI conferenceon Human factors in computing systems - Alt. CHI2007).
[3] Wulff, W. and Mahling, D. E. An assessment of HCI: issues
and implications. SIGCHI Bulletin, 22, 1 1990), 80-87.
[4] Newman, W. A preliminary analysis of the products of HCI
research, using pro forma abstracts. ACM, City, 1994.
[5] Garfield, E. The History and Meaning of the Journal Impact
Factor. Journal of the American Medical Association, 295, 1(January 4, 2006 2006), 90-93.
[6] Lotka, A. J. The frequency distribution of scientificproductivity. Journal of the Washington Academy of
Sciences, 16, 12 1926), 317-323.
[7] Bartneck, C. What Is Good? - A Comparison Between The
Quality Criteria Used In Design And Science. ACM, City,
2008.
[8] Bartneck, C. and Rauterberg, M. HCI Reality - An Unreal
Tournament. International Journal of Human Computer
Studies, 65, 8 2007), 737-743.
[9] Jefferson, T., Wager, E. and Davidoff, F. Measuring the
Quality of Editorial Peer Review. Journal of the American
Medical Association, 287, 21 (June 5, 2002 2002), 27862790.
[10] Lawrence, P. A. The politics of publication. Nature, 422,
6929 2003), 259-261.
[11] Horrobin, D. F. The philosophical basis of peer review andthe suppression of innovation. Journal of the American
Medical Association, 263, 10 (March 9, 1990 1990), 14381441.
[12] Enserink, M. SCIENTIFIC PUBLISHING: Peer Review andQuality: A Dubious Connection? Science, 293, 5538
(September 21, 2001 2001), 2187a-2188.
[13] Peters, D. P. and Ceci, S. J. Peer-review practices ofpsychological journals: The fate of published articles,708