# On the proper use of bibliometrics to evaluate individual researchers

Denis Jerome, Jean-François Bach

► To cite this version:

Denis Jerome, Jean-François Bach. On the proper use of bibliometrics to evaluate individual researchers. Académie des Sciences. 2011. hal-05034021

HAL Id: hal-05034021

https://hal.science/hal-05034021v1

Submitted on 14 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

INSTITUT DE FRANCE
Académie des sciences

ON THE PROPER USE OF BIBLIOMETRICS TO EVALUATE
INDIVIDUAL RESEARCHERS

<figure><img src="image_3.png" /></figure>

Report presented on 17 January 2011
to the Minister of Higher Education and Research
VIII. Conclusion ................................................................................................................ p. 27

Annex 1 : Composition of the working group and experts consulted ................................ p. 29

Annex 2 : Summary and recommendations of the Report published in 2009 by the Académie Evaluation of individual researchers in theoretical and experimental sciences ................................................................................................................ p. 31

Annex 3 : Bibliometric practices by disciplines................................................................ p. 37

Annex 4 : Publication ethics for scientific works – The Vancouver Criteria .................... p. 48

Annex 5 : Glossary ................................................................................................................ p. 65
# On the proper use of bibliometrics to evaluate individual researchers

## Summary

Individual evaluation of researchers is still a subjective process that suffers from numerous potential biases. The Académie has examined the use of quantitative bibliometrics, which are considered to be more objective, and has made a number of recommendations on rigorous rules that should be followed when using bibliometrics to support qualitative evaluations. Such rules should be recognized internationally, at least at the European level. The issue of bibliometric evaluation is a complex one and is still being debated. Strong opinions have arisen for and against its use that depend greatly on the scientific field.

## I - Importance and limits of peer evaluation

### I - 1 I Importance of peer evaluation

Peer evaluation has long been the only way to evaluate researchers. It is irreplaceable for assessing the scientific contribution of a researcher in terms of original ideas, quality of work, conceptual and technological innovation, and more generally assessing the impact and dissemination of the researcher's work.

### I - 2 Limits of peer evaluation

Such evaluations pose practical problems linked to the enormous effort required to examine applications in detail that is amplified by the excessive number of evaluations requested by administrative bodies. Furthermore in a number of cases, peer evaluation can be tainted by subjectivity and in some cases by the insufficient expertise of the evaluators, potential conflicts of interest, group processes and favoritism. All ethical issues should be reported in writing by evaluators, as was suggested by the Académie in its Report of 8 July 2009 to the Minister.

In spite of such flaws, bibliometrics cannot be a substitute for qualitative peer evaluation, although experts of a particular field can use bibliometrics, with all due precautions, as a tool to help in the evaluation.

## II - Basics of bibliometrics

The term “bibliometrics” already generates confusion. It does not measure a researcher’s production but citations to his/her publications. It is based on the calculation of various indices (number of citations; integrated factors, such as the *h* factor; and others) based on bibliographic databases that cover all, or almost all, scientific publications and citations in most disciplines.
## II - 1 Main indices

Several bibliometric indices are used frequently. The number of publications has little value because it does not take into account the quality of the publications. The total number of citations is more informative, but suffers from certain biases, in particular the exaggerated weight of one or two highly cited articles in spite of the fact that they may not particularly be more important. Integrated factors, such as the very widely used *h*- and *g*-indices, usefully complement the number of citations. Finally, the impact factor measures the journals and not researchers, but it is often taken into account to evaluate the quality of an article. This practice, widely used in some disciplines such as biology and medicine, is dangerous because many prestigious journals with a high impact factor also contain a significant percentage of publications of average quality. The fact remains, however, that the publication of an article in one of the highest-level journals represents an element of recognition, provided that the researcher has contributed significantly to the work in question. It is also important to note that there are quantitative criteria for evaluation that are not strictly speaking bibliometric, such as number of invited conferences, awarding of important grants, prizes, patents and software development.

## II - 2 Databases

Databases are of good quality and constantly improving for most disciplines, but it should be kept in mind that not all disciplines are covered (especially those in the Social and Human Sciences). Care must be taken that the persons who compute the bibliometric indices have complete access to the best databases. Databases can be usefully supplemented with descriptive entries for each article referenced as is done by the *Mathreviews* database for Mathematics (bibliographic file with comments).

## II - 3 Advantages and potential drawbacks of bibliometrics

Bibliometrics is seemingly easy to use and provides an evaluator with numbers that are attractive for their simplicity and factual nature. It involves nevertheless numerous biases. It is important to mention that in order to carry out bibliometrics in an unquestionable fashion, time, rigor and experience are necessary. It is also essential to remember that no index or set of indices alone can summarize the quality of a researcher's scientific production. Moreover, the importance of bibliometrics in some disciplines may encourage researchers to adapt their publications and even their work to the journal in which they wish to publish their articles rather than engaging in original and creative research.

## II - 4 Validating data

The calculation of indices can lead to many errors as evidenced by their variability in the databases. This report presents the main weaknesses of bibliometrics and how to avoid them. Ideally, as his own best expert, a researcher should calculate his own indices (in the disciplines where the databases are available) before submitting them for validation by persons in charge of indices at the level of a research institution or academic establishment. The idea of a unique identifier associated with each researcher has been adopted by some databases. Researchers should also provide the review panel with the electronic *pdf* files of the main publications listed in their application so that any use of bibliometrics can be supplemented by the examination of the work itself.
## II - 5 Distribution and reference values

Bibliometric indices have no intrinsic value. They can only be understood relative to the distribution of index values for a particular field and by taking into account the age of the researchers concerned.

## II - 6 Authorship

In some disciplines, especially in biology, the position of a researcher's name in the order of authors to a publication is of considerable importance as it reflects the personal contribution of the scientist to the work published and consequently the notoriety that he/she may gain. Significant efforts must be made when computing bibliometric indices to ensure that articles of a particular author are treated differently depending on the position of his/her name in the list of authors. More generally, publication lists should specify the exact contribution of each author, especially concerning the "short lists" provided by candidates. This point should be given further consideration.

# III – When and how to use bibliometrics

## III - 1 When should bibliometrics be used?

In the case of peer panels covering a single discipline where members usually know the candidates well, recourse to bibliometrics is not necessary except for a quick overview. In this context, values of indices should not be considered a decisive element.

In the case of interdisciplinary panels, it may be useful to rely on bibliometrics to speed up the process when making a first selection among candidates, provided panel members keep in mind the considerable differences that exist between disciplines and sub-disciplines.

Bibliometric indices are of no value when evaluating young scientists just at the beginning of their career. Bibliometrics should only be used when recruiting senior scientists.

## III - 2 How should bibliometrics be used?

Bibliometrics should only be used in conjunction with a qualitative evaluation (except for the first round of candidate selection as mentioned above). More generally, indices should be adapted to take into account both the length of a career since their value increases cumulatively with age and eventual changes in productivity or thematic orientation during a career. Indices should not be the same or should be given a different weight depending on the objectives of the evaluation: recruitment, promotion, awarding of grants or distinctions.

In accordance with international practice, general bibliometric data should be accompanied by a close examination of the 5, 10 or 20 best publications (depending on the field and scientific seniority) chosen by the candidate. Thus, jury members should not merely rely on the numbers provided by bibliometrics, they should also take into account all the bibliographic comments linked to the publications chosen by the candidate.

In those cases where the final evaluation does not correspond to the bibliometric indices, explicit explanation for the reasons why a particular piece of work was judged very important by the panel in spite of its few citations must be provided. Bibliometric indices should be
systematically included in a candidate's application as a tool for aiding evaluation, but should
certainly not be the sole criteria.

IV – Specificity by discipline

Disciplines, and even sub-disciplines, each have their own specificity in matters of
publications and use of bibliometrics. This constitutes a potential major bias that should be
taken into account when evaluating a researcher and should also be tied to the size of the
particular scientific community. Bibliometrics does not allow a comparison of researchers
from different disciplines and even sub-disciplines.

Apart from the size of the scientific community which impacts on the total number of
citations in a particular field, there are significant specificities, such as the absence of good
databases in Social and Human Sciences, a reluctance on the part of the mathematics
community to use bibliometrics and major differences in the number and order of authors
listed in articles.

V – Improvement of bibliometrics

Well used, bibliometrics can become a useful tool in the hands of peers. The Académie recommends that the following studies be carried out in order to improve the unofficial and all too frequent use of bibliometrics:

**V - 1 Retrospective tests** to compare the decisions actually taken by peer panels (CNRS, IUF, ERC) and the results of a purely bibliometric-based evaluation of the candidates. Similar studies previously undertaken in France by the CNRS should be consulted and further investigated as well as those carried out in other countries, in particular by their Academies.

**V - 2 Studies to refine existing indicators** and define relevant bibliometric indices to use in the context of individual evaluations, where the usage of bibliometrics has appeared only relatively recently. There should be an in-depth examination of the notion of authorship. Creation of a steering committee for individual bibliometrics within the framework of the Observatoire des Sciences et Techniques (OST).

**V - 3 Development of standards** that discern originality, innovation, diffusion and creation of schools of thought, in particular through the history of recent major discoveries in the context of bibliometrics (Fields medals, Nobel prizes, Gold and Silver medals of the CNRS, etc.)

**V - 4 Establishment of rules of good practice for the use of bibliometrics** during researcher evaluation in response to a request by the national evaluation agency for higher education and research (AERES), one of the missions of which is the validation of evaluation procedures for researchers.
**On the proper use of bibliometrics to evaluate
individual researchers**

**Recommendations**

**Recommendation 1: The use of bibliometric indices for evaluating individual researchers is of no value unless a number of prerequisites are met:**

- The evaluation should focus on the articles and not the journals.

- Data quality, standardization, significance of deviation and robustness of indices must be validated.

- Bibliometric evaluations should only compare researchers in the same scientific field and over their whole career. It is important to consider bibliometric data against the specific distribution of values of the researcher's field and also to take into account the rate of career progression.

- Users of bibliometrics must justify their conclusions. It will force them to develop a solid expertise in this area.

It is important to be aware that some researchers might chose to steer their activity in such a way as to get articles accepted in journals with a high impact factor rather than engaging in original and creative research and persisting with a thematic continuity, at least for several years.

Finally, since evaluations are based on peer judgement, the question arises as to whether the evaluators should not themselves be submitted to a bibliometric evaluation.

**Recommendation 2: Bibliometrics should not be reduced to numbers, it must be accompanied by an in-depth consideration of bibliometric and bibliographic data, and if possible the articles themselves**

It should be pointed out that some French Fields Medal winners in mathematics and Nobel laureates in chemistry and physics have surprisingly very modest bibliometric indices.

- Any bibliometric evaluation should be tightly associated to a close examination of a researcher's work, in particular to evaluate its originality, an element that cannot be assessed through a bibliometric study.

- The Académie recommends that for all individual evaluations, especially in cases where the panel cannot reach a consensus, a close examination of the bibliometric data of the 5, 10 or 20 most cited articles (or those chosen by the candidate) should be undertaken along with a close scrutiny of the bibliographic comments accompanying these publications. Such a selection and the respective electronic *pdf* files provided by the scientist would facilitate close examination of his/her work.
- A comparison of the citations of a researcher's article in a given journal to the mean number of citations within same journal over a given period is envisaged. This will add value to articles that are frequently cited in low impact journals.

- A comparison of the number of citations of an article to the statistical data of another article published at the same time and in the same field should also be undertaken.

- It would be interesting to know where a given article stands compared to the most cited articles in the field: within 0.01%, 0.1%, etc.? The ISI sub-database *Essential Science Indicators* (see *Additional Resources*) greatly facilitates this examination in the major disciplines. Further analysis by sub-disciplines may be necessary. In the ISI bibliographic files, it is also possible to check how citations changed over time and who has cited the article.

- Qualitative and (semi-quantitative) bibliometrics would be useful in certain close examinations where the quality of the citations and their quantification is made: knowing which articles (or types of articles) have cited a given article (or person) not only can reveal who has appreciated the work but also be used to assess its interdisciplinarity, longevity, scope and timeliness.

- Concerning a bibliographic analysis, we recommend that the example of the *Mathematical Reviews* database be encouraged and extended to all other fields.

**Recommendation 3: Bibliometric indices should be used differently depending on the purpose of the evaluation, such as recruitment, promotion, grants and distinctions.**

- Bibliometric indices should not be used for researchers with a career spanning less than 10 years in order to prevent their only pursuing research in areas of high citation levels. This would impede researcher creativity at the start of a career.

- Bibliometrics should also be excluded when recruiting young researchers. At the *chargé de recherche* CR2 (researcher) or *maîtres de conférences* (lecturer) levels, a candidate has only a small number of publications. The panel must read and try to understand with greater care the works proposed by the candidate.

- In the case of recruitment for or promotion to senior positions, bibliometric indices can be used by the peer panel (see below).

- In the case of promotion to senior research or teaching positions, using indices and bibliometrics can help to establish a distribution of the candidates and to eliminate those whose performance is too weak.

- Recruitment for senior level research or teaching positions is closer to the preceding case than to that of young persons. A preliminary screening through bibliometrics is thus possible when there are too many candidates.

- In cases where the final evaluation does not correspond to the bibliometric indices, explicit explanation for the reasons of the decision taken by the panel must be provided.

- Bibliometric evaluation of candidates applying for a research grant or an award (prize, medal, election to an academy among others) must be treated differently according to the context and the age of the researchers and greater importance must be given to the originality of the work which generally is not properly taken into account by bibliometrics.
**Recommendation 4: Greater importance should be given whenever possible to the position of a researcher's name in the order of authors and the exact contribution of each author**

When an article is signed by several authors, the position of a researcher's name in the order of authors is of considerable importance as it reflects the personal contribution of the scientist to the work published. In disciplines where it is usual to list numerous authors or in disciplines where authors are listed in alphabetical order or according to other variable and complex rules, it is not possible to easily judge the contribution of any one author.

- Articles to which a given author contributed significantly and articles where the author was only a collaborator should be treated differently.

- The concept of authorship needs to be clarified. We recommend that all journals in all fields use the Vancouver authorship criteria (see annex 4).

- It may be useful to also get information on the other authors of an article.

**Recommendation 5: Bibliometric evaluation should become an object of study in order to improve its value. France must participate in this process.**

All the recommendations above need to be further examined. In order to do so, the Académie recommends the creation of a Steering Committee to examine the use of bibliometrics in individual evaluations, for example within the framework of the Observatoire des Sciences et Techniques (OST) which is a public body with a long experience in bibliometrics. It would be composed of a small group of experts from various disciplines and agencies, whose task will be to study the limitations of indices and their use and suggest how to improve them. This committee should engage in research that will help refine existing indices and make practical suggestions to be validated at the European level. Its recommendations should be based on a number of tests and studies such as retrospective tests and the development of criteria to detect originality, innovation, dissemination and impact of a work.
# ON THE PROPER USE OF BIBLIOMETRICS TO EVALUATE INDIVIDUAL RESEARCHERS

## Introduction

Bibliometrics has played an increasing role in evaluating individual researchers (the focus of this report) as well as research groups and institutions. This can be explained by its ease of use and the overview it provides on a researcher's career. At the same time, bibliometrics appears not to have been always well used and has proven an object of serious wrongdoing when used in isolation.

In its Report of 8 July 2009, the Académie des Sciences emphasised that peers should play a decisive role in the individual evaluation of researchers (see Annex 2). Unfortunately, there have been many cases of improper and poor qualitative evaluation by peer panels due to conflicts of interest, favoritism, local interests, group processes, insufficient expertise of evaluators, superficial examination of applications. The question thus arose how to ensure better execution of peer evaluation.

To overcome such shortcomings, the evaluation of the impact of a researcher's work based on quantitative analysis, which is considered to be more objective, was suggested for certain disciplines as a tool to help qualitative evaluation by peers. Bibliometrics commonly refers to this use.

It should be pointed out that bibliometrics is not necessarily objective and that it suffers from many biases. It is usually reduced to a few numbers and used in an extremely reductive manner in spite of the fact that current databases from which these indices are computed hold an enormous amount of information which, properly taken into account, could significantly help qualitative evaluation.

This report focuses on the use of lists of publications and indices based on the citation of these publications. The report will review the current situation and explore new directions for improvement.
# I. Bibliometrics definition and objectives

Bibliometrics, or better bibliometric evaluation, usually refers to a series of procedures that contribute to evaluating the scientific production of a scientist (or a group of scientists) on the basis of the number of publications, the prestige of the journals in which articles are published and citations to these publications. Clearly, bibliometrics does not measure the quality of a researcher's work but only citations to the work, without prejudging the reasons that led to the citation. As will be seen in this report, several indices have been suggested to serve as a base to individual bibliometric evaluation. It is important to state at the outset that no single index can by itself lead to an adequate evaluation of a researcher's work nor does reliance on several indices. The term *bibliometrics* itself is even somewhat regrettable since it includes the root *metric* which implies a concept of measure while the bibliometric unit of measure varies according to disciplines and sub-disciplines.

Everyone agrees that all scientific activity must eventually lead to an adequate dissemination of its results. This usually takes the form of publications in peer-reviewed scientific journals and, in some disciplines, other forms (such as open archives, conferences, books) that reflect the contribution of a researcher to the scientific progress of his/her field. With time, it has become evident over the years that the hierarchy established between scientific journals has led researchers to preferentially submit their articles to journals with the greatest prestige. Publishing in these "good journals" has become an objective that has in turn given notoriety. Therefore, quite naturally in the case of equally good articles, those published in these journals will be cited more often than those cited in less prestigious ones. Similarly and closely linked to the preceding observation is that the best articles usually give rise to a high number of citations easily counted by current computing means. This has led to the hypothesis that the number of citations correlates to the importance of an article. These concepts form the basis of bibliometrics use, which historically was designed to define scientific fields and later to evaluate journals.

Bibliometrics generated great enthusiasm within most of the scientific community. Its use seemed easy and allowed for a rapid and therefore less expensive evaluation of a researcher's work than qualitative examination. However over time, due to its ease bibliometrics came to be excessively utilised at the expense of qualitative evaluation. Sometimes it was used in a hidden and improper way because users were unfamiliar with its many shortcomings and used non-validated data.

This report treats all these topics with the retrospection needed to consider bibliometrics within the context of scientific evaluation. It puts forward recommendations for a better use of bibliometrics and for technical improvement of the procedures regarding its use.
## II. Respective roles of bibliometrics and qualitative evaluation by peers

Knowing how to evaluate scientific work and hence the quality of the results produced by researchers who publish the work is important. It is essential for the recruitment and promotion of researchers and the awarding of distinctions and prizes. It is also fundamental when deciding the amount of individual research grants. Finally, it is very useful in assessing the quality of the authors of a piece of work or article. This large number of objectives itself constitutes a problem since it leads to the continuous evaluation of researchers, which is time and energy consuming for both the evaluators and evaluatees and is required in addition to other time-consuming tasks such as peer-reviewing of manuscripts. For many years, before bibliometrics was available, qualitative evaluation was limited to just considering the number of publications a researcher produced. Later in this report, we will discuss the biases of this procedure. Fortunately bibliometrics was –and still is– complemented by the qualitative analysis of the work, most often based on scientific articles, patents or the impact of the discoveries made at a fundamental or applied level. In most cases, scientific evaluation was carried out by experts in the same field, “peers” meeting as a panel or committee. When it is carried out to evaluate individuals, such qualitative evaluation takes into account other criteria in addition to scientific work (such as teaching and collective interest activities), however this is not the place to discuss the procedures of qualitative evaluation. The Académie published in 2009 a detailed report on the topic (http://www.academiesciences.fr/actualites/textes/recherche_08_07_09.pdf). It should be remembered that several of these additional criteria include quantitative elements, especially the number of patents (to be modulated by the issuance of an industrial licence), the number of invited conferences or international grants obtained, the development of software in computer sciences, to which can now be added job offers to change laboratories (in the U.S.A, for instance the well-attended *March Meeting* in Physics is a platform for job opportunities) and other quantitative indices used mainly in Anglo-Saxon countries that are not covered by bibliometrics. The real problem is that of the respective roles of qualitative peer evaluation and bibliometric evaluation.

### II. 1 The weaknesses of peer evaluation

At this point it would be worth mentioning that bibliometrics developed and came to be used in part because of inadequate qualitative evaluation in some disciplines. The first report of the Académie, mentioned above, presented the most frequent shortcomings: the quality of the evaluators; their personal ethics; their objectivity; the transparency and quality of the evaluations; and the superficial analysis of the candidates’ work in part due to the excessive number of panels evaluators are required to sit on. In short, although it is necessary to avoid an excessive reliance on bibliometrics, it is important to keep in mind that bibliometrics is necessary to improve qualitative evaluations.

The issue is in fact complex. First of all, the value of qualitative evaluations varies according to disciplines and institutions. It is clear that in most disciplines, qualitative evaluations clearly include some elements of bibliometrics, whether these are used directly, indirectly,
knowingly or unknowingly. As previously and repeatedly mentioned, the fact remains that
evaluations must fundamentally be qualitative even if they involve the use of bibliometrics as
a tool, a use that is usually justified. The variable quality of qualitative evaluations makes it
difficult to use them as a standard against which to validate bibliometric evaluation. We will
expand later on this real difficulty. Briefly, qualitative evaluation should be improved, when
necessary, by eliminating the conflict of interests and incompetence of the evaluators and by
integrating bibliometrics in the most pertinent and discipline-specific manner possible.

## II. 2 Bibliometrics as an evaluation tool

Bibliometrics has obvious advantages. It is seemingly easy to carry out and provides factual
elements of information when properly used. It has a considerable disadvantage in that it
summarizes with numbers, in a potentially biased way, the scientific production of
researchers without taking into account the multiple complexities involved in assessing the
originality and quality of scientific work. Furthermore, the fact that the pertinence of
bibliometrics, and consequently its use, differs hugely among different disciplines and sub-
disciplines must also be taken into account – we will examine this in detail later. These
important observations explain why any serious evaluation should remain based on
qualitative evaluation by peers. It should be noted at this stage that the members of the
present workgroup all agreed with the observation that bibliometrics is no panacea but only
a tool to be used wisely by peers. The latter already use bibliometric tools knowingly or
unknowingly, in a direct or indirect way, for example when letters of recommendations that
are often based on bibliometric criteria are joined to a researcher's application. In any case,
evaluators examine the list of publications fully aware of the quality of the journals in which
the articles are published and one would hope that they also examine the articles
themselves. Whatever reservations one may have regarding bibliometrics, it must be
acknowledged that it has a place in many disciplines (but certainly not all in all, in particular
not in Mathematics and Social and Human Sciences (SHS), we will come back to this point
later). To deny its interest is both unjustified and useless because it will continue to be used
anyway. It seems more appropriate to identify its limitations and to elaborate good use
practices.

It should be mentioned that to assess the scientific production of an institution, a scientific
community, a region or a country, a quantitative evaluation is quite appropriate. The use of
bibliometrics can be very useful and even essential in such a context. The only requirement
is that the indices used be sufficiently pertinent.

## II. 3 The pitfalls of bibliometrics

Before going into technical details regarding bibliometric procedures, it should be
emphasized that the general concept that the more a scientific work is cited the more
important it must be an oversimplification. There are many reasons why an author
references an article other than the quality of the work. Whatever the motivation of the
author, all references are equally treated as citations. It is well known that “important”
articles are preferentially cited, and that bibliographic references are often chosen based on opportunism rather than just the quality of the work cited. Articles published in prominent reviews are favoured compared to those of equal quality published in lesser journals (the authors may think that their own articles may gain extra value by citing references published in distinguished reviews). In certain cases, authors believe, although it has yet to be proven, that it helps to reference articles published in the journal where the manuscript is to be submitted. Such a practice, disavowed by the Académie, is encouraged in certain disciplines by the scientific publishers. To these shortcomings must be added the biases created by self-citations, citations of prominent colleagues (potential reviewers of the submitted article) or with whom the author has personal relations or collaborations, the non-citation of competitors or even some network dynamics that encourages preferential citation among members of a scientific group. The minor but still significant practice of frequently publishing papers that only serve to underline errors in the results or their interpretation should be mentioned. Furthermore and depending on cultural specificities, preference may be given to citations to scientists of the same country, or of different countries, especially if these are American as is often the practice in France, or to articles written in English rather than in French in disciplines where language is an issue (mathematics, SHS). On the other hand, some articles may not be cited because they have become quasi-classics or because they are too unusual.

Another shortcoming of bibliometrics is undoubtedly the excessive importance given to it by some scientific fields such as biology and medicine. An article is “important” because it is published in a prestigious journal, *Nature* or *Science*, although it is well known that such excellent journals also contain articles of lesser interest that get few citations (over 50% of articles published in *Nature* have received since 2008 no or at best only one citation). Even worse is the tendency by some researchers to organise their work and their publication strategy according to the journals in which they hope to publish their results so as to improve their bibliometric performance at the expense of originality and boldness. To publish in a prominent journal becomes sometimes a goal more important than the scientific objective of the work. Such publication “professionals” gain a bibliometric advantage, a trend that is unfortunate and far from exceptional. It is interesting to note that, according to *Physics World* (November 2010), the two pieces of work that were rewarded the Physics Nobel Prize in 2010 had been refused twice by *Nature* before being finally published in *Science*. It is astonishing that highly important works were not accepted for publication by *Nature*!

Finally, as highlighted in the first report of the Académie, bibliometrics does not take into account a certain number of elements that are important in evaluating full-time researchers and academic researchers, in particular originality of the research, conceptual innovation, research applications, scientific and industrial utility. It should be added that the diffusion and impact of a work can be measured through a bibliometric study of the collaborators of the researcher considered.
All these pitfalls are serious and to them must be added the numerous sources of error in the use of bibliometrics that are detailed below. All these different elements should make us exceedingly cautious about its use. It should be said again that it is impossible to evaluate a researcher solely on the basis of quantitative indices. This is immediately evident from the fact that a number of renowned researchers, in particular Nobel laureates, have extremely low bibliometric indices while, inversely, the contribution of some researchers with high indices is not as important as might be expected from the value of their indices compared to the rest of the scientific community. In this respect, the example of the two 2010 Fields Medals, Cédric Villani who was cited 1520 times by 629 authors while Ngô Bao Chau who was cited only 102 times by 52 authors is worth mentioning since no mathematician would see a disparity of level between these two laureates. There are many similar examples that show that works of great significance have been very little cited in the years immediately following their publication and came into the prominence only much later.

### III. Diversity of customs and practices among disciplines

It should be noted at the outset that we do not know of any other country where bibliometrics is officially used for evaluating scientists individually, although its use in practice is widely known.

Many Anglo-Saxon countries use bibliometrics officially to evaluate the performance of their universities and research bodies. Studies of bibliometrics have developed considerably in the last 20 years and led to a copious and increasing number of publications on the topic as evidenced by the interest of such a journal as *Nature* (for the most recent issues concerning bibliometrics, see *Nature* 17 June 2010 and 8 July 2010).

At a personal level, the results of a survey carried out by *Nature* of 150 scientists and department heads (volume 465, page 860, issue of 17 June 2010) show that 70% of those surveyed thought that bibliometric indices were used for recruitment and promotions but 63% thought the use of quantitative measurements was inadequate. This proves that in every country, using only bibliometrics in evaluations is not perceived as satisfactory.

American and British universities and research bodies rely much more on curriculum vitae, interviews and recommendation letters than bibliometric considerations for hiring and promotion. By contrast, bibliometrics is widely used by Chinese and Asian universities in general for academic hiring, but there is an increasing trend towards a greater reliance on recommendation letters.

Bibliographic and bibliometric practices vary significantly between disciplines and even sub-disciplines. The variability concerns the use of bibliometrics as well as the quality of the
databases. Such differences, to which must be added the considerable disparity in the size of
the scientific communities concerned, are reasons why we should avoid generalization
regarding the attitudes towards bibliometrics and refrain from comparing bibliometric indices
of researchers belonging to different disciplines or sub-disciplines. The bibliometric practices
of the main disciplines are detailed in Annex 3.

There is a considerable difference in practices observed between on the one hand Mathematics and Social and Human Sciences, which rely little or not at all on bibliometrics to evaluate researchers and, on the other hand, other disciplines such as Biology and Medicine which use it widely. Other significant differences should be noted, in particular concerning the number and order of authors on a publication. As will be seen in the next chapter, the order is alphabetical in some disciplines while in others it reflects the contribution of each author. Scientific communities also have their own respective standards as to the amount of work required to be listed as an author. Finally, the propensity to cite is also a cultural trait that differs between scientific communities and is evident for example by the number of citations journals will allow at the end of an article. These remarks should provide an incentive to always consider bibliometric indices within the context of a particular discipline and to always refer to the distribution of indices in that discipline.

IV. **<u>Authors and the importance of their respective contributions</u>**

The issue of authorship is an element that depends on the discipline. Customs and practices
in matters of writing and authorship vary according to disciplines and even sub-disciplines.

The problem is most acute in Biology because the Life Sciences have an intensive
bibliometric culture, especially in France. In this field, the average number of authors varies
between 5 and 10 and sometimes more. In practice, the first author is the student or post-doc
who did the work, the second author is the person closest to him/her, then starting backwards
from the last name, the list includes by order the thesis director, the group leader, the
laboratory head, etc. The difficulty resides with the authors whose names are in the middle of
a list and have contributed less than the other authors but who get bibliometric recognition.
This creates a confusion between authors and collaborators that often results in over-rated
citation indices for some scientists and becomes an element of serious abuse. To this should
be added the issue of "corresponding authors" who communicate information that is not
always accurate on the respective contributions of each author.

The question is equally serious in the Medical Sciences where the research activity of a
hospital has significant financial consequences on its budget. This activity is measured based
on the publications of its researchers and on the basis of a point system awarded to
researchers according to the place of their name in the lists of authors.
By contrast, in Mathematics, the question does not arise since over half the papers have only one author, less than 10% have three and a very small percentage have more. The order is strictly alphabetical. In fundamental mathematics, researchers rarely publish with their students contrary to what happens in applied mathematics.

In Physics and its sub-fields, customs are varied. In experimental particle and high energy physics in particular, hundreds of authors are listed and their respective contributions are hard to identify. In these fields, authorship does not influence laboratory financing nor researchers' careers. There are no particular problems in theoretical physics. By contrast, articles in experimental condensed or soft matter physics have a long list of authors (often more than ten), especially when studies use large equipment. In physics, the laboratory head no longer gets systematically included in the list of authors to an article.

In Geosciences, just like in biology, articles usually have less than 10 and frequently less than 5 authors. In general, the order correlates with the importance of the contribution, by decreasing order. The first author is usually the person who has done most of the work, usually a doctoral student, but sometimes it can be the principal investigator or a senior researcher who contributed the essential ideas. Sometimes but rarely the last author is the group leader. Increasingly, technicians are named as co-authors.

In Chemistry, bibliometrics is not officially used, however the usual indices (h factor, total number of citations and number of citations per article) are widely mentioned during preliminary discussions when evaluating the career of researchers who have been active for more than ten to twelve years. The size of the community and the international dissemination of the work ensure an appropriate use of these indices by highly qualified evaluators. In practice, the chemistry sections of the French national centre for scientific research (CNRS) and national council of universities (CNU) committees avoid taking into account bibliometric indices.

In Economics, authors are listed in alphabetical order, which makes the identification of each author's contribution difficult. The number of authors is limited (most publications have only a single author and less than 5% have more than four). By tradition, all authors are considered as having made equal contributions to the work.

In Sociology and many SHS disciplines, university professors very frequently use their students' work without giving them the status of author. The practice is changing and authorship by several contributors is becoming less rare, but the reticence of publishers leads to a reduction in the number of authors. The problem of author number limitation does not exist for scientific journals, however the order is nearly systematically alphabetical and gives no indication about the contribution of each author. In Social and Human Sciences, works can be published in many different forms such as single author books or collective books.
To these examples should be added the particular cases of some research areas, in particular
in emerging fields, which often bring together whole research groups and where publications
are usually signed by a multitude of authors.

All the above bring us back to the concept of authorship. Who can or should be considered an author? The person without whom the publication would not have occurred (according to the authorship criteria enunciated by Guy Ourisson), the person capable of defending the contents of the publication in front of peers, or according to other criteria?

Once the author or authors have been identified, their respective contribution should be explicitly and clearly indicated, and this is already the case in some English and American journals (for instance *PNAS* and *Nature*). The current and unchallenged system for patent authorships, where a percentage corresponding to each author's contribution is applied, could be copied. In practice, such a system can be rather complicated and unjust especially if there are numerous authors. It should be recalled that although an experimental piece of work is usually a collective effort, the original idea on which the work is based is often that of one person.

In this context, it is interesting to recall the Vancouver authorship criteria (http://www.icmje.org/ethical_1author.html) (see Annex 4). Unfortunately, very few people known about them. They were used to establish criteria of authorship for medical articles but we suggest that they be applied to other disciplines and strictly followed.

We close this chapter with a few remarks. The authorship problem could be minimised by requiring evaluators to examine a selection of the candidates' articles. These articles (5, 10 or more according to the context) should be chosen by the researchers concerned. The prestige of the journal and the number of citations should not necessarily be taken into account. It is interesting to observe that the European Research Council (ERC) requests a list of the 10 best publications in which the candidate is the senior author. The bibliometric information concerning the selected articles (journal impact factor, number of citations, discipline, title of the authors of the citations) should be cross-checked against the global indices discussed below. Finally, it would be of interest to examine the coauthor(s) of the candidate being evaluated.

A question that still arises is that of the value that should be given to a citation when a given author's name is situated in the middle of the author-list and the contribution of that particular author to the ideas and execution of the work is known to have been modest. The issue is most problematic when an author is often in this situation. One solution would be to introduce an adjustment factor to the citation, but this would need further exploration. The simplest is still to let peer panels examine the main list of publications of a given author, check the position of his/her name in the list of authors and take this into account when interpreting indices.
Another problem arises when technicians are listed among the authors. This is a serious issue that goes beyond the homage given to such staff, especially when the institution that employs them takes into account publications for their promotion. The contribution of such staff requires specific discussion. In general, with few exceptions, any author listed has made a significant contribution to the published work and it is normal that his/her name should appear among the authors as an acknowledgment from the laboratory to their contribution. Not to make a difference among the contributions is unfair for those who have played a crucial role.

# V. The choice of indices and databases

The term “index” should be understood here in its bibliometric sense, as a factor to help selection committees and guide researchers in evaluating the impact of their work compared to that of their colleagues in the same discipline.

As mentioned above, the most frequently used bibliometric indices are based on the number of articles published and the citations they gave rise to, whether one considers the citations of a given author, a group of authors or citations to articles that were published in a given journal over a defined period of time.

## V. 1 Databases

A number of databases can be used today to compute bibliometric indices. The most frequently used are Web of Knowledge (WoK) by ISI-Thomson (Reuters) and SCOPUS (Elsevier). In this report, we will mostly refer to the ISI database, WoK, since CNRS and UMR researchers as well as most public research institutions have access to the full ISI database through an institutional subscription by the French Institute for Scientific and Technical Information (INIST). Overall, the quality of both databases is good, in particular for chemistry, physics, biology and medicine with a 90% coverage, but they must be used cautiously. Some databases, such as in *Mathreviews*, even contain abstracts of the articles or the letters to the editor. At present, such databases are not suitable for SHS disciplines and they can only be effectively used in only very few areas for these disciplines. The existence of free access databases that are limited to a particular discipline should be mentioned, such as the SAO/NASA Astrophysics Data System (ADS) operated by the *Smithsonian Astrophysical Laboratory* at Harvard, for researchers in astronomy, astrophysics and physics, and hosted in France by the Centre de données astronomiques de Strasbourg (http://cdsads.u-strasbg.fr/). It should be noted that the ADS database mainly covers astronomy, astrophysics and only partially other areas of physics. For researchers engaged in pluridisciplinary work, it is essential to use databases that completely cover all their disciplines.
More generally, there is also the issue of how publications other than original articles listed in databases should be considered. Summaries should not be taken into account, but in this respect again practices are different according to disciplines. In mathematics for example, the fact that a young mathematician at the start of his/her career is an active reviewer of the *Mathreviews* database is quite appreciated by evaluation committees. Letters to the editor, general reviews and editorials are not original articles, but may be considered as representing a notoriety index and should be taken into account in bibliometric analyses. With Web of Knowledge, it is possible to filter the various types of publications of a particular author, such as articles, letters, comments, books, conference proceedings and others. It is worth mentioning that even a very short article can contain an important innovation. The Nobel Prize winner P.G. de Gennes published usually short communications in the Compte Rendus de l'Académie des Sciences.

How far a database reaches back in time is variable from one base to another. The oldest articles (for instance, pre-1975) are not always listed. Regarding the ISI WoK database, nearly all articles are taken into account when a journal is indexed in the database. The problem lies in the fact that the type of subscription some researchers have only allows them partial access to the WoK database, which can penalize senior researchers and be damaging when they submit applications for certain types of funding (for example, ERC grants). Finally, information is sometimes missing. For instance, there were periods of time where the database did not list names beyond the tenth author for multi-author articles.

## V. 2 The impact factor

The impact factor (IF) of scientific journals was the first index to gain wide publicity. It was originally intended for professionals in the publishing world. Its role was diverted from its intended purpose by researchers. It is defined as the average number of citations to articles published in a given journal over a given period of time. As a first approximation, the IF correlates well with the quality of the journal, except that the period of time over which the IF is computed is likely to be too short (two to five years in the ISI's Journal Citation Reports database); articles which give rise to citations over many years and often have the most impact on scientific progress are not fully taken into account by such a measurement. It should also be mentioned that impact factors are subject to manipulation by major journals.

Publishing houses are very interested in seeing their impact factor increase, which will improve the prestige of the journal and consequently the number of subscribers and single article requests. Publishing houses have developed strategies to increase their IF, among which are decreasing the number of articles accepted and favouring certain generalist journals or fashionable fields which will give rise to a higher number of citations.
A weak to low impact factor disproportionally harms certain reputable journals that contain a good number of excellent articles by a simple dilution effect due to the publication of serious but marginal and little cited articles. As an example, the *Journal of Immunology* published by the American Society of Immunology contains a high percentage of excellent articles that would have been appropriate for publication in the most prestigious journals but were excluded for various reasons. Its IF has nonetheless decreased significantly in recent years by dilution of important articles with low cited articles. It should be remembered that by definition the IF is an average that does not include the distribution of individual values around the mean value.

It is difficult to interfere with the practices of publishing houses; in most cases they are privately owned. The success of some journals managed by learned societies or non-profit businesses can however be noted with satisfaction.

It is unfortunate that the IF has become a measure of the quality of journals, to such a point that many researchers consider the IF of the journals in which they publish as a measure of quality of their own work. Nearly all the members of the present workgroup and the foreign experts consulted agreed on the fact that the IF of the journal in which an article has been published should not be considered for evaluating the scientific production of a researcher, except maybe in the case of young researchers since the number of their citations cannot yet be used. Even in this case, the IF does not deserve the prominent place it is often given when recruiting young researchers. In France, a researcher is often recruited for life on the basis of one or two publications in a high impact journal. This is true for many disciplines. Researchers who are recruited on such a basis and whose contribution to these articles is often unclear do not fulfill the expectations placed on them.

It should be emphasized again how important it is to take into consideration the quality of the work as well as the candidate's ability to present and discuss it during an interview.

## V. 3 Total number of citations

The second index used in bibliometrics is the total number of citations of a given author. Such an index is interesting but biased for two main reasons: the position of the author's name within the author list is not taken into account (see the above section on authorship) and the fact that certain articles can have a very high citation index for reasons sometimes unrelated to their importance (for example, a technical description, a reagent, a GMO or a resource book).

## V. 4 New indices, in particular *h*, *g*, and others

Recently, new bibliometric indices for the evaluation of researchers have been created to address the objections mentioned above concerning the IF and the total number of
publications. These new indices are by-products of the databases that reference articles and their citations. These new bibliometric indices were established without an underlying theoretical model and their common use is based on hypotheses that need to be strengthened by systematic studies. So far, on the one hand there are bibliometrics developers who are constantly refining the properties of their indices and on the other hand users who fight each other about their validity through examples and counter-examples without real validation.

The most popular is the *h*-index (h stands for Hirsch, the father of this index). The *h*-index is calculated by classifying an author’s publications by decreasing number of citations. The rank of the publication cited a number of times equal to the nominal value of the rank is the *h*-index. For instance, a researcher with an *h*-index of 47 has published 47 articles each of which was cited at least 47 times. The mean value of the *h*-index depends tightly on the discipline, a point that will be discussed below. The *h*-index is interesting but suffers from a number of weaknesses. It gives an advantage to senior researchers who have had a long research career (the *h*-index increases regularly with age) and normalized variants – for examples that take into account the number of years the researcher has been active – are artificial and of little use. Interestingly, it has been demonstrated that the normalized *h*-index adjusted for the number of years the researcher has been active reaches a peak after which it decreases with time for older researchers (P. Jensen *et al.*, *Scientometrics*, Vol. 78, No. 3 (2009) 467–479). As with the total number of citations, the *h*-index includes publications to which a given author has contributed very little or not at all. Among the *h* most cited articles, the *h*-index cannot distinguish between an article that received just over *h* citations and one that received many more. This index does not give an advantage to articles with a very high impact, in particular articles that have a lasting impact over time (i.e. those that continue to be widely cited).

The *g*-index was introduced by L. Egghe to compensate certain deficiencies of the *h*-index and to acknowledge excellent productivity. A researcher’s *g*-index will have a value of 83 if the researcher’s 83 most cited articles have received at least 6889 citations, that is *g*-squared citations. This factor has the advantage of giving value to highly cited articles with a long lifetime. Such articles contribute to increasing the value of the *g*-index over time while they do not affect the *h*-index. All kinds of *g*-index variants can be contemplated in order to better reflect the distribution of the number of citations of the most cited articles. The *g*-index is less well known today and less used than the *h*-index, maybe because its significance is less evident at first and its access is less widely spread. However, a simple computing tool (http://pasquier.claude.free.fr/publications/publisdata.php) can calculate it based on the ISI database bibliometric files.

Another approach consists in adjusting the citations to take into account the impact factor of the journal into which a given researcher's articles are published or the notoriety of the authors who cite a given work (however, which author should be chosen if there are many remains a question). The practice of relying on the impact factor involves many biases, in
particular it promotes a positive feedback loop that leverages the IF effect, doubling
therefore its effect.

V. **5 Conclusion on the choice of indices**

No single index is entirely satisfactory. It appears essential in practice never to use any one in isolation. One of the major difficulties is to keep indices simple. If they are too complex, they can potentially seem esoteric and become sources of conceptual errors that will be difficult to identify due to their complexity. More generally, some thought and even proper research on the continuing improved use of bibliometric indices should be stressed. The report will come back to this point in section VIII. In any case, all indices that will be developed in the future should be recognized and adopted internationally, at least by the European scientific community.

VI. How to use bibliometrics to evaluate individual researchers

VI. **1 What indices should be used?**

No single index is satisfactory when considered in isolation. It is best to use a set of indices,
for example the h- or g-index, and the total number of citations or alternatively a series of
numbers (number of publications which have been cited more than 10, 30, 100, 300, 1000
times). It is important to associate bibliometrics with the 5, 10 or 20 best publications
chosen by the researcher (the exact number should depend on the discipline and seniority)
and take into account their respective number of citations and the impact factors of the
journals, but keeping in mind the reservations mentioned above. By contrast, considering
journal impact factors in isolation is dangerous and should be avoided. Concerning the total
number of publications, it may be interesting to know whether a given author belongs to the
1%, 5% or 10% most cited authors in a particular discipline or whether an article belongs to
the 0,01%, 0,1%, etc. most cited articles in the discipline in the last 10, 15 etc. years (except
for SHS where databases are not reliable). These numbers could also be illustrated
graphically to have a more complete overview of a career.

**VI. 2 How to calculate and validate indices?**

Indices should not be computed by non-specialists (especially administrative staff) who
might use easily accessible data in a perfunctory manner. Due to the large number of
possible material errors, indices should be validated. A non-validated index has no value,
and this should be kept strictly in mind when dealing with such factors. First of all, it is
important that the list of data (publications) for each author be confirmed by the researcher
concerned, as is done in university hospital centres. In fact, researchers should be asked to
calculate their respective indices in so far as instructions for calculation are accessible. Their
calculations should be validated because researchers can make errors to their advantage or disadvantage. The problem is, who should conduct these validations? Mirroring the procedure adopted by certain universities and bodies such as the ERC, validation should be carried out by staff certified to undertake such tasks. Due to the experience that a dedicated staff will acquire, the workload and costs associated to the procedure will be relatively low and only concern disciplines suited to bibliometrics.

## VI. 3 Errors to avoid when using bibliometrics for individual researcher evaluation

Two types of errors seriously impair the use of bibliometrics:

### - Conceptual errors

The following errors should be mentioned here: use of bibliometrics in isolation; omitting to consider indices in the context of the discipline; omitting to take into account the position of an author's name in multiple author articles when author order is not alphabetical; using only one index; failing to follow good practices in order to avoid material errors (see paragraph below); using the journal's impact factor in evaluating the quality of an article; using averages when it is known that averages can present huge disparities between disciplines and penalise scientists who publish pedagogic articles which receive few citations and artificially bring down averages.

Renewed consideration should be given to the limitations associated with indices that are not normalised for the discipline (for instance, h-index) and which therefore can only be used to compare researchers from the same scientific community, a limitation that applies to their whole career.

The interest in an index should be judged based on the objective one has: for example, there is a contradiction when thematic mobility is promoted based on indices that serve it inadequately.

### - Material errors

Computing bibliometric indices is not a problem in so far as the person undertaking the calculation has all the tools necessary and has been specifically trained. It should be stressed that these calculations need to be checked and that such verifications require time and experience. It is not enough simply to directly consult a site which mentions the name of a researcher.

It is important to stress that many bibliometric analyses are inaccurate because data has not been properly collected (there can be a three-fold variation between a bad quality database and a good one), errors in computing the indices or incorrect interpretation. Many material errors can be made when applying bibliometrics. Several are well known to everyone but not always taken into account.

The main errors to be avoided are the following:
- Homonyms: this is an acute issue for popular surnames. Often the first initials are not sufficient to identify a given researcher. Associating the name of the city where the researcher works can help; however, researchers often change geographical location.

- Name change for women after marriage or variations in the use of initials.

- Use of incomplete databases that do not cover all the journals of a discipline, do not go back far enough in time or that have a user limited access policy depending on the subscription contract.

Using an identifier associated with each researcher was suggested by the ISI-Thomson company and in Brazil by the Lattes database initiative (http://lattes.cnpq.br/english/index.htm, one of the most meticulous databases currently available). It should be extended to other databases so as to avoid a great number of errors and provide considerable improvement in the reliability of the information. Time is needed however to generalise the practice and make it available for use.

## VI. 4 Who should use the indices ?

Indices should be accurate and used properly under the conditions defined above. The latter are indeed hard to satisfy. This is why simply declaring that indices should be used by peers is not enough, the peers should be well aware of or have experience with these difficulties. In disciplines that use bibliometrics, indices should only be used by peer panels who will only look at them within the context of an overall and essentially qualitative evaluation. In that case, bibliometrics can be a useful tool. Peers who use them must be able to justify their conclusions and this requirement will help them develop a good expertise.

In practice these indices are used in other contexts, sometimes in a hidden way, for example by university presidents and institute directors for recruitments or promotions. In France, the latter's decision is usually taken after consulting a recruitment committee that includes scientists from one or more fields. If the presidents and directors do not take into account the opinions of the recruitment committee, there is a great danger because indices of equal value may have very different meanings depending on the discipline and even the sub-discipline. Furthermore, material errors are frequent (because validation is rare) and bibliometrics is currently not associated to a qualitative evaluation.

Indices should be useful in the case of interdisciplinary panels that are requested to judge applications from candidates of widely different disciplines only specialists are able to evaluate. Bibliometrics can in such cases be used to make an initial selection among the candidates, provided that it is used by experts and that the variability in index distribution that exists between disciplines is taken into account. Although it is much less useful in the case of recruitment committees covering a single discipline where members usually know the candidates well, it can still be useful to make a first selection when there are many applicants.
Finally, these indices can be interesting to the researcher in that they will encourage the researcher to publish, they provide a means of knowing where he stands in his own discipline and whether his work is recognised and by whom. More generally, it should be recalled that the main function of an evaluation is not to penalise a researcher but to encourage him, if necessary, to improve the way he works.

## VI. 5 How should indices be used?

Due to the numerous potential biases, indices should never be used in the case of researchers who have been active for less than 10 years (including the doctoral thesis). The evaluation should only be done by peers on the basis of an interview of the candidate and a close reading of the candidate's publications.

It would be advisable for indices to be clearly mentioned in the curriculum vitae of senior researchers and academics before evaluation by their peers for key promotions and grade changes in disciplines where such indices can be computed.

Furthermore, criteria vary depending on the goal of the evaluation (recruitment, promotion, grants, prizes, fundamental or applied research), the discipline, the length of the career and career path of the candidate. Also, attitudes regarding publication have changed throughout the years and these generational differences should be taken into account.

Generally, any bibliometric data should be understood relative to the distribution of index values for a particular field and even for a specific homogeneous area of activity of the researcher.

The use of bibliometrics for disciplines where there are few citations (mathematics and many social and human sciences) should be avoided and only used with the greatest caution in the case of interdisciplinary researchers.

Such bibliometric data should also be indicated on the evaluators' curriculum vitae.

## VI. 6 Should bibliometric indices be systematically mentioned on applications?

It should be noted that practices vary among the different bodies. Some such as the ERC request that bibliometric data be indicated on the applications. When this is not the case, reviewers frequently try to compute the indices and their calculations often contain some of the potential errors mentioned above.

It would be preferable to ask researchers to provide their own bibliometric data (number of publications, h-index or any other factor such as g-index, a fixed number of their most cited publications and the impact factor of the journals they were published in) and include them in their applications.

This of course does not exclude that candidates provide a list of the 5, 10 or 20 publications they think are their best irrespective of the number of citations.

The number of publications to be submitted to an evaluation committee depends on the aim of the evaluation and the age of the researcher. The electronic files (.pdf) of the publications
should be included. This requirement can be easily met by the candidates and would considerably alleviate the work of the members of the committee.

## VI. 7 Addition of bibliographic notes to supplement numbers

The bibliographic notes accompanying a publication which serve as a basis for calculating bibliometric indices hold important information about the publication and its authors: the name of the coauthors, the citation trend over time, who has cited the article and what are the other fields this article has had an impact on? Whenever possible, bibliometrics should be supplemented by the examination of the bibliographic files associated with the articles chosen by the candidate.

## VI. 8 Importance of considering citations to an article relative to the citation distribution for the journal

Based on the data of the JCR database (ISI), it is possible to evaluate the level of citations of an article in a given discipline compared to the average level of citations for articles in the same discipline published in the same journal. This information can be very useful and would not penalise, but favour, authors who have published highly cited papers in journals with a modest impact factor. An author could then be judged on the content rather than the reputation of the journal, and that could even lead to a positive discrimination of some sorts. However, defining discipline and sub-discipline boundaries is a complex question that has not yet been solved.

## VII. Importance of a national debate on the improvement of indices

The Académie suggests that a national debate be held on the bibliometric evaluation of full-time and academic researchers and to envisage different studies to improve the use of bibliometrics to be led by a small representative group of experts in close partnership with bibliometrics users, in particular the Observatoire des Sciences et Techniques (OST).

### VII. 1 Retrospective tests

should be undertaken to compare the decisions actually taken by peer panels (CNRS, INSERM, IUF, ERC, etc.) against the results of a purely bibliometric-based evaluation of the candidates and the career evolution of these candidates.

- Retrospective analysis of a population of researchers that were promoted but would not have been on the sole basis of their bibliometric indices, and vice versa. The analysis would be carried out using CNRS data and would result in the creation of a database for the years 2004-2010. It should be complemented by a survey of the members of the national committee who took part in the deliberations and of the successful and unsuccessful candidates.

- Similar studies should be undertaken of other evaluation panels (academic research (IUF), European Research Council, etc.) to compare the decisions made by these panels and
bibliometric indices. There are two real problems associated with this: identifying
reputable panels and obtaining the lists of unsuccessful applications. As in the case of the
CNRS study, a survey of the members of the panels should complement this analysis.

- A study of index distribution for the recipients of the most prestigious awards should be undertaken. A workgroup should carry out a large-scale study of Nobel Prize, Fields Medal, CNRS Gold and Silver Medal winners, members of the French Academy of Sciences and of major foreign academies and, even, a study of the history of the recent important scientific breakthroughs, all from a bibliometrics point of view.

- Long-term monitoring of researcher indices should be carried out to establish a baseline so as to detect the “shooting stars”, the case of researchers who have changed direction during their career should be examined and the predictive value of the indices used should be evaluated.

- Discrepancies between bibliometric and qualitative evaluation by peers should be analysed and the elements that led to such disparities quantified: local interests, discipline specificities, network effects, friendships, influences of all kinds, consideration of factors other than bibliometric, index limitations (frequently cited technical publications, team work, etc.).

It would be useful to check whether such studies have already been undertaken in foreign universities and, if so, contact them (for example, the Lund University in Sweden).

**VII. 2 Development of standards that discern originality, innovation, diffusion and creation of schools of thought, to be used as “à la carte” indices.** In this respect, as suggested above, it would be interesting to study the history of the recent major discoveries in the context of bibliometrics (Fields medals, Nobel prizes, Gold and Silver medals of the CNRS, etc.).

**VII. 3 Studies to refine existing indices** and define relevant bibliometric indices to use in the context of individual evaluations, where the usage of bibliometrics has appeared only relatively recently. There should be an in-depth examination of the notion of authorship.

**VII. 4 Development of new indices.** Due to the shortcomings associated with the indices discussed in this report, the development of new ones should be envisaged. The issue is not easy, because adding new indices will just make bibliometric evaluation more complex and less transparent. It is advisable that the development and publication of new indices not be a commercial venture as is currently the case with the ISI (Thomson-Reuters) and SCOPUS (Elsevier) databases. The astronomy-astrophysics and physics fields have proved that it is possible to have non-profit databases such as the ADS database operated by the *Smithsonian Institute* (see section V.I). It would be interesting to extend a similar initiative to other wider disciplines such as chemistry or biology; however, this may be a gigantic endeavour. The Académie cannot do this. Such an initiative can only be done at the European level.
**VII. 5 Establishment of rules of good practice for the use of bibliometrics during researcher evaluation** in response to a request by the national agency for higher education and research evaluation (AERES), one of the missions of which is the validation of evaluation procedures for researchers.

**VIII. Conclusion**

Due to the continuous development and constant evolution of databases, bibliometrics is playing an increasing role as a tool to help in the evaluation of individual researchers. This is explained by the apparent ease and rapidity with which indices can be consulted in contrast to the complexity of a qualitative evaluation by peers, whose burden is exacerbated by the excessive number of evaluations that are requested of them. Furthermore, bibliometrics provides quantitative elements regarding a researcher's publications and citations while a qualitative evaluation involves a higher level of subjectivity. Taking into account indices based on citations and examining the bibliographic notes associated with a limited number of publications chosen by the candidate will help and facilitate the work of the evaluation panel.

By contrast, bibliometrics has many disadvantages that have led some disciplines to limit its use or even not use it at all as in mathematics and social and human sciences. First of all, contrary to a widespread notion, bibliometrics does not measure a researcher's scientific production or its impact, it only gives a numerical assessment of the citations to each of his/her articles. If only one index or even a set of indices is used, bibliometrics can lead to serious errors in judgement. For instance, the bibliometric indices of certain great scientists who received the most prestigious awards are very low. Finally, bibliometric indices often influence researchers' behaviour, some may choose to steer their publication and citation activities in such a way as to improve their bibliometric indices rather than engaging in original and creative research. By doing so, researchers modify the correlation between scientific quality and citations which is the very basis of bibliometric indices.

While recognising the need to use bibliometrics to make a first selection among candidates in some disciplines and in situations where a great number of researchers are to be evaluated, it is important to be aware of its limitations. Its use should be strictly restricted to peers, who are the only persons who are able to consider bibliometrics in the context of a qualitative evaluation. In particular, peers should justify their conclusions when these differ from those obtained solely on the basis of bibliometric criteria. All values calculated should be strictly considered relative to the distribution of values in the relevant discipline. Finally, care should be taken to ensure that the values used are correct, for instance by having them validated by the researcher concerned.

After a decade of use, bibliometrics should take its rightful place in researcher evaluation and its use should be as relevant and transparent as possible while limiting the abuses it might lead
to, in particular when used in isolation outside the context of a qualitative evaluation by peers and without any consideration for the particular discipline. Such evolution requires a thorough debate at the national and international levels. Major efforts are needed to better assess the contribution bibliometrics can make to researcher evaluation, keeping in mind the global aim of improving evaluation overall. The evaluation procedure should be both qualitative and quantitative (keeping in mind that other quantitative criteria exist that are not taken into account by bibliometrics such as invited conferences, major grants and awards) while eliminating as much as possible all direct and indirect conflicts of interests.

A steering committee should be created within the framework of the Observatoire des Sciences et Techniques (OST). Its task will be to advance the present analysis along the major directions of study that have been identified. This issue is of major consequence for the individual evaluation of researchers and will also influence the evaluation of laboratories and institutions, in particular for major international rankings.
# ANNEX 1
## COMPOSITION OF THE WORKING GROUP AND EXPERTS CONSULTED

### Members of the Academy

**Jean-François BACH (coordinator)**
Secrétaire Perpétuel de l'Académie des sciences
Emeritus Professor, Université René Descartes

**Denis JÉROME (coordinator)**
Emeritus Director of Research, CNRS - Université Paris-Sud

**Jean-Michel BONY**
Professor, École Polytechnique

**Pierre BRAUNSTEIN**
Director of Research, CNRS - Université de Strasbourg

**Catherine CESARSKY**
High Commissioner, Atomic Energy Commission

**Jean DALIBARD**
Director of Research, CNRS – Professor, École Polytechnique

**Christian DUMAS**
Professor, Ecole Normale Supérieure de Lyon

**Jacques FRIEDEL**
Emeritus Professor, Université Paris-Sud

**Etienne GHYS**
Director of Research, CNRS - Ecole Normale Supérieure de Lyon

**Michel LE MOAL**
Emeritus Professor, Université Victor Ségalen de Bordeaux

**Bernard MEUNIER**
Chief Executive Officer, PALUMED

**Olivier PIRONNEAU**
Professor, Université Pierre-et-Marie Curie

**André SENTENAC**
Scientific Adviser, Life Sciences Division, Atomic Energy Commission
Alain-Jacques VALLERON
Professor, Université Pierre-et-Marie Curie

## External members

Ghislaine FILLIATREAU
Director, Observatoire des Sciences et Techniques

Pierre GLORIEUX
Director, Research Units Sections, National evaluation agency for research and higher education (AERES)

Pablo JENSEN
Director of Research, CNRS - Ecole Normale Supérieure de Lyon, Representing the CNRS

Axel KAHN
President, Université René Descartes, Representing the Conference of University Presidents (CPU)

Claire LEMERCIER
Researcher, CNRS - Centre de Sociologie des Organisations

Laurent LINNEMER
Professor of Economy, Ecole Polytechnique and Center for Research in Economy and Statistics (CREST)

Florence WEBER
Director, Département de Sciences Sociales - Ecole Normale Supérieure

## Consultants

Dr. Jonathan ADAMS
Director of Research Evaluation, Evidence, Thomson Reuters - Leeds (Grande-Bretagne)

Pr. Sir Richard FRIEND
Professor - Cavendish Laboratory - Cambridge (UK)

## Coordinating Editor

Brigitte d'ARTEMARE
Chief of Staff - Académie des sciences
ANNEX 2

REPORT PUBLISHED IN 2009 BY THE ACADÉMIE
EVALUATION OF INDIVIDUAL RESEARCHERS IN THEORETICAL AND
EXPERIMENTAL SCIENCES

(http://www.academie-sciences.fr/actualites/textes/recherche_08_07_09.pdf)

**Summary and Recommendations**

Evaluation of research units and researchers has been practised for a long time already and is
now considered a normal process by the scientific community. Its expanding use has become
a topical issue due to the recent French law, the Universities' Freedom and Responsibilities
(LRU) law that transfers new powers to the universities at the local level. The French
university context is complex, work conditions for academic researchers are far from
homogenous, yet they have to be evaluated nationally despite differing infrastructures,
equipment availability, student educational background and course options. With regard to
this specific context, the Académie des Sciences has put forward some recommendations
based on three important principles: competence, transparency and ethics.

# 1. The code of ethics

- Evaluators' mandate should be short (3 years) with a renewal on a yearly basis of one third of all committee members.

- Committees should include one expert from outside the field and a high proportion of examiners from other French or foreign institutions (the LRU law specifies 50% in the case of recruitments).

- The procedure and criteria used in an individual evaluation, as well as their adaptations to specific fields or sub-fields, should be posted at the national, institutional and university levels.

- A special effort should be made to identify conflicts and common interests that are not immediately clear, and any ethical issue should be brought to light in advance.

- Each member of an evaluation committee has an obligation of confidentiality; the president is the only person authorized to give more detailed information in case of dispute.

- The full report should be communicated to the examinees without any modifications, confidentiality as to the report's authors being preserved by the evaluation committee and its president.
- A commitment to the code of ethics should be signed by each evaluator. Any failure to comply to the code of ethics should be considered serious professional misconduct.

## 2. Criteria and tools for evaluating research activities

Any evaluation of research quality and productivity should integrate several levels of analysis.

### Qualitative evaluation

Qualitative analysis is the most important facet of an in-depth evaluation. It should be based on an analysis of the scientific work and if necessary augmented by a timely interview. Bibliometric data and other quantitative criteria cannot be a substitute for an evaluation by peers, however once the data has been fully examined and understood, it can help decision-making.

### Quantitative evaluation

Bibliometric indicators may be quite useful if used properly, readjusted to the context of the field and integrated into a qualitative evaluation.

- Bibliometric indices should not be used alone to establish a ranking.

- Greater importance should be given to article citations than to the impact factor of the journal in which the work is published (except in the case of young researchers). The *h* and *g* indices based on citations are useful but of limited interest and should be complemented with new indicators.

- The number of authors in a citation should be taken into account as well as the place of the author's name in fields where the order is not alphabetic.

- The Académie des Sciences suggests organising an inter-organism and interdisciplinary action, together with the science and techniques observatory (OST) and the national agency for higher education and research evaluation (AERES), to reflect on the use of bibliometric tools and the creation of new indicators. Tools currently used should be validated with retrospective tests.

### Other criteria of recognition

The scientific quality of a researcher can be evaluated based on many other criteria than those relying on bibliometry, in particular managerial, supervision and leadership skills, the writing of academic books and books for the greater public, the number of languages they are translated into, invited talks to conference plenary sessions, leading participation in international programmes, presidency of an international scientific association, chief-editor positions in international journals, award of significant contracts, awards of prizes and
national or international distinctions, membership of French and foreign academies, other
distinctions such as nomination to the Institut universitaire de France, organization of summer
schools, symposia, high level international meetings, etc.

Evaluation of research applications

In contrast to what is observed in other countries, and although much progress has been made
in this respect, industrial projects and applications are not sufficiently taken into account
when evaluating researchers in France.

- Industrial application should become an essential evaluation criterion for those involved in applied research and it should become a factor leading to promotion similar to publications.

- An evaluation scale should be established giving a significant place to the relevance of the research.

- Criteria for evaluating research outcomes that do not directly lead to immediate applications, such as software and prototypes, but that are nonetheless important should be defined.

In the end, it is peer committees that review the evaluation criteria mentioned above, and they
should do so mainly based on a personal analysis of publications and interviews. Such an
evaluation should include quantitative indicators but also take into account the novelty of the
research and its relevance.

**3. Criteria for evaluating teaching activities**

The LRU law and the recent decree of 23 April 2009 that defines the regulatory measures applicable to academic researchers establish the obligation of evaluating three types of activities: research, teaching and common interest activities. This is made necessary by the fact that the relative importance of these three types of activities may vary during a career. Concerning the evaluation of teaching in all its forms, the Académie des Sciences recommends the following:

- Evaluation of teaching activities may be carried out following several approaches that lead to the production of an evaluation scale at the local and national levels, the local evaluation being the most relevant. An important criterion is student rating of courses, a delicate point that may lead to perverse effects.

- The evaluation of teaching activities should also include objective criteria such as content and novelty of courses (publication of teaching material, manuals, posting of courses and lab material on-line, exhibits, etc.)

- An annual record of teaching obligations should be published each year by each institution,
and teaching exemptions should be clearly mentioned and justified.

- The institution should publish each year the percentage of students that successfully finished their study requirements (L1, L2, L3, M1, M2, doctorate) and what they moved on to do, as well as their possible employment prospects per level of studies and at the end of the thesis. These elements should be taken into account as much as possible to evaluate academic researchers.

- The best maîtres de conférences (lecturers) who devote most of their time to teaching (initial training and continuous training) and are unanimously recognized for their pedagogic qualities, should benefit from local promotions such as *Hors Classe* (Exceptional Teacher) or receive bonuses from their institution. Inversely, those who neglect their teaching duties should bear the consequences.

## 4. Evaluation of common interest activities

- Administrative and common interest activities should be taken into account when evaluating academic and full-time researchers under the new regulation in force, in particular regarding activities that require responsibility (coordinating the first academic year, department leader, international cooperation missions, advising students on courses and jobs, cooperation with industry, patents, promotion of scientific and technical knowledge, etc.).

- Institutions should publish a record of non-teaching responsibilities fulfilled by academic researchers.

- Currently, there are no objective criteria to evaluate these activities. A specific scale should be established to evaluate common interest activities.

## 5. Evaluation frequency and format

The current frequency of evaluation is too high. The Académie puts forward the following recommendations:

- In-depth evaluations and routine performance assessments should be distinguished.

- The number and frequency of in-depth valuations should be limited to the important steps in a researcher's or teacher's career, i.e. recruitment and important promotions and transfers.

- Recruitment is a key step because the staff recruited will become a *fonctionnaire d'Etat* (civil servant).

*A nation-wide two-step process should be adopted, with a first cut-off on dossier and a second cut-off after an oral presentation followed by an in-depth interview. Because scientific creativity and novelty are hard to judge based solely on bibliometric data or prepared presentations, the in-depth examination by peers should take on this
evaluation role. The ability to teach should also be tested based on pertinent seminars.

- The lists of national qualifications created to make up for the heterogeneity of thesis levels and habilitations to direct research (HDR) are not fully satisfactory. With the recently instituted autonomy of universities, a significant redefinition of the criteria required for obtaining these qualifications should be undertaken. The essential role bestowed onto graduate schools and university scientific committees should also be redefined. These recommendations must rapidly lead to diplomas recognized for their quality.

*Due to compulsory preliminary registration on aptitude lists, there are four evaluation steps involved in becoming to become a professor in France, as compared to two in other countries similar to ours. A general reflection on this topic should be carried out.

- Performance evaluations should be limited and done as part of the standard four-year university activity contract, on the basis of the simplified form used to monitor the normal activity of the staff.

## 6. Evaluators

- The scientific competence of evaluators is fundamental. The Académie des Sciences proposes that a list of prerequisites be prepared and published by the AERES for each category of evaluators.

- Important measures should be taken to ensure the good will of the best evaluators, by making their task easier, reducing the duration of the mandates and making sure this activity is taken into account when appraising administrative or common interest responsibilities.

- All evaluators should be evaluated to guarantee their competence.

- Although evaluation committees specific to each university should be managed locally, they should include a significant number of external examiners (clause specified in the LRU law in case of a recruitment panel).

- The respective roles of national evaluations (essential for research activity) and local evaluations (more appropriate for evaluating teaching and common interest activities) should be distinguished.

## 7. Follow-up on evaluations

- One of the major difficulties of the evaluation system is that there is frequently absence of an impact. Hence, care should be taken to only perform evaluations when these can lead to a promotion or a career reorientation.

- A distinction should be made between assessing the quality of an activity and the progress of a career and avoid mixing up evaluation and reorientation.
- Careers should be monitored by *ad hoc* committees that take on the role of career counsellors. This system for human resources management should be adopted by each university.

- Universities should post their scientific and educational specificities such that calls for candidates are unbiased.

---
# ANNEX 3
## BIBLIOMETRIC PRACTICES BY DISCIPLINE

As stated many times in this report, customs regarding bibliometric evaluation vary according to the discipline and even sub-discipline. This annex presents an overview of bibliometric practices in the major scientific disciplines with more detail than in section III. The following description is based on the contributions provided by the representatives of these disciplines within our workgroup.

### Bibliometrics in Mathematics

Mathematicians are very reluctant to use bibliometric tools for evaluating researchers. This position is not specific to French mathematicians, it is common to mathematicians worldwide.

The comprehensive report of the International Mathematical Union (http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf) states among its conclusions that "While numbers appear to be "objective", their objectivity can be illusory. The meaning of a citation can be even more subjective than peer review. The sole reliance on citation data provides at best an incomplete and often shallow understanding of research — an understanding that is valid only when reinforced by other judgments. Numbers are not inherently superior to sound judgments".

The reason behind the mathematicians’ refusal to use bibliometrics is not because they shy from “modern” methods, but because they have tools at their disposal which are far more efficient than those bibliometrics can provide, and they use these systematically in their evaluations. The reasons stem from the fact that the community of mathematicians is relatively small (~40000 worldwide of which ~4000 in France) and that mathematicians have organised themselves at the international level long ago (approximately two thirds of a century ago).

There are two mathematical databases, Zentralblatt Math (of the *European Mathematical Society*) and MathSciNet (*Mathematical Reviews on the web* of the *American Mathematical Society*). The second is the most widely used database and it contains references to all mathematical articles published worldwide since 1940. For each of them, it provides a one-
half to three page critical analysis of the results of the most significant articles, prepared by a mathematician and not by an author. Although its primary aim is to be a tool to help research, this database is systematically used by whoever needs to evaluate mathematicians (recruitment, promotions, awards, etc.). It gives for each of them the list of publications and a critical analysis, their citations (by whom and in what articles). The problem of homonyms is thus solved. The whole of the mathematical community has a long tradition of working together to create outstanding databases that are not limited to just tables with numbers.

Other disciplines should maybe draw inspiration from this success and be encouraged to create similar databases.

Bibliometric data can of course be extracted from these databases. The relevance of such data can be judged from the example of the two recent French Fields Medals awardees: Cédric Villani was cited 1520 times by 629 authors while Ngô Bao Châu was cited 102 times by 52 authors yet no mathematician would see a disparity in the levels of the two laureates.

In conclusion, the relatively small size of the mathematics community, the underlying harmony in this field and the existence of outstanding databases explain why mathematicians prefer a qualitative evaluation by peers essentially based on a reading of articles. In mathematics, bibliometrics can only make a very marginal contribution to the individual evaluation of researchers.

## Bibliometrics in Physics

Physics directly concerns five sections of the Comité National de la Recherche Scientifique (CoNRS) and due to multiple on-going collaborations also some Biology and Chemistry, especially Materials Sciences, sections.

Professors are evaluated at the national level by the CNU (national council of universities) and research performance, number of publications, number of invited conferences, number of doctoral students and the *h*-index play a determining role. At a local level, involvement in the common interest activities of the university is taken into account during an evaluation. In this case, the *h*-index is considered less significant.

Nearly all candidates first list in their CVs and list of publications articles published in *Nature* and *Science*, then those published in *Physical Review Letters* and finally in *Physical Review*, often without providing the number of citations received by these articles. Candidates often omit to mention articles published in what they think are less prestigious journals such as those published by European scholarly associations because they fear a negative impact on the evaluation panel. The impact factor of a journal, which we critised so strongly in this report, plays too great a role in decisions concerning the evaluation of a researcher.
Generally, when indices are used, it is in the most simplistic way although the ISI databases cover physics journals well, even for articles published in French. Conference proceedings published by journals are starting to be taken into account. The ISI database is considered to cover 80 to 100% of the relevant publications in physics. Books are not yet well represented.

The trend towards a greater number of authors is starting to be problematic and there is a concern that the aim is to enhance everyone's citation count. The position of a name in the list of authors is not as significant as in biomedical disciplines although a trend in that direction is starting to emerge.

Knowledge dissemination in Physics is essentially through publications in scientific journals, with a clear preference for English and American journals. New unpublished results are rarely disclosed in conferences, except for preliminary results presented in posters by doctoral students. Physicists also frequently use servers like arXiv or Hal to deposit articles before or during the publication submission process and published articles that have received the approval of the publisher for deposition on these servers as long as the editorial layout of the journal is not used.

The use of the Hal (TEL) server should be encouraged and even become mandatory for on-line thesis deposition, an excellent initiative by the CNRS that allows considerably increased visibility of the full work of doctoral students.

Although publishing in a prestigious journal is in itself commendable, some thought should be given to a practice that leads to a certain article format and even promotes some topics (when the editor-in-chief of a journal wishes to favour certain fields for commercial reasons) and ultimately results in a loss of originality and creativity.

In conclusion, the research evaluation system, in Physics and related fields, should take greater account of the innovation, pertinence and visibility (citations) of the works rather than the simple prestige of the journal or review in which they are published. Bibliometrics with bibliographic files could contribute to address this situation.

## Bibliometrics in Mechanical Sciences, Computing Sciences and Applied Mathematics

In France, these disciplines are centralised and the number of researchers is sufficiently small (less than 5000 in France for each of the 3 fields) that good information about a researcher is available without having to use bibliometrics. The problem arises mainly for young scientists and for evaluating researchers' activity in real-time, for example over the 4-year period required by the system for a promotion.
Young researchers are “evaluated” mainly orally by giving a seminar. Assessing whether a researcher has been active over a 4-year period is more problematic as it is difficult to find other criteria than the publication list. Administrative responsibilities associated with research and its related activities (organising conferences, editorial responsibilities, etc.) are important factors in an evaluation. This information is usually available on the researcher’s personal webpage. It is important that researchers maintain an attractive website and update its content regarding all their scientific activities, including publications.

In Applied Mechanics and Applied Mathematics, transfer of new knowledge is partly done through publications and partly through conferences at international meetings.

In Computer Sciences, transfer is essentially through invited conferences, if possible at major international meetings with high recognition in the discipline (for example, SIGGRAPH for computer graphics). Such meetings usually do not publish their proceedings, but they archive them on their own Internet sites. Publication in journals has a role only in some areas of theoretical computing.

This scientific community is not particularly hostile to bibliometric indices and uses them as support information, certainly not as main criteria. Over a long career, these indices give reliable information on the reputation of a researcher if one wishes to know whether he/she is well-known or not, but a precise ranking is not possible based on the indices. Their use by persons unfamiliar with the researcher's field is considered dangerous and is disapproved by the community.

## Bibliometrics in Astrophysics

In Astrophysics, bibliometrics is generally used to evaluate researchers for hiring, promotions and grant awards. NASA keeps a free access bibliographic database (ADS) but does not claim it to be perfect or complete. This database provides citations to articles and many use it to count citations and calculate the *h*-index. For example, for promotions at higher levels of a scientific career at the European Southern Observatory (ESO), candidates must have been cited a certain number of times and have published a certain number of highly cited articles. Similarly, ERC evaluation panels consult the number of citations of the candidates and even sometimes their *h*-index (which is easily obtained using ADS when the candidates do not provide it). All these elements are useful to their discussions. It should be kept in mind however that a database such as ADS is not complete and this can heavily penalise the bibliometric performance of multi-disciplinary researchers.

It is well understood by everyone that these are only indices and that some adjustments are required:
- When the candidate is one among many authors to a highly cited paper, it is important to know what was the candidate's contribution to the publication, which is usually done by questioning the senior authors of the article.

- The number of citations must be examined within the context of the sub-discipline, for example cosmology articles receive many more citations than articles of equal importance in solar physics. All good evaluation panels know how to make this adjustment in a more or less qualitative way.

- Certain articles of average importance can reach a very high level of citations by claiming a value to a parameter that is necessary for other works and becomes then a reference value. In this case again, good panels are not misled.

- Certain excellent articles that solve a real problem are seldom cited because they “close” a topic. Inversely, incorrect articles can obtain numerous citations because they elicit a great number of rebuttals.

Once these adjustments are made, a good correlation is observed between the level of citations and the *h*-index on the one hand and on the other the “real” evaluation criteria including depth, originality and productivity. Overall, astrophysicists use bibliometric indices appropriately. However, in general, greater importance is given to the content of the five or ten most significant articles listed by the candidate.

## Bibliometrics in Geosciences

In the Geosciences as in Biology, articles generally have less than 10 authors and most often less than 5. The order reflects in general the (decreasing) importance of the contributions. The first author usually is the author who did most of the work, usually a doctoral student, sometimes a more experienced researcher as principal investigator or because he/she provided a crucial idea. Sometimes, in rare occasions, the last author is the head of the laboratory. Increasingly, the main research technicians who worked on the project are listed as co-authors. Bibliometrics is increasingly used by the CNRS commissions concerned and commissions for the recruitment of academic researchers, especially in cases of promotion (lecturer to professor, researcher to director of research and higher). Bibliometrics is infrequently used for starting researchers (less than 10 years including the thesis).

## Bibliometrics in Chemistry

In Chemistry, although bibliometrics is not used officially, the usual indices (*h*-index, total number of citations, number of citations per article) are taken into account quite seriously during preliminary discussions when evaluating the career or achievements of researchers who have been active for more than ten to twelve years. Due to the size of the community and the international dissemination of works, there are high quality evaluators who can use
bibliometric indices in a relevant manner. Practically, the Chemistry sections of the CNRS and the CNU avoid using bibliometric indices. It is advisable that indices be clearly mentioned on the CVs of senior researchers before evaluation by peer panels for important promotions (Research Director 2nd to 1st class and 1st class to exceptional class).

## Bibliometrics in Biology

Bibliometrics is widely used in Biology and Medicine. Most researchers strive to publish their articles in the small number of prestigious journals, such as the generalist journals *Science* and *Nature*, and to a lesser degree *PNAS* or in the best known specialised journals.

The success of a researcher is measured as much by the fact that the work has been accepted for publication in highly prestigious journals as by the originality of discoveries made. The problem is complicated by the fact that high quality work, especially work relying on state-of-the-art equipment, is accepted more readily than other types of studies by these major journals. In this context, it is easy to see that the impact factor of a journal is of great importance, greater in researchers' minds than bibliometric indices.

Another complication is the position of a researcher's name in the often long list of authors of an article. The young scientist or student who did the actual lab work is 1st or 2nd author. The thesis director, group leader or laboratory director are listed last. The middle authors generally held a secondary role even though they benefited from the publication on equal footing with the first and last listed authors. This excessive situation led the major journal publishers, in particular *Nature*, *Science* and *Cell*, to create specialised journals under their label, for example *Nature Immunology* or *Science Translational Medicine*.

We are reaching a non-nuanced situation where the only articles considered excellent are those published in high impact journals. This penalises many highly interesting articles that are refused by such journals on the grounds that they are not absolutely excellent or modern or because they are victims of the highly discriminating review procedure of these journals.

The situation is such that in some cases the importance of the journal influences the work of some researchers. They adapt their work to increase their chances of being published in these journals rather than engage in creative research that the referees of these prestigious journals do not always take into account.

One final point is that of sub-disciplines. The impact factor of a journal and the number of citations tightly depend on the size of the community associated with each discipline or sub-discipline. It is therefore very important to compare the bibliometric indices of an article to those of articles in the same discipline or sub-discipline. General journals, in theory, include all disciplines but they usually favour some fields and methodologies.
## Bibliometrics in Plant Biology

Historically until the mid-20th century, plant biology (as opposed to botanics), animal biology (as opposed to zoology) and medicine were one of the pillars of biology in its broadest sense. As a sub-discipline, plant biology is in itself quite heterogeneous and includes many specialties from cellular biology to genetics (and then genomics), developmental biology, pathology, physiology, biochemistry and ecology. The latter together with its animal counterpart has recently become a discipline in its own right. A distinction can be made between researchers using a descriptive approach relying heavily on correlations for their demonstrations (in ecology and population biology) and researchers with a mechanistic approach, based in particular on biochemistry and molecular genetics. They are evaluated separately by different sections of the CNU, CNRS and the National institute for agronomic research (INRA).

The size of the community in France is on the order of one thousand researchers. It is difficult to get a precise number because they are divided for the most part between the universities, INRA, the Muséum d'Histoire Naturelle and the CNRS and to a lesser degree at the CEA, IRD and other institutes. The best known scientists clearly belong to groups associated with a scientific and technical research public establishment (EPST) or a public industrial and commercial establishment (EPIC).

How are individual evaluations carried out? At the national level, the CNU, CNRS and INRA make a distinction between well-known scientists (generally senior scientists, research directors or professors and a few junior researchers and lecturers) and the ones who are not or not yet well-known (in particular junior scientists).

- In general, evaluation panels examine the publication list and the reputation of the journals where the scientists publish their results. Some candidate applications even provide the impact factor of the journals. Some journals are rightly or not considered prestigious (*Nature*, *Science*, *PNAS*, *Cell*, or *Plant Cell* the most specialised journal of the discipline). Publishing in these journals is a mark of established recognition. The work required to access such journals has usually been done over 2-4 years and involved several persons.

- Research in experimental sciences is a competitive, personal intellectual activity carried out as a group. Biology depends on numerous techniques and methods that require collaborations, a fact that makes individual evaluation difficult.

- A clear distinction should be made between truly innovative researchers who do not always follow current trends from those who are less so but are nonetheless technically outstanding (they are usually research support staff rather than researchers) who publish a lot, even in excellent journals. Often in France, recruitment favours this latter profile to complement the skills of a research group.
- The position of a researchers’ name in the list of authors is an important element for biology overall. The first author is usually a doctoral student or post-doc and has carried out the bulk of the work. The senior author is the researcher who directed the work and contributed the basic idea, usually preliminary results the validity of which need to be tested. The other authors often made less significant contributions, in particular in the many cases of occasional technical collaborations (for example, use of technical services).

- The impact of the researcher's work is an element that is often taken into account in an evaluation: capacity to contribute to the progression of the discipline, to create a school of thought, to attract foreign researchers on sabbatical leave, etc.

In conclusion, to date, bibliometric indices are not yet used automatically by evaluation panels. As a recommendation, the precise contribution of each author should be made clear. All the authors should be able to explain the full content of an article they have co-signed and to explain their contribution (conceptual, methodological, technical, provider of biological samples, etc.) to the work.

## Bibliometrics in the Medical Sciences

Evaluation in the Medical Sciences is highly affected by the fact that since the public health insurance sector reform of 2004, scientific publications are explicitly taken into account for funding hospitals. Hospital funding depends on the number of procedures they provide which are awarded a value based on a codified process. Involvement in certain general interest activities, in particular research – mainly undertaken in teaching hospitals – is not taken into account in this process and is instead rewarded specifically under a special line-item budget (called the “MIGAC” envelope). Research activity is recognised through the systematic compilation of the publications from the hospitals, classified into 3 classes according to their quality. The research activity of an individual or medical service is given a value obtained by multiplying the index of the journal (8 points for journals in class A, 4 points for journals in class B, 1 point for journals in class C) with an index based on the ranking of the researcher within the list of authors (4 points for 1st author, 2 points for 2nd or last author, 1 point for all the others). The score varies from 1 to 32 and funding, which is awarded globally to the hospital, is calculated by multiplying the number of points by the value given to a point.

This system undoubtedly has an influence on the way academic bodies evaluate researchers because they have at their disposal a simple and up-to-date tool (the SIGAPS software). This software is a welcome development since it delineates, more clearly than the CNU evaluation committees, the objective contribution of the candidates to medical research activity. Automation should not become the rule. The examination of individual applications is required to identify the publications where the researcher being evaluated took the initiative of the work or had a prominent role and to distinguish them from the articles where the
candidate is only one among many authors whose only contribution was to allow the use of a
technical facility or provided patients for the analysis. From this point of view, referees can
follow the Vancouver criteria to judge the true contribution of an author. It should be noted
that the SIGAPS software makes no distinction between a letter to a journal, a review article
or an original article. Some medical committees are well aware of all these problems and
publish guidelines for the candidates. Such a step should be encouraged.

For instance (section 4604 of the CNU):

The candidate must show that he/she has proven integration and leadership skills and that he/she is capable of participating in a research group located within the university to which he/she is being nominated; such skills will be judged based on past and on-going publication activities and on the projects defined by agreement with the clinical research directorship of the teaching hospital (contracts, PHRC, STIC).

The minimum number of publications required is 5 original articles as a first, second or last author in international journals with a high impact factor in the discipline, ranked A or B by SIGAPS, or of equivalent ranking. The list of publications will be used to examine the integration of the candidate into the research groups and assess his/her publication capacity.

The updated SIGAPS data for an individual researcher will be used to evaluate the scientific production profile of the candidate. The originality of the work, its relevance and the candidate's dynamism and investment in the discipline (participation in national and international conferences) will be taken into account.

When recruiting or promoting professors, it is useful to evaluate the production of the “second generation” researchers, that is the candidate’s students. Applications should also contain references to their production.

Bibliometrics in Economy

The following cannot summarise all the points of view of the economics community. The
population of economists is in itself difficult to define. The title of Section 37 of the CNRS is
“Economy and Management” and excludes Statistics, covered by Section 1 (Mathematics).
By contrast, the CNU makes the following distinction: Economics (Section 5), Management
Sciences (Section 6) while Statistics comes under the Applied Mathematics and Mathematical
Applications section (Section 26). Some economists work at the boundary of other disciplines
such as geography, history and sociology.

If one only considers CNRS Section 37 and CNU Section 5, then the number of economists in France is on the order of 2800.

As in all other disciplines, recruitment and promotions are based on peer evaluation.
Bibliometric indices are used to help evaluators. Section 37 of the CNRS has published a list
of 690 journals and has given each of them a grade from 0 to 4 (0 being considered the best
grade<fnref n="1" />). This list is now widely used. Section 37 of the CNRS and Section 5 of the CNU, the national evaluation agency for research and higher education (AERES) and recent academic evaluation panels (promotion from maître de conférences to professor) use this list which makes it easier to compare the publication profiles of the researchers being evaluated. Although this list is far from complete<fnref n="2" /> and economists disagree as to its contents, its creation and use (along with other criteria) provides an incentive (especially for young researchers) to publish more and better articles.

An important point is that on average economists do not publish very much<fnref n="3" /> (without any judgement here as to whether this should be considered good or bad). For example, 1% of the 2800 economists who publish the most have published about 30 articles (in the EconLit database). To be in the top 5%, 13 articles are required and 8 articles are required to be in the top 10%. This changes with age, young researchers publish more than those of generations who are about to retire.

By contrast, the use of citation factors has not become a common practice<fnref n="4" />.

Recently, Section 5 of the CNU was divided as to the use of a minimum threshold for the number of publications in good journals to evaluate a maître de conférences (for promotion to a professor). A threshold was applied but certain members of the CNU protested against the exclusive use of the publication criteria.

A two-fold conclusion emerges: on the one hand, bibliometric factors (number of publications weighed by the mean quality of the journals or factor directly based on the number of citations for the articles) can be used mostly for the most productive researchers (below a certain percentile, the profile of all researchers is too similar). Their use can shed meaningful light on the choice of candidates for certain promotions (such as promotion to a first-class professorship or CNRS promotion from researcher to research director). Although bibliometric indices are inadequate to make a distinction among the younger candidates, their (even partial) use can be a good incentive for young researchers to improve their standing.

In conclusion, we have two comments. First, it is important to use numbers (value of a given index for a given researcher) associated with a context that gives them meaning, such as relative to a wider framework (a distribution). For example, an index I of a researcher R has a

<footnote n="1">¹ This grade was issued following a qualitative evaluation of the journals by the national committee of CNRS Section 37, and not by mechanical application of bibliometric factors such as the impact factor. In Germany, the German Economic Association published a similar list in 2008.</footnote>

<footnote n="2">² For example, the American Economic Association database (EconLit) contains 1050 journals. Many statistical journals have been excluded from both the CNRS and EconLit lists.</footnote>

<footnote n="3">³ One possible explanation may be the length of the publication process. A working document is published in a journal two to three years after it has been written (and sometimes later).</footnote>

<footnote n="4">⁴ For example, only 300 economy journals (approximately) are in the SSCI citation database, only one of which is a French journal (and many economists still prefer publishing in French) and use of *Google Scholar* seems relatively difficult. Furthermore, the citation distribution is even less fair than that of publications.</footnote>
value of *x* and this ranks him/her among the *n* best researchers of his generation for his field. Secondly, any evaluation should be made according to a pre-established clear framework for analysis and should be summarised in a report communicated to the person concerned.

**Bibliometrics in the Social and Human Sciences**

The Social and Human Sciences, a wide diversity of situations can be found depending on the discipline. However, a number of general observations and proposals can be made.

It is not currently possible, and will not be on the short term, to calculate the number of citations reliably enough so as to use them in an evaluation process. The huge differences observed for calculations made for the same researcher using two different databases (*ISI Web of Science* or *Google Scholar*) reflect this difficulty. These databases are either too small or too widely inclusive and cannot pertinently reflect the scientific activity in SHS – and Thomson Reuters managers confirm this observation. There are a number of explanations for this. Publications may take several forms beyond articles. In the case of books, it is not possible to draw a boundary between scientific and more general publications. Each researcher has a low total number of publications but overall there is a high number of publications for each discipline. There may be legitimate reasons for not publishing only in English (or French). There is a higher citation frequency for older articles (pre-2000 and even pre-1980).

It would be wrong to use bibliometric indices for individual evaluation in SHS – including
recruitment, the nexus of all difficulties in SHS in matters of evaluation. However, following
the discussions of the working group, several recommendations can be put forward to
improve individual evaluation, including its bibliographic and even bibliometric aspects:

- use a standard CV format in each discipline, valid for all evaluations and institutions, that makes a clear distinction between publications that have been peer-reviewed and others, including books;

- as is the case for all other journals, SHS journal collections should aim at obtaining a CNRS label (given by the CoNRS sections) which is associated with funding;

- encourage the creation of a Web portal containing summaries of SHS publications, at the French or preferably at European and even world level (part of this project is on-going under the name recensio.net);

- encourage the presence on evaluation panels of two scientists from different disciplines (within SHS or outside SHS when this is justified by the profile of the candidate), who will challenge the “peers” in the strictest sense to explain their judgement and avoid any favoritism.

The general recommendations put forward in the present report also apply to the SHS.
ANNEX 4

PUBLICATION ETHICS FOR SCIENTIFIC WORKS

THE VANCOUVER CRITERIA

Publication Ethics : Sponsorship, Authorship and Accountability
International Committee of Medical Journal Editors

Uniform Requirements for Manuscripts Submitted to Biomedical
Journals: Writing and Editing for Biomedical Publication

Updated April 2010

The following information is available to be viewed/
printed in Adobe Acrobat pdf format.

I. Statement of Purpose

A. About the Uniform Requirements

B. Potential Users of the Uniform Requirements

C. How to Use the Uniform Requirements

II. Ethical Considerations in the Conduct and Reporting of Research

A. Authorship and Contributorship

1. Byline Authors

2. Contributors Listed in Acknowledgments

B. Editorship

1. The Role of the Editor

2. Editorial Freedom

C. Peer Review

D. Conflicts of Interest

1. Potential Conflicts of Interest Related to
Individual Authors' Commitments

2. Potential Conflicts of Interest Related to
Project Support

3. Potential Conflicts of Interest Related to
Commitments of Editors, Journal Staff, or
Reviewers

E. Privacy and Confidentiality

1. Patients and Study Participants

2. Authors and Reviewers

F. Protection of Human Subjects and Animals in Research

III. Publishing and Editorial Issues Related to Publication in Biomedical Journals

A. Obligation to Publish Negative Studies

B. Corrections, Retractions, and "Expressions of Concern"

C. Copyright

D. Overlapping Publications

1. Duplicate Submission

2. Redundant Publication

3. Acceptable Secondary Publication

4. Competing Manuscripts Based on the Same Study

a. Differences in Analysis or Interpretation

b. Differences in Reported Methods or Results

5. Competing Manuscripts Based on the Same Database

E. Correspondence

F. Supplements, Theme Issues, and Special Series

G. Electronic Publishing

H. Advertising

I. Medical Journals and the General Media

J. Obligation to Register Clinical Trials

IV. Manuscript Preparation and Submission

A. Preparing a Manuscript for Submission to Biomedical Journals

1. a. General Principles

b. Reporting Guidelines for Specific Study Designs

2. Title Page

3. Conflict-of-Interest Notification Page

4. Abstract and Key Words

5. Introduction

6. Methods

a. Selection and Description of Participants

b. Technical Information

c. Statistics

7. Results

8. Discussion

9. References

a. General Considerations Related to References

b. Reference Style and Format

10. Tables

11. Illustrations (Figures)

12. Legends for Illustrations (Figures)

13. Units of Measurement

14. Abbreviations and Symbols

B. Sending the Manuscript to the Journal

V. References

A. Print References Cited in this Document

B. Other Sources of Information Related to Biomedical Journals

VI. About the International Committee of Medical Journal Editors

VII. Authors of the Uniform Requirements

VIII. Use, Distribution, and Translation of the Uniform Requirements

IX. Inquiries

1
# I. STATEMENT OF PURPOSE

## I. A. About the Uniform Requirements

A small group of editors of general medical journals met informally in Vancouver, British Columbia, in 1978 to establish guidelines for the format of manuscripts submitted to their journals. This group became known as the Vancouver Group. Its requirements for manuscripts, including formats for bibliographic references developed by the National Library of Medicine (NLM), were first published in 1979. The Vancouver Group expanded and evolved into the International Committee of Medical Journal Editors (ICMJE), which meets annually. The ICMJE has gradually broadened its concerns to include ethical principles related to publication in biomedical journals.

The ICMJE has produced multiple editions of the Uniform Requirements for Manuscripts Submitted to Biomedical Journals. Over the years, issues have arisen that go beyond manuscript preparation, resulting in development of a number of Separate Statements on editorial policy. The entire Uniform Requirements document was revised in 1997; sections were updated in May 1999 and May 2000. In May 2001, the ICMJE revised the sections related to potential conflict of interest. In 2003, the committee revised and reorganized the entire document and incorporated the Separate Statements into the text. The committee prepared this revision in 2010.

The total content of the Uniform Requirements for Manuscripts Submitted to Biomedical Journals may be reproduced for educational, not-for-profit purposes without regard for copyright; the committee encourages distribution of the material.

Journals that agree to use the Uniform Requirements are encouraged to state in their Instructions to Authors that their requirements are in accordance with the Uniform Requirements and to cite this version. Journals that wish to be listed on www.icmje.org as a publication that follows the Uniform Requirements should contact the ICMJE secretariat office.

The ICMJE is a small working group of general medical journals, not an open-membership organization. Occasionally, the ICMJE will invite a new member or guest when the committee feels that the journal or organization will provide a new perspective. Open membership organizations for editors and others in biomedical publication include the World Association of Medical Editors www.WAME.org, the Council of Science Editors (www.councilscienceeditors.org/), and the European Association of Science Editors (www.ease.org.uk).

## I. B. Potential Users of the Uniform Requirements

The ICMJE created the Uniform Requirements primarily to help authors and editors in their mutual task of creating and distributing accurate, clear, easily accessible reports of biomedical studies. The initial sections address the ethical principles related to the process of evaluating, improving, and publishing manuscripts in biomedical jour-

nals and the relationships among editors and authors, peer reviewers, and the media. The latter sections address the more technical aspects of preparing and submitting manuscripts. The ICMJE believes that the entire document is relevant to the concerns of both authors and editors.

The Uniform Requirements can provide many other stakeholders—peer reviewers, publishers, the media, patients and their families, and general readers—with useful insights into the biomedical authoring and editing process.

## I. C. How to Use the Uniform Requirements

The Uniform Requirements state the ethical principles in the conduct and reporting of research and provide recommendations relating to specific elements of editing and writing. These recommendations are based largely on the shared experience of a moderate number of editors and authors, collected over many years, rather than on the results of methodical, planned investigation that aspires to be "evidence-based." Wherever possible, recommendations are accompanied by a rationale that justifies them; as such, the document serves an educational purpose.

Authors will find it helpful to follow the recommendations in this document whenever possible because, as described in the explanations, doing so improves the quality and clarity of reporting in manuscripts submitted to any journal, as well as the ease of editing. At the same time, every journal has editorial requirements uniquely suited to its purposes. Authors therefore need to become familiar with the Instructions to Authors specific to the journal they have chosen for their manuscript—for example, the topics suitable for that journal, and the types of papers that may be submitted (for example, original articles, reviews, or case reports)—and should follow those instructions.

# II. ETHICAL CONSIDERATIONS IN THE CONDUCT AND REPORTING OF RESEARCH

## II. A. Authorship and Contributorship

### II. A. 1. Byline Authors

An "author" is generally considered to be someone who has made substantive intellectual contributions to a published study, and biomedical authorship continues to have important academic, social, and financial implications (1). An author must take responsibility for at least one component of the work, should be able to identify who is responsible for each other component, and should ideally be confident in their co-authors' ability and integrity. In the past, readers were rarely provided with information about contributions to studies from persons listed as authors and in Acknowledgments (2). Some journals now request and publish information about the contributions of each person named as having participated in a submitted study, at least for original research. Editors are strongly encouraged to develop and implement a contributorship policy, as well as a policy on identifying who is responsible for the integrity of the work as a whole.

While contributorship and guarantorship policies ob-
viously remove much of the ambiguity surrounding contri-
butions, they leave unresolved the question of the quantity
and quality of contribution that qualify for authorship.
The ICJME has recommended the following criteria for
authorship; these criteria are still appropriate for journals
that distinguish authors from other contributors.

* Authorship credit should be based on 1) substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; 2) drafting the article or revising it critically for important intellectual content; and 3) final approval of the version to be published. Authors should meet conditions 1, 2, and 3.

* When a large, multicenter group has conducted the work, the group should identify the individuals who accept direct responsibility for the manuscript (3). These individuals should fully meet the criteria for authorship/ contributorship defined above, and editors will ask these individuals to complete journal-specific author and conflict-of-interest disclosure forms. When submitting a manuscript authored by a group, the corresponding author should clearly indicate the preferred citation and identify all individual authors as well as the group name. Journals generally list other members of the group in the Acknowledgments. The NLM indexes the group name and the names of individuals the group has identified as being directly responsible for the manuscript; it also lists the names of collaborators if they are listed in Acknowledgments.

* Acquisition of funding, collection of data, or general supervision of the research group alone does not constitute authorship.

* All persons designated as authors should qualify for authorship, and all those who qualify should be listed.

* Each author should have participated sufficiently in the work to take public responsibility for appropriate portions of the content.

Some journals now also request that one or more au-
thors, referred to as "guarantors," be identified as the per-
sons who take responsibility for the integrity of the work as
a whole, from inception to published article, and publish
that information.

Increasingly, authorship of multicenter trials is attrib-
uted to a group. All members of the group who are named
as authors should fully meet the above criteria for author-
ship/contributorship.

The group should jointly make decisions about
contributors/authors before submitting the manuscript for
publication. The corresponding author/guarantor should
be prepared to explain the presence and order of these
individuals. It is not the role of editors to make authorship/
contributorship decisions or to arbitrate conflicts related to
authorship.

II. A. 2. Contributors Listed in Acknowledgments

All contributors who do not meet the criteria for au-
thorship should be listed in an acknowledgments section.
Examples of those who might be acknowledged include a

person who provided purely technical help, writing assis-
tance, or a department chairperson who provided only gen-
eral support. Editors should ask corresponding authors to
declare whether they had assistance with study design, data
collection, data analysis, or manuscript preparation. If such
assistance was available, the authors should disclose the
identity of the individuals who provided this assistance and
the entity that supported it in the published article. Finan-
cial and material support should also be acknowledged.

Groups of persons who have contributed materially to
the paper but whose contributions do not justify author-
ship may be listed under such headings as "clinical inves-
tigators" or "participating investigators," and their function
or contribution should be described—for example, "served
as scientific advisors," "critically reviewed the study pro-
posal," "collected data," or "provided and cared for study
patients." Because readers may infer their endorsement of
the data and conclusions, these persons must give written
permission to be acknowledged.

II. B. Editorship

II. B. 1. The Role of the Editor

The editor of a journal is the person responsible for its
entire content. Owners and editors of medical journals
have a common endeavor—publication of a reliable, read-
able journal produced with due respect for the stated aims
of the journal and for costs. Owners and editors, however,
have different functions. Owners have the right to appoint
and dismiss editors and to make important business deci-
sions in which editors should be involved to the fullest
extent possible. Editors must have full authority for deter-
mining the editorial content of the journal. The concept of
editorial freedom should be resolutely defended by editors
even to the extent of their placing their positions at stake.
To secure this freedom in practice, the editor should have
direct access to the highest level of ownership, not to a
delegated manager.

Editors of medical journals should have a contract that
clearly states their rights and duties, the general terms
of the appointment, and the mechanisms for resolving
conflict.

An independent editorial advisory board may be use-
ful in helping the editor establish and maintain editorial
policy.

II. B. 2. Editorial Freedom

The ICMJE adopts the World Association of Medical
Editors' definition of editorial freedom. According to this
definition, editorial freedom, or independence, is the con-
cept that editors-in-chief have full authority over the edi-
torial content of their journal and the timing of publica-
tion of that content. Journal owners should not interfere in
the evaluation, selection, or editing of individual articles
either directly or by creating an environment that strongly
influences decisions. Journal owners should not require ed-
itors to publish supplements as part of their contractual
agreements. Editors should base decisions on the validity of the work and its importance to the journal's readers, not on the commercial success of the journal. Editors should be free to express critical but responsible views about all aspects of medicine without fear of retribution, even if these views conflict with the commercial goals of the publisher. Editors and editors' organizations are obligated to support the concept of editorial freedom and to draw major transgressions of such freedom to the attention of the international medical, academic, and lay communities.

## II. C. Peer Review

Unbiased, independent, critical assessment is an intrinsic part of all scholarly work, including the scientific process. Peer review is the critical assessment of manuscripts submitted to journals by experts who are not part of the editorial staff. Peer review can therefore be viewed as an important extension of the scientific process. Although its actual value has been little studied and is widely debated (4), peer review helps editors decide which manuscripts are suitable for their journals and helps authors and editors to improve the quality of reporting. A peer-reviewed journal submits most of its published research articles for outside review. The number and kinds of manuscripts sent for review, the number of reviewers, the reviewing procedures, and the use made of the reviewers' opinions may vary. In the interests of transparency, each journal should publicly disclose its policies and average turn-around times in its Instructions to Authors.

## II. D. Conflicts of Interest

Public trust in the peer-review process and the credibility of published articles depends in part on how well conflict of interest is handled during writing, peer review, and editorial decision making. Conflict of interest exists when an author (or the author's institution), reviewer, or editor has financial or personal relationships that inappropriately influence (bias) his or her actions (such relationships are also known as dual commitments, competing interests, or competing loyalties). These relationships vary from being negligible to having great potential for influencing judgment. Not all relationships represent true conflict of interest. On the other hand, the potential for conflict of interest can exist regardless of whether an individual believes that the relationship affects his or her scientific judgment. Financial relationships (such as employment, consultancies, stock ownership, honoraria, and paid expert testimony) are the most easily identifiable conflicts of interest and the most likely to undermine the credibility of the journal, the authors, and of science itself. However, conflicts can occur for other reasons, such as personal relationships, academic competition, and intellectual passion.

All participants in the peer-review and publication process must disclose all relationships that could be viewed as potential conflicts of interest. Disclosure of such relationships is also important in connection with editorials

and review articles, because it can be more difficult to detect bias in these types of publications than in reports of original research. Editors may use information disclosed in conflict-of-interest and financial-interest statements as a basis for editorial decisions. Editors should publish this information if they believe it is important in judging the manuscript.

## II. D. 1. Potential Conflicts of Interest Related to Individual Authors' Commitments

When authors submit a manuscript, whether an article or a letter, they are responsible for disclosing all financial and personal relationships that might bias their work. To prevent ambiguity, authors must state explicitly whether potential conflicts do or do not exist. Authors should do so in the manuscript on a conflict-of-interest notification page that follows the title page, providing additional detail, if necessary, in a cover letter that accompanies the manuscript. (See Section IV. A. 3. Conflict-of-Interest Disclosure. The ICMJE developed a uniform disclosure form that ICMJE member journals piloted in 2009. The second version of the form is now available. Other journals are welcome to adopt this form.)

Authors should identify individuals who provide writing or other assistance and disclose the funding source for this assistance.

Investigators must disclose potential conflicts to study participants and should state in the manuscript whether they have done so.

Editors also need to decide whether to publish information disclosed by authors about potential conflicts. If doubt exists, it is best to err on the side of publication.

## II. D. 2. Potential Conflicts of Interest Related to Project Support

Increasingly, individual studies receive funding from commercial firms, private foundations, and government. The conditions of this funding have the potential to bias and otherwise discredit the research.

Scientists have an ethical obligation to submit creditable research results for publication. Researchers should not enter into agreements that interfere with their access to all of the data and their ability to analyze them independently, and to prepare and publish manuscripts. Authors should describe the role of the study sponsor, if any, in study design; collection, analysis, and interpretation of data; writing the report; and the decision to submit the report for publication. If the supporting source had no such involvement, the authors should so state. Biases potentially introduced when sponsors are directly involved in research are analogous to methodological biases. Some journals, therefore, choose to include information in the Methods section about the sponsor's involvement.

Editors may request that authors of a study funded by an agency with a proprietary or financial interest in the
outcome sign a statement, such as "I had full access to all of the data in this study and I take complete responsibility for the integrity of the data and the accuracy of the data analysis." Editors should be encouraged to review copies of the protocol and/or contracts associated with project-specific studies before accepting such studies for publication. Editors may request a statistical analysis of all data by an independent biostatistician. Editors may choose not to consider an article if a sponsor has asserted control over the authors' right to publish.

## II. D. 3. Potential Conflicts of Interest Related to Commitments of Editors, Journal Staff, or Reviewers

Editors should avoid selecting external peer reviewers with obvious potential conflicts of interest—for example, those who work in the same department or institution as any of the authors. Authors often provide editors with the names of persons they feel should not be asked to review a manuscript because of potential, usually professional, conflicts of interest. When possible, authors should be asked to explain or justify their concerns: that information is important to editors in deciding whether to honor such requests.

Reviewers must disclose to editors any conflicts of interest that could bias their opinions of the manuscript, and they should recuse themselves from reviewing specific manuscripts if the potential for bias exists. As in the case of authors, silence on the part of reviewers concerning potential conflicts may mean either that conflicts exist and the reviewer has failed to disclose them or conflicts do not exist. Reviewers must therefore also be asked to state explicitly whether conflicts do or do not exist. Reviewers must not use knowledge of the work, before its publication, to further their own interests.

Editors who make final decisions about manuscripts must have no personal, professional, or financial involvement in any of the issues they might judge. Other members of the editorial staff, if they participate in editorial decisions, must provide editors with a current description of their financial interests (as they might relate to editorial judgments) and recuse themselves from any decisions in which a conflict of interest exists. Editorial staff must not use information gained through working with manuscripts for private gain. Editors should publish regular disclosure statements about potential conflicts of interests related to the commitments of journal staff.

## II. E. Privacy and Confidentiality

### II. E. 1. Patients and Study Participants

Patients have a right to privacy that should not be violated without informed consent. Identifying information, including names, initials, or hospital numbers, should not be published in written descriptions, photographs, or pedigrees unless the information is essential for scientific purposes and the patient (or parent or guardian) gives written informed consent for publication. Informed consent for this purpose requires that an identifiable patient be

shown the manuscript to be published. Authors should disclose to these patients whether any potential identifiable material might be available via the Internet as well as in print after publication. Patient consent should be written and archived with the journal, the authors, or both, as dictated by local regulations or laws. Applicable laws vary from locale to locale, and journals should establish their own policies with legal guidance. Since a journal that archives the consent will be aware of patient identity, some journals may decide that patient confidentiality is better guarded by having the author archive the consent and instead providing the journal with a written statement that attests that they have received and archived written patient consent.

Nonessential identifying details should be omitted. Informed consent should be obtained if there is any doubt that anonymity can be maintained. For example, masking the eye region in photographs of patients is inadequate protection of anonymity. If identifying characteristics are altered to protect anonymity, such as in genetic pedigrees, authors should provide assurance, and editors should so note, that such alterations do not distort scientific meaning.

The requirement for informed consent should be included in the journal's Instructions for Authors. When informed consent has been obtained, it should be indicated in the published article.

### II. E. 2. Authors and Reviewers

Manuscripts must be reviewed with due respect for authors' confidentiality. In submitting their manuscripts for review, authors entrust editors with the results of their scientific work and creative effort, on which their reputation and career may depend. Authors' rights may be violated by disclosure of the confidential details during review of their manuscript. Reviewers also have rights to confidentiality, which must be respected by the editor. Confidentiality may have to be breached if dishonesty or fraud is alleged but otherwise must be honored.

Editors must not disclose information about manuscripts (including their receipt, content, status in the reviewing process, criticism by reviewers, or ultimate fate) to anyone other than the authors and reviewers. This includes requests to use the materials for legal proceedings.

Editors must make clear to their reviewers that manuscripts sent for review are privileged communications and are the private property of the authors. Therefore, reviewers and members of the editorial staff must respect the authors' rights by not publicly discussing the authors' work or appropriating their ideas before the manuscript is published. Reviewers must not be allowed to make copies of the manuscript for their files and must be prohibited from sharing it with others, except with the editor's permission. Reviewers should return or destroy copies of manuscripts after submitting reviews. Editors should not keep copies of rejected manuscripts.

Reviewer comments should not be published or other-
erwise publicized without permission of the reviewer, au-
thor, and editor.

Opinions differ on whether reviewers should remain anonymous. Authors should consult the Information for Authors of the journal to which they have chosen to submit a manuscript to determine whether reviews are anonymous. When comments are not signed, the reviewers' identity must not be revealed to the author or anyone else without the reviewers' permission.

Some journals publish reviewers' comments with the manuscript. No such procedure should be adopted without the consent of the authors and reviewers. However, reviewers' comments should be sent to other persons reviewing the same manuscript, which helps reviewers learn from the review process. Reviewers also may be notified of the editor's decision to accept or reject a manuscript.

II. F. Protection of Human Subjects and Animals in Research

When reporting experiments on human subjects, au-
thors should indicate whether the procedures followed
were in accordance with the ethical standards of the re-
sponsible committee on human experimentation (institu-
tional and national) and with the Helsinki Declaration of
1975, as revised in 2008 (5). If doubt exists whether the
research was conducted in accordance with the Helsinki
Declaration, the authors must explain the rationale for
their approach and demonstrate that the institutional re-
view body explicitly approved the doubtful aspects of the
study. When reporting experiments on animals, authors
should indicate whether the institutional and national
guide for the care and use of laboratory animals was
followed.

III. PUBLISHING AND EDITORIAL ISSUES RELATED TO PUBLICATION IN BIOMEDICAL JOURNALS

III. A. Obligation to Publish Negative Studies

Editors should seriously consider for publication any
carefully done study of an important question, relevant to
their readers, whether the results for the primary or any
additional outcome are statistically significant. Failure to
submit or publish findings because of lack of statistical
significance is an important cause of publication bias.

III. B. Corrections, Retractions, and "Expressions of Concern"

Editors must assume initially that authors are report-
ing work based on honest observations. Nevertheless, two
types of difficulty may arise.

First, errors may be noted in published articles that
require the publication of a correction or erratum on part
of the work. The corrections should appear on a numbered
page, be listed in the Table of Contents, include the com-
plete original citation, and link to the original article and
vice versa if online. It is conceivable that an error could be
so serious as to vitiate the entire body of the work, but this
is unlikely and should be addressed by editors and authors

on an individual basis. Such an error should not be con-
fused with inadequacies exposed by the emergence of new
scientific information in the normal course of research.
The latter requires no corrections or withdrawals.

The second type of difficulty is scientific fraud. If sub-
stantial doubt arises about the honesty or integrity of work,
either submitted or published, it is the editor's responsibil-
ity to ensure that the question is appropriately pursued,
usually by the authors' sponsoring institution. Ordinarily,
it is not the responsibility of the editor to conduct a full
investigation or to make a determination—that responsi-
bility lies with the institution where the work was done or
with the funding agency. The editor should be promptly
informed of the final decision, and if a fraudulent paper
has been published, the journal must print a retraction. If
this method of investigation does not result in a satisfac-
tory conclusion, the editor may choose to conduct his or
her own investigation. As an alternative to retraction, the
editor may choose to publish an expression of concern
about aspects of the conduct or integrity of the work.

The retraction or expression of concern, so labeled,
should appear on a numbered page in a prominent section
of the print journal as well as in the online version, be
listed in the Table of Contents page, and include in its
heading the title of the original article. It should not simply
be a letter to the editor. Ideally, the first author of the
retraction should be the same as that of the article, al-
though under certain circumstances the editor may accept
retractions by other responsible persons. The text of the
retraction should explain why the article is being retracted
and include a complete citation reference to that article.

The validity of previous work by the author of a fraud-
ulent paper cannot be assumed. Editors may ask the au-
thor's institution to assure them of the validity of earlier
work published in their journals or to retract it. If this is
not done, editors may choose to publish an announcement
expressing concern that the validity of previously published
work is uncertain.

Editors who have questions related to editorial or sci-
entific misconduct may find it useful to consult the excel-
lent flow charts that the Committee on Publication Ethics
(COPE) has developed (http://www publicationethics
.org.uk). COPE, which was formed in 1997, is a forum in
which editors of peer-reviewed journals can discuss issues
related to the integrity of the scientific record; it supports
and encourages editors to report, catalogue, and instigate
investigations into ethical problems in the publication pro-
cess. COPE's major objective is to provide a sounding board
for editors struggling with how best to deal with possible
breaches in research and publication ethics.

III. C. Copyright

Many biomedical journals ask authors to transfer
copyright to the journal. However, an increasing number
of "open-access" journals do not require transfer of copy-
right. Editors should make their position on copyright
transfer clear to authors and to others who might be interested in using editorial content from their journals. The copyright status of articles in a given journal can vary: Some content cannot be copyrighted (for example, articles written by employees of the U.S. or some other governments in the course of their work); editors may agree to waive copyright on others; and still others may be protected under serial rights (that is, use in publications other than journals, including electronic publications, is permitted).

### III. D. Overlapping Publications

#### III. D. 1. Duplicate Submission

Most biomedical journals will not consider manuscripts that are simultaneously being considered by other journals. Among the principal considerations that have led to this policy are: 1) the potential for disagreement when two (or more) journals claim the right to publish a manuscript that has been submitted simultaneously to more than one; and 2) the possibility that two or more journals will unknowingly and unnecessarily undertake the work of peer review, edit the same manuscript, and publish the same article.

However, editors of different journals may decide to simultaneously or jointly publish an article if they believe that doing so would be in the best interest of public health.

#### III. D. 2. Redundant Publication

Redundant (or duplicate) publication is publication of a paper that overlaps substantially with one already published in print or electronic media.

Readers of primary source periodicals, whether print or electronic, deserve to be able to trust that what they are reading is original unless there is a clear statement that the author and editor are intentionally republishing an article. The bases of this position are international copyright laws, ethical conduct, and cost-effective use of resources. Duplicate publication of original research is particularly problematic because it can result in inadvertent double-counting or inappropriate weighting of the results of a single study, which distorts the available evidence.

Most journals do not wish to receive papers on work that has already been reported in large part in a published article or is contained in another paper that has been submitted or accepted for publication elsewhere, in print or in electronic media. This policy does not preclude the journal from considering a paper that has been rejected by another journal, or a complete report that follows publication of a preliminary report, such as an abstract or poster displayed at a professional meeting. It also does not prevent journals from considering a paper that has been presented at a scientific meeting but was not published in full, or that is being considered for publication in a proceedings or similar format. Brief press reports of scheduled meetings are not usually regarded as breaches of this rule, but they may be if additional data or copies of tables and figures amplify such reports. The ICMJE does not consider results posted

in clinical trial registries as previous publication if the results are presented in the same, ICMJE-accepted registry in which initial registration of trial methods occurred and if the results are posted in the form of a brief structured abstract or table. The ICMJE also believes that the results registry should either cite full publications of the results when available or include a statement that indicates that the results have not yet been published in a peer-reviewed journal.

When submitting a paper, the author must always make a complete statement to the editor about all submissions and previous reports (including meeting presentations and posting of results in registries) that might be regarded as redundant or duplicate publication. The author must alert the editor if the manuscript includes subjects about which the authors have published a previous report or have submitted a related report to another publication. Any such report must be referred to and referenced in the new paper. Copies of such material should be included with the submitted manuscript to help the editor decide how to handle the matter.

If redundant or duplicate publication is attempted or occurs without such notification, authors should expect editorial action to be taken. At the least, prompt rejection of the submitted manuscript should be expected. If the editor was not aware of the violations and the article has already been published, then a notice of redundant or duplicate publication will probably be published with or without the author's explanation or approval.

Preliminary reporting to public media, governmental agencies, or manufacturers of scientific information described in a paper or a letter to the editor that has been accepted but not yet published violates the policies of many journals. Such reporting may be warranted when the paper or letter describes major therapeutic advances or public health hazards, such as serious adverse effects of drugs, vaccines, other biological products, medicinal devices, or reportable diseases. This reporting should not jeopardize publication, but should be discussed with and agreed upon by the editor in advance.

#### III. D. 3. Acceptable Secondary Publication

Certain types of articles, such as guidelines produced by governmental agencies and professional organizations, may need to reach the widest possible audience. In such instances, editors sometimes deliberately publish material that is also being published in other journals, with the agreement of the authors and the editors of those journals. Secondary publication for various other reasons, in the same or another language, especially in other countries, is justifiable and can be beneficial provided that the following conditions are met.

1. The authors have received approval from the editors of both journals (the editor concerned with secondary publication must have a photocopy, reprint, or manuscript of the primary version).
2. The priority of the primary publication is respected by a publication interval of at least 1 week (unless specifically negotiated otherwise by both editors).

3. The paper for secondary publication is intended for a different group of readers; an abbreviated version could be sufficient.

4. The secondary version faithfully reflects the data and interpretations of the primary version.

5. The footnote on the title page of the secondary version informs readers, peers, and documenting agencies that the paper has been published in whole or in part and states the primary reference. A suitable footnote might read: "This article is based on a study first reported in the [title of journal, with full reference]."

Permission for such secondary publication should be free of charge.

6. The title of the secondary publication should indicate that it is a secondary publication (complete republication, abridged republication, complete translation, or abridged translation) of a primary publication. Of note, the NLM does not consider translations to be "republications" and does not cite or index translations when the original article was published in a journal that is indexed in MEDLINE.

7. Editors of journals that simultaneously publish in multiple languages should understand that NLM indexes the primary language version. When the full text of an article appears in more than one language in a journal issue (such as Canadian journals with the article in both English and French), both languages are indicated in the MEDLINE citation (for example, Mercer K. The relentless challenge in health care. Healthc Manage Forum. 2008 Summer;21(2):4-5. English, French. No abstract available. PMID:18795553.)

### III. D. 4. Competing Manuscripts Based on the Same Study

Publication of manuscripts to air the disputes of co-investigators may waste journal space and confuse readers. On the other hand, if editors knowingly publish a manuscript written by only some of a collaborating team, they could be denying the rest of the team their legitimate co-authorship rights and journal readers access to legitimate differences of opinion about the interpretation of a study.

Two kinds of competing submissions are considered: submissions by coworkers who disagree on the analysis and interpretation of their study, and submissions by coworkers who disagree on what the facts are and which data should be reported.

Setting aside the unresolved question of ownership of the data, the following general observations may help editors and others address such problems.

### III. D. 4. a. Differences in Analysis or Interpretation

If the dispute centers on the analysis or interpretation of data, the authors should submit a manuscript that

clearly presents both versions. The difference of opinion should be explained in a cover letter. The normal process of peer and editorial review may help the authors to resolve their disagreement regarding analysis or interpretation.

If the dispute cannot be resolved and the study merits publication, both versions should be published. Options include publishing two papers on the same study, or a single paper with two analyses or interpretations. In such cases, it would be appropriate for the editor to publish a statement outlining the disagreement and the journal's involvement in attempts to resolve it.

### III. D. 4. b. Differences in Reported Methods or Results

If the dispute centers on differing opinions of what was actually done or observed during the study, the journal editor should refuse publication until the disagreement is resolved. Peer review cannot be expected to resolve such problems. If there are allegations of dishonesty or fraud, editors should inform the appropriate authorities; authors should be notified of an editor's intention to report a suspicion of research misconduct.

### III. D. 5. Competing Manuscripts Based on the Same Database

Editors sometimes receive manuscripts from separate research groups that have analyzed the same data set (for example, from a public database). The manuscripts may differ in their analytic methods, conclusions, or both. Each manuscript should be considered separately. If interpretation of the data is very similar, it is reasonable but not mandatory for editors to give preference to the manuscript that was received first. However, editorial consideration of multiple submissions may be justified under these circumstances, and there may even be a good reason to publish more than one manuscript because different analytical approaches may be complementary and equally valid.

### III. E. Correspondence

The corresponding author/guarantor has primary responsibility for correspondence with the journal, but the ICMJE recommends that editors send a copy of any correspondence to all listed authors.

Biomedical journals should provide the readership with a mechanism for submitting comments, questions, or criticisms about published articles, as well as brief reports and commentary unrelated to previously published articles. This probably but not necessarily takes the form of a correspondence section or column. The authors of articles discussed in correspondence should be given an opportunity to respond, preferably in the same issue in which the original correspondence appears. Authors of correspondence should be asked to declare any competing or conflicting interests.

Published correspondence may be edited for length, grammatical correctness, and journal style. Alternatively, editors may choose to publish unedited correspondence, for example in rapid-response sections on the Internet. The
journal should declare its editorial practices in this regard. Authors should approve editorial changes that alter the substance or tone of a letter or response. In all instances, editors must make an effort to screen discourteous, inaccurate, or libelous statements and should not allow ad hominem arguments intended to discredit opinions or findings.

Although editors have the prerogative to reject correspondence that is irrelevant, uninteresting, or lacking co-gency, they have a responsibility to allow a range of opinions to be expressed. The correspondence column should not be used merely to promote the journal's or the editors' point of view.

In the interests of fairness and to keep correspondence within manageable proportions, journals may want to set time limits for responding to published material and for debate on a given topic. Journals should also decide whether they would notify authors when correspondence bearing on their published work is going to appear in standard or rapid-response sections. Journals should also set policy with regard to the archiving of unedited correspondence that appears online. These policies should be published both in print and electronic versions of the journal.

### III. F. Supplements, Theme Issues, and Special Series

Supplements are collections of papers that deal with related issues or topics, are published as a separate issue of the journal or as part of a regular issue, and are usually funded by sources other than the journal's publisher. There is evidence that supplement content can be of lower quality than the content of the parent journal (6). Because funding sources can bias the content of supplements through the choice of topics and viewpoints, journals should consider adopting the following principles. These same principles apply to theme issues or special series that have external funding and/or guest editors.

1. The journal editor must be given and take full responsibility for the policies, practices, and content of supplements, including complete control of the decision to select authors, peer reviewers, and content for the supplement. Editing by the funding organization should not be permitted.

2. The journal editor must retain the authority to send supplement manuscripts for external peer review and to reject manuscripts submitted for the supplement. These conditions should be made known to authors and external supplement editors before beginning editorial work on the supplement.

3. The journal editor must approve the appointment of any external editor of the supplement and take responsibility for the work of the external editor.

4. The source of the idea for the supplement, sources of funding for the research, publication, and products of the funding source that are considered in the supplement should be clearly stated and prominently located in the

supplement, preferably on each page. Whenever possible, supplements should be funded by more than one sponsor.

5. Advertising in supplements should follow the same policies as those of the rest of the journal.

6. Journal editors must enable readers to distinguish readily between ordinary editorial pages and supplement pages.

7. Journal editors and supplement editors must not accept personal favors or remuneration from sponsors of supplements.

8. Secondary publication in supplements (republication of papers published elsewhere) should be clearly identified by the citation of the original paper. Supplements should avoid redundant or duplicate publication. Supplements should not republish research results, but republication of guidelines or other material in the public interest might be appropriate.

9. The principles of authorship and disclosure of potential conflicts of interest discussed elsewhere in this document should be applied to supplements.

### III. G. Electronic Publishing

Most biomedical journals are now published in electronic as well as print versions, and some are published only in electronic form. Because electronic publishing (which includes the Internet) is the same as publishing in print, in the interests of clarity and consistency the recommendations of this document should be applied to electronically published medical and health information.

The nature of electronic publication requires some special considerations, both within and beyond this document. At a minimum, Web sites should indicate the following: names, appropriate credentials, affiliations, and relevant conflicts of interest of editors, authors, and contributors; documentation and attribution of references and sources for all content; information about copyright; disclosure of site ownership; and disclosure of sponsorship, advertising, and commercial funding.

Linking from one health or medical Internet site to another may be perceived as an implicit recommendation of the quality of the second site. Journals thus should exercise caution in linking to other sites: when users are linking to another site, it may be helpful to provide an explicit statement that they are leaving the journal's site. Links to other sites posted as a result of financial considerations should be clearly indicated as such. All dates of content posting and updating should be indicated. In electronic layout as in print, advertising and promotional messages should not be juxtaposed with editorial content, and commercial content should be clearly identified as such.

Electronic publication is in flux. Editors should develop, make available to authors, and implement policies on issues unique to electronic publishing. These issues include archiving, error correction, version control, choice of the electronic or print version of the journal as the journal of record, and publication of ancillary material.
Under no circumstances should a journal remove an
article from its Web site or archive. If a correction or re-
traction becomes necessary, the explanation must be la-
belled appropriately and communicated as soon as possible
on a citable page in a subsequent issue of the journal.

Preservation of electronic articles in a permanent ar-
chive is essential for the historical record. Access to the
archive should be immediate and controlled by a third
party, such as a library, instead of the publisher. Deposi-
tion in multiple archives is encouraged.

### III. H. Advertising

Most medical journals carry advertising, which gener-
ates income for their publishers, but advertising must not
be allowed to influence editorial decisions. Journals should
have formal, explicit, written policies for advertising in
both print and electronic versions; Web site advertising
policy should parallel that for the printed journals. Editors
must have full and final authority for approving advertise-
ments and enforcing advertising policy.

When possible, editors should make use of the judg-
ments of independent bodies for reviewing advertising.
Readers should be able to distinguish readily between ad-
vertising and editorial material. The juxtaposition of edi-
torial and advertising material on the same products or
subjects should be avoided. Interspersing advertising pages
within articles interrupts the flow of editorial content and
should be discouraged. Advertising should not be sold on
the condition that it will appear in the same issue as a
particular article.

Journals should not be dominated by advertising, but
editors should be careful about publishing advertisements
from only one or two advertisers, as readers may perceive
that these advertisers have influenced the editor.

Journals should not carry advertisements for products
that have proved to be seriously harmful to health—for
example, tobacco. Editors should ensure that existing reg-
ulatory or industry standards for advertisements specific to
their country are enforced, or develop their own standards.
The interests of organizations or agencies should not con-
trol classified and other nondisplay advertising, except
where required by law. Finally, editors should consider all
criticisms of advertisements for publication.

### III. I. Medical Journals and the General Media

The public's interest in news of medical research has
led the popular media to compete vigorously for informa-
tion about research. Researchers and institutions some-
times encourage reporting research in the nonmedical me-
dia before full publication in a scientific journal by holding
a press conference or giving interviews.

The public is entitled to important medical informa-
tion within a reasonable amount of time, and editors have
a responsibility to facilitate the process. Biomedical jour-
nals are published primarily for their readers, but the gen-
eral public has a legitimate interest in their content: An

appropriate balance between these considerations should
guide the journal's interaction with the media. Doctors in
practice need to have reports available in full detail before
they can advise their patients about the reports' conclu-
sions. Moreover, media reports of scientific research before
the work has been peer-reviewed and fully vetted may lead
to dissemination of inaccurate or premature conclusions.

An embargo system has been established in some
countries to prevent publication of stories in the general
media before publication of the original research in the
journal. The embargo creates a "level playing field," which
most reporters appreciate since it minimizes the pressure
on them to publish stories they have not had time to pre-
pare carefully. Consistency in the timing of public release
of biomedical information is also important in minimizing
economic chaos, since some articles contain information
that has great potential to influence financial markets. On
the other hand, the embargo system has been challenged as
being self-serving of journals' interests and an impediment
to rapid dissemination of scientific information.

Editors may find the following recommendations use-
ful as they seek to establish policies on these issues.

* Editors can foster the orderly transmission of med-
ical information from researchers, through peer-reviewed
journals, to the public. This can be accomplished by an
agreement with authors that they will not publicize their
work while their manuscript is under consideration or
awaiting publication and an agreement with the media that
they will not release stories before publication of the orig-
inal research in the journal, in return for which the journal
will cooperate with them in preparing accurate stories.

* Editors need to keep in mind that an embargo sys-
tem works on the honor system; no formal enforcement or
policing mechanism exists. The decision of a signifi-
cant number of media outlets or biomedical journals not to
respect the embargo system would lead to its rapid
dissolution.

* Very little medical research has such clear and ur-
gently important clinical implications for the public's
health that the news must be released before full publica-
tion in a journal. However, if such exceptional circum-
stances occur, the appropriate authorities responsible for
public health should decide whether to disseminate infor-
mation to physicians and the media in advance and should
be responsible for this decision. If the author and the ap-
propriate authorities wish to have a manuscript considered
by a particular journal, the editor should be consulted be-
fore any public release. If editors acknowledge the need for
immediate release, they should waive their policies limiting
prepublication publicity.

* Policies designed to limit prepublication publicity
should not apply to accounts in the media of presentations
at scientific meetings or to the abstracts from these meet-
ings (see Redundant Publication). Researchers who present
their work at a scientific meeting should feel free to discuss
their presentations with reporters, but they should be dis-
couraged from offering more detail about their study than
was presented in the talk.

* When an article is soon to be published, editors should help the media prepare accurate reports by providing news releases, answering questions, supplying advance copies of the journal, or referring reporters to the appropriate experts. This assistance should be contingent on the media's cooperation in timing the release of a story to coincide with publication of the article.

* Editors, authors, and the media should apply the above-stated principles to material released early in electronic versions of journals.

### III. J. Obligation to Register Clinical Trials

The ICMJE believes that it is important to foster a comprehensive, publicly available database of clinical trials. The ICMJE defines a clinical trial as any research project that prospectively assigns human subjects to intervention or concurrent comparison or control groups to study the cause-and-effect relationship between a medical intervention and a health outcome. Medical interventions include drugs, surgical procedures, devices, behavioral treatments, process-of-care changes, and the like.

The ICMJE member journals will require, as a condition of consideration for publication in their journals, registration in a public trials registry. The details of this policy are contained in a series of editorials (see Editorials, under Frequently Asked Questions). The ICMJE encourages editors of other biomedical journals to adopt similar policy.

The ICMJE does not advocate one particular registry, but its member journals will require authors to register their trial in a registry that meets several criteria. The registry must be accessible to the public at no charge. It must be open to all prospective registrants and managed by a not-for-profit organization. There must be a mechanism to ensure the validity of the registration data, and the registry should be electronically searchable. Trial registration with missing fields or fields that contain uninformative terminology is inadequate.

It is important to note that the ICMJE requires registration of trial methodology but does not require registration of trial results; it recognizes the potential problems that could arise from the posting of research results that have not been subjected to an independent peer-review process. However, the ICMJE understands that the U.S. Food and Drug Administration Amendments Act of 2007 (FDAAA) does require researchers to register results. The ICMJE will not consider results to be previous publication if they are posted in the same primary clinical trial registry as the initial registration and if the results are posted in the tabular form dictated by the FDAAA. Researchers should be aware that editors of journals that follow the ICMJE recommendations may consider more detailed description of trial results and results published in registries other than the primary registry (in the case of FDAAA, ClinicalTrials.gov) to be prior publication. The ICMJE anticipates that

the climate for results registration will change dramatically over coming years and the ICMJE may need to amend these recommendations as additional agencies institute other mandates related to results registration.

The ICMJE recommends that journals publish the trial registration number at the end of the abstract. The ICMJE also recommends that, whenever a registration number is available, authors list this number the first time they use a trial acronym to refer to either the trial they are reporting or to other trials that they mention in the manuscript.

## IV. MANUSCRIPT PREPARATION AND SUBMISSION

### IV. A. Preparing a Manuscript for Submission to a Biomedical Journal

Editors and reviewers spend many hours reading manuscripts, and therefore appreciate receiving manuscripts that are easy to read and edit. Much of the information in a journal's Instructions to Authors is designed to accomplish that goal in ways that meet each journal's particular editorial needs. The following information provides guidance in preparing manuscripts for any journal.

#### IV. A. 1. a. General Principles

The text of observational and experimental articles is usually (but not necessarily) divided into the following sections: Introduction, Methods, Results, and Discussion. This so-called "IMRAD" structure is not an arbitrary publication format but rather a direct reflection of the process of scientific discovery. Long articles may need subheadings within some sections (especially Results and Discussion) to clarify their content. Other types of articles, such as case reports, reviews, and editorials, probably need to be formatted differently.

Electronic formats have created opportunities for adding details or whole sections, layering information, crosslinking or extracting portions of articles, and the like only in the electronic version. Authors need to work closely with editors in developing or using such new publication formats and should submit supplementary electronic material for peer review.

Double-spacing all portions of the manuscript—including the title page, abstract, text, acknowledgments, references, individual tables, and legends—and generous margins make it possible for editors and reviewers to edit the text line by line and add comments and queries directly on the paper copy. If manuscripts are submitted electronically, the files should be double-spaced to facilitate printing for reviewing and editing.

Authors should number all of the pages of the manuscript consecutively, beginning with the title page, to facilitate the editorial process.
#### IV. A. 1. b. Reporting Guidelines for Specific Study Designs

Research reports frequently omit important information. Reporting guidelines have been developed for a number of study designs that some journals may ask authors to follow. Authors should consult the Information for Authors of the journal they have chosen.

The general requirements listed in the next section relate to reporting essential elements for all study designs. Authors are encouraged also to consult reporting guidelines relevant to their specific research design. A good source of reporting guidelines is the EQUATOR Network (http://www.equator-network.org/home/).

#### IV. A. 2. Title Page

The title page should have the following information:

1. Article title. Concise titles are easier to read than long, convoluted ones. Titles that are too short may, however, lack important information, such as study design (which is particularly important in identifying randomized, controlled trials). Authors should include all information in the title that will make electronic retrieval of the article both sensitive and specific.

2. Authors' names and institutional affiliations. Some journals publish each author's highest academic degree(s), while others do not.

3. The name of the department(s) and institution(s) to which the work should be attributed.

4. Disclaimers, if any.

5. Contact information for corresponding authors. The name, mailing address, telephone and fax numbers, and e-mail address of the author responsible for correspondence about the manuscript (the "corresponding author;" this author may or may not be the "guarantor" for the integrity of the study). The corresponding author should indicate clearly whether his or her e-mail address can be published.

6. The name and address of the author to whom requests for reprints should be addressed or a statement that reprints are not available from the authors.

7. Source(s) of support in the form of grants, equipment, drugs, or all of these.

8. A running head. Some journals request a short running head or footnote, usually no more than 40 characters (including letters and spaces) at the foot of the title page. Running heads are published in most journals, but are also sometimes used within the editorial office for filing and locating manuscripts.

9. Word counts. A word count for the text only (excluding abstract, acknowledgments, figure legends, and references) allows editors and reviewers to assess whether the information contained in the paper warrants the amount of space devoted to it, and whether the submitted manuscript fits within the journal's word limits. A separate word count for the Abstract is useful for the same reason.

10. The number of figures and tables. It is difficult for editorial staff and reviewers to determine whether the figures and tables that should have accompanied a manuscript were actually included unless the numbers of figures and tables are noted on the title page.

#### IV. A. 3. Conflict-of-interest Notification Page

To prevent potential conflicts of interest from being overlooked or misplaced, this information needs to be part of the manuscript. The ICMJE has developed a uniform disclosure form for use by ICMJE member journals (http://www.icmje.org/coi_disclosure.pdf). Other journals are welcome to adopt this form. Individual journals may differ in where they include this information, and some journals do not send information on conflicts of interest to reviewers. (See Section II. D. Conflicts of Interest.

#### IV. A. 4. Abstract

Structured abstracts are preferred for original research and systematic reviews. The abstract should provide the context or background for the study and should state the study's purpose, basic procedures (selection of study subjects or laboratory animals, observational and analytical methods), main findings (giving specific effect sizes and their statistical significance, if possible), principal conclusions, and funding sources. It should emphasize new and important aspects of the study or observations. Articles on clinical trials should contain abstracts that include the items that the CONSORT group has identified as essential (http://www.consort-statement.org/? = 1190 ).

Because abstracts are the only substantive portion of the article indexed in many electronic databases, and the only portion many readers read, authors need to be careful that they accurately reflect the content of the article. Unfortunately, the information contained in many abstracts differs from that in the text (7). The format required for structured abstracts differs from journal to journal, and some journals use more than one format; authors need to prepare their abstracts in the format specified by the journal they have chosen.

The ICMJE recommends that journals publish the trial registration number at the end of the abstract. The ICMJE also recommends that, whenever a registration number is available, authors list that number the first time they use a trial acronym to refer to either the trial they are reporting or to other trials that they mention in the manuscript.

#### IV. A. 5. Introduction

Provide a context or background for the study (that is, the nature of the problem and its significance). State the specific purpose or research objective of, or hypothesis tested by, the study or observation; the research objective is often more sharply focused when stated as a question. Both the main and secondary objectives should be clear, and any