Research Counts, Not the Journal

Miguel Abambres, Tiago Ribeiro, Ana E Sousa, Eva Lantsoght

► To cite this version:

Miguel Abambres, Tiago Ribeiro, Ana E Sousa, Eva Lantsoght. Research Counts, Not the Journal. 2018. hal-02074859v3

HAL Id: hal-02074859

https://hal.science/hal-02074859v3

Preprint submitted on 15 Nov 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Research Counts, Not the Journal

Miguel Abambres <fnref n="1" />, Tiago Ribeiro <fnref n="2" />, Ana Sousa <fnref n="2" /> and Eva Lantsoght <fnref n="3" />, <fnref n="4" />

**Abstract:** 'If there is one thing every bibliometrician agrees, is that you should never use the journal impact factor (JIF) to evaluate research performance for an article or an individual – that is a mortal sin'. Few sentences could define so precisely the uses and misuses of the Journal Impact Factor (JIF) better than Anthony van Raan's. This manuscript presents a critical overview on the international use, by governments and institutions, of the JIF and/or journal indexing information for individual research quality assessment. Interviews given by Nobel Laureates speaking on this matter are partially illustrated in this work. Furthermore, the authors propose complementary and alternative versions of the journal impact factor, respectively named Complementary (CIF) and Timeless (TIF) Impact Factors, aiming to better assess the average quality of a journal – never of a paper or an author. The idea behind impact factors is not useless, it has just been misused.

**Keywords:** Scientific Papers, Impact Factor, Indexing Databases, Research Quality, Scientific Performance, Scientometrics.

## 1. Introduction

The Journal Impact Factor (JIF), today calculated by Clarivate Analytics (previously intellectual property of Thomson Reuters), was originally proposed by Irving H. Sher and Eugene Garfield in 1963 [1] as a tool to help librarians identify journals worth purchasing; not as a measure of the research quality in an article. The JIF is computed in each annual issue of the Journal Citation Reports (JCR, an annual publication by Clarivate Analytics), and represents the average number of citations per article in that year, concerning only papers published in that journal in the last two years:

$$ JIF_{year \ n} = \frac{\# \text{WoS citations in year } n \text{ to papers published in years } (n-1) \text{ and } (n-2)}{\# \text{ papers published in years } (n-1) \text{ and } (n-2)} \quad (1) $$

where '# WoS citations' is the number of citations from articles published in journals indexed in Web of Science (WoS) database.

The number of inbound citations in the JCR only takes into account citing sources indexed in the WoS database. A similar metric is the 5-year impact factor (IF), which is calculated alike the former, but based

<footnote n="1">¹ R&D, Abambres' Lab, 1600-275 Lisbon, Portugal; abambres@netcabo.pt</footnote>

<footnote n="2">² Independent Researcher, Lisbon, Portugal</footnote>

<footnote n="3">³ Researcher, Department of Engineering Structures, Delft University of Technology, The Netherlands; E.O.L.Lantsoght@tudelft.nl</footnote>

<footnote n="4">⁴ Professor, Politécnico, Universidad San Francisco de Quito, Quito, Ecuador</footnote>
on five years (instead of two) prior to the JCR year. Citing Clarivate Analytics [2]: "the JIF is a journal-level metric, thus it does not apply to individual or subgroups of papers published in the journal, nor to authors of papers, research groups or institutions." Even so, it's well-known in the scientific community that the JIF has increasingly become a means of judging the scientific quality of a journal and of everything that is published in it [3, 4].

According to Pudovkin [5], and confirmed by the authors, there is extensive literature criticizing the JIF (mis)uses. The main argument against the use of JIF for characterization of individual papers and scientists is a lack of strong correlation between its value and the citedness of the individual papers published in the journal. There is no guarantee that an article published in a high IF journal will attract profuse citations [6]. It is known [7] that only a fraction of the papers published in a high IF journal receive most citations contributing to the JIF, regularly known in scientometrics as the 'invitation's paradox' [8]. By the end of the last century, Seglen [9] reported that 15 and 50 per cent of the most cited papers account for 50 and 90 percent of the citations used to compute the JIF, respectively. Moreover, (i) information abounds about classic highly-cited papers that were initially rejected by high IF journals [10, 11], and (ii) data from several lines of evidence suggest that the reliability of scientific experiments decreases with increasing JIF [12].

This manuscript presents a critical overview on the use of the journal impact factor and/or journal indexing information for the scientific performance assessment of a researcher. The authors present a research-based opinion article, which addresses a serious problem that, in our opinion, remains fully topical until it is solved. Particularly, Figures 1-7 partially illustrate interviews given by Nobel Laureates on this topic. Lastly, complementary and alternative versions of the JIF are proposed, respectively called Complementary (CIF) and Timeless (TIF) Impact Factors, aiming to better assess the average quality of a journal – never of a paper or an author. The idea behind impact factors is not useless; it has just been misused. The authors believe that in the ever-evolving landscape of science, it is necessary to keep the discussion going and to keep searching for improvements.

## 2. The JIF and indexing information as individual research quality measures

Deans, sponsors, government agencies and employment panels use the JIF as a convenient, yet flawed, performance measure [13]. According to Pudovkin's 30-year experience [5] as a member of research staff recruitment committees at the institute of Marine Biology and the Russian Academy of Sciences, members in those committees typically just look at the journal titles. In many Universities in non-English-speaking countries, only papers published in WoS-indexed journals count towards merit [14]. The authors of this article oppose these types of judgement and support the opinions shared by many scientists as presented in the next section. In what follows, several examples are given of countries and institutions supporting these types of (mis)evaluation.

Instituto Superior Técnico, one of the most renowned schools of engineering and technology in Portugal (the only Portuguese member of the CLUSTER – Consortium Linking Universities of Science and Technology for Education and Research [15]), part of the University of Lisbon, considers the journal impact factor quartiles published in WoS and/or Scopus databases to compute a quantitative measure of the scientific performance of its faculty staff (as described in chapter II, article 17 of UL [16]) – papers from journals in the 1st (Q1), 2nd or 3rd quartiles are given six, four or two times higher classification than the Q4
counterpart, respectively. Instituto Superior Técnico is also known, at least in some departments, for disregarding or giving much less credit to articles not published in Q₁/Q₂ WoS- (e.g., UL 2017b, d [17, 18]) or Scopus- (e.g., [19]) indexed journals when assessing candidates for faculty positions in public tenders.

Ostravská univerzita v Ostravě [20] shows an example in the Czech Republic of a job application requiring a list of publications in WoS-indexed journals and the number of citations provided by that database, meaning that what matters the most is not the content of the publications, but where they are indexed. Furthermore, the inbound citations ‘from journals’ not indexed in WoS are of no value for that evaluation committee.

Spanish law rewards the researchers for publishing in journals that are deemed ‘prestigious’ by WoS (Q₁, Q₂, Q₃) [21].

In Brazil, the evaluation of graduate programs relies heavily on JIFs. Journals and their papers are classified within seven categories with decreasing IF ranges (A₁, A₂, B₁, B₂, B₃, B₄, C), known as the ‘Qualis System’ [22]. In 2001, the Brazilian ministry of science and technology created a program to affiliate ten promising young doctorates to research centers. In the same year, the Brazilian national research council (CNPq) opened a two-year research fellowship call. The selection criterion for both programs was primarily based on the (i) number of publications and (ii) JIF [23]. After interviewing some postgraduate students about their concerns in academia, one typical statement was ‘*The adviser doesn’t care about my thesis so much. He believes that a thesis is the consequence of good work, and good work means papers published in good journals*’ [23].

Chinese policies offering financial reward based on WoS-indexed publications began in earnest in the 1990s, aiming to increase production and international visibility. In some Chinese and Swedish institutions, PhD students should publish at least two articles ‘with an average IF’ of four to get their degree [4].

In India, recruitment, awards, fellowships and promotions are determined by IF [21].

For the evaluation and ranking of researchers in Ecuador [24], their publications are categorized as follows: (i) “Level 1”: publications indexed by WoS or Scopus, Q1 or Q2 in JCR or Scimago Journal Ranking (SJR), (ii) “Level 2”: publications indexed by WoS or Scopus, Q3 or Q4 in JCR or SJR, (iii) “Level 3”: relevant works and indexed articles not covered by Levels 1 and 2. For faculty hiring [25] peer-reviewed papers count for 2 points per paper, and a maximum of 4 points, and, additionally, indexed papers count for 2 points per paper and a maximum of 10 points. For the overall ranking of universities in Ecuador, similar metrics are used. A group of 74 Ecuadorian academics already criticized this practice [26], bringing up the argument (amongst others) that only 242 Latin American journals (out of more than 5000) are indexed in WoS and Scopus, and those typically rank low. Moreover, the currently used metrics result in low ratings for researchers and institutions in the humanities and social science fields.

**3. Scientific community against the way research quality is assessed**

The 2012 San Francisco Declaration on Research Assessment (DORA) [27] was outlined by a group of journal editors and publishers at a meeting of the American Society for Cell Biology (ASCB) and its aim is to promote a world in which the content of a research article is not assessed by the IF of the journal where it appears. Up until now (September 26th, 2018) the declaration gathered 580 and 12779 online institutional and individual signatures, respectively. Momentum is being built, particularly in the UK, where recently all seven research councils announced their support [28].
Many authors agree that the IF is not a reliable instrument for measuring the quality of a scientist [29-31]. Anthony van Raan, former director of the Centre for Science and Technology Studies at Leiden University (The Netherlands), once said [32]: 'If there is one thing every bibliometrician agrees, is that you should never use the JIF to evaluate research performance for an article or an individual – that is a mortal sin'. Similar unequivocal opinions can be found from other bibliometricians [33]. A more honest measure of the quality of a paper may be derived from the number of times it is cited [3, 34-36].

Randy Schekman, the 2013 Nobel prize winner in medicine, said his lab would no longer send research to the top-tier journals Nature, Cell and Science, which in his opinion are distorting the scientific process and represent a 'tyranny' that must be broken [37]. Schekman criticizes those journals for artificially restricting the number of accepted papers, as well as the JIF as a marketing metric used by many prominent journals [37]. Besides Schekman, many other Nobel Laureates have spoken out (see Table 1), not directly against top-tier journal's decisions, but against the obsession on publishing in high-impact journals as a consequence of the biased scientific assessment policies taken by governments and institutions. Figures 1-7 partially illustrate the interviews given by seven Laureates on that topic.

<figure><img src="image_2.png" /><figcaption>**Figure 1.** Some of Michael Brown's answers on 'What do you think of the emphasis placed on IFs?' [38, 53].</figcaption></figure>

<caption>**Table 1.** Opinions of Nobel Laureates about the importance placed on JIFs.</caption>
<table><thead><tr><td>Nobel Prize Winner</td><td>Field, Year</td><td>Q & A</td><td>Facts and Video Refs</td></tr></thead><tbody><tr><td>Michael Brown</td><td>Medicine, 1985</td><td>Figure 1</td><td>[38, 39]</td></tr><tr><td>Joseph Goldstein</td><td>Medicine, 1985</td><td>Figure 2</td><td>[40, 41]</td></tr><tr><td>Peter Doherty</td><td>Medicine, 1996</td><td>Figure 3</td><td>[42, 43]</td></tr><tr><td>Paul Nurse</td><td>Medicine, 2001</td><td>Figure 4</td><td>[44, 45]</td></tr><tr><td>Martin Chalfie</td><td>Chemistry, 2008</td><td>Figure 5</td><td>[46, 47]</td></tr><tr><td>Bruce Beutler</td><td>Medicine, 2011</td><td>Figure 6</td><td>[48, 49]</td></tr><tr><td>Brian Kobilka</td><td>Chemistry, 2012</td><td>Figure 7</td><td>[50-52]</td></tr></tbody></table>

4. Novel metrics aiming to make the impact factor more valuable

Current 5-year IF formulae do not account for research published more than five years ago, regardless of how many citations it still receives or how cutting-edge it still remains. Clear examples are masterpieces still prominent today, such as Randall and Sundrum (1999) [54] or Li et al. (2009) [55], respectively with 10357 (370 in 2018) and 17113 (2747 in 2018) citations until September 26th, 2018 (source: Google Scholar). In another example [34] concerning the Molecular Pathology journal and the Journal of Clinical Pathology, which have a combined JIF, the total number of citations to their papers in the year 2000 was no less than 7144 (making them rank 7th among all pathology journals for the total number of citations). However, only 930 of those more than 7144 citations were to papers published in 1998-1999, meaning that no more than 13 per cent of citations was used to compute the 2000's JIF. Considering the former, aiming to balance the way research performance is measured, and trying to keep the current Impact Factor definitions useful, a new metric is suggested. The 'Complementary Impact Factor (CIF)' is proposed as the exact complement of the current JIF or 5-year IF. Therefore, it accounts only for citations to articles more than two or five years old, respectively. It is calculated as shown in Eq. (2) as the complement for the 2-year JIF and in Eq. (3) as the complement for the 5-year JIF.

$$ CIF_{2,year n} = \frac{\# \text{WoS citations in year } n \text{ to papers published in years } < (n-2)}{\# \text{ papers published in years } < (n-2)} \quad (2) $$

$$ CIF_{5,year n} = \frac{\# \text{WoS citations in year } n \text{ to papers published in years } < (n-5)}{\# \text{ papers published in years } < (n-5)} \quad (3) $$
<figure><img src="image_3.png" /><figcaption>**Figure 2.** Some of Joseph Goldstein's answers on 'Is it important to publish in high impact journals?' [40, 56].</figcaption></figure>

As an alternative to CIF and the current IF metrics, the authors would also like to propose a single novel impact factor named 'Timeless Impact Factor (TIF)', where the differences for the current IFs are that it accounts for citations (i) to all publications ever published in the journal, (ii) received until the end of the year preceding the calculation, and (iii) from any source (e.g., WoS, SCOPUS, others) – not just those from WoS-indexed journals (a minute proportion of the citing ones [3]). Moreover, it is proposed that each citation is weighted as function of its source, as proposed by Abambres and Arab [57]– 1.0 for WoS, 0.8 for SCOPUS, and 0.5 for others. The expression of the TIF then reads:

$$TIF_{year n} = \frac{\sum_{i=1}^{N_p} \sum_{j=1}^{n_i} w_{ij}}{N_p}, \quad (4)$$

where (i) $N_p$ is the total number of papers published in the journal before year $n$, (ii) $n_i$ is the number of papers citing paper $i$ before year $n$, and (iii) $w_{ij}$ is the weighted citation from citing article $j$ to cited article $i$ (1.0, 0.8 or 0.5 if the citing article is indexed in WoS, SCOPUS, or other databases, respectively). When the dispersion on citations needs to be studied, the metric using the Gini coefficient as proposed in [58] can be analyzed.
<figure><img src="image_4.png" /><figcaption>**Figure 3.** Some of Peter Doherty's answers on 'How important is a journal's impact factor?' [42, 59].</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 4.** Some of Paul Nurse's answers on 'How can you judge the quality of a researcher?' [45, 61].</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>**Figure 5.** Some of Martin Chalfie's answers on 'What do you think of Impact Factors?' [46, 62].</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 6.** Some of Bruce Beutler's answers on 'Is it important to publish in the top journals?' [49, 62].</figcaption></figure>
As a final remark, the authors suggest the withdrawal of all self-citations (citations from journal articles to the journal in which they are published) for the calculation of any type of impact factor. Not that it is necessarily unethical, but because it helps reducing the extent of 'fabricated IFs' [60]. It is believed that the previously proposed metrics could make the concept of impact factor far more useful, as an average measure of the journal quality.

<figure><img src="image_8.png" /><figcaption>**Figure 7.** Some of Brian Kobilka's answers on 'What do you think of IFs' [51, 52, 63].</figcaption></figure>

**5. Final Remarks**

This paper highlighted several viewpoints about the widespread misuse of the journal impact factor and indexing databases in the assessment of individual performance of scientists. The authors seriously disagree with this form of evaluation, as already did several Nobel Laureates who publicly spoke out against it. Since the concept inherent to impact factors is not useless, two novel metrics (Complementary and Timeless Impact Factors) are proposed in this work, aiming to contribute to a more sustainable academia.

**Author Contributions:** Conceptualization and first-draft writing: M.A.; All other contributions: equally distributed among all authors (AS's contribution was given before April 1st, 2018).
**Conflicts of Interest:** The authors declare no conflict of interest.

## References

1. Garfield, E., *The history and meaning of the journal impact factor*. JAMA, 2006. 295(1): p. 90-93.

2. Clarivate Analytics. *Journal Citation Reports*. 2018 [cited September 26th 2018]; Available from [URL](https://ClarivateAnalytics.com/journal-citation-reports)

3. Eston, R., *Editorial: The Impact Factor: a misleading and flawed measure of research quality*. Journal of Sports Sciences, 2005. 23(1): p. 1-3.

4. Quan, W., B. Chen, and F. Shu, *Publish or impoverish: An investigation of the monetary reward system of science in China* (1999-2016). Aslib Journal of Information Management, 2017. 69(5): p. 486-502.

5. Pudovkin, A.I., *Comments on the Use of the Journal Impact Factor for Assessing the Research Contributions of Individual Authors*. Frontiers in Research Metrics and Analytics, 2018. 3(2).

6. Ranjan, C.K., *Bibliometric Indices of Scientific Journals: Time to overcome the obsession and think beyond the Impact Factor*. Medical Journal Armed Forces India, 2017. 73(2), doi: [10.1016/j.mjafi.2017.03.008](https://doi.org/10.1016/j.mjafi.2017.03.008)

7. Robert, R. and S. Gnanavel, *Impact factor: The holy grail of research*. Indian Journal of Psychological Medicine, 2015. 37(2): p. 248-249.

8. Vinkler, P., *Introducing the Current Contribution Index for characterizing the recent, relevant impact of journals*. Scientometrics, 2009. 79(2): p. 409-420.

9. Seglen, P.O., *Why the impact factor of journals should not be used for evaluating research*. BMJ, 1997. 314(7079): p. 497.

10. Anonymous, *Coping with peer rejection*. Nature, 2003. 425, doi: [10.1038/425645a](https://doi.org/10.1038/425645a)

11. North, G., Q & A. Current Biology, 2004. 14(4): p. R143-R144.

12. Brembs, B., *Prestigious Science Journals Struggle to Reach Even Average Reliability*. Frontiers in Human Neuroscience, 2018. 12(37).

13. Abbasi, K., *Let's dump impact factors*. BMJ, 2004. 329(7471).

14. Smeyers, P. and N.C. Burbules, *How to Improve your Impact Factor: Questioning the Quantification of Academic Quality*. Journal of Philosophy of Education, 2011. 45(1): p. 1-17.

15. CLUSTER. *Members – CLUSTER*. 2018 [cited 2018 October 4th 2018]; Available from: [https://cluster.org/about-us/member-universities/](https://cluster.org/about-us/member-universities/)

16. University of Lisbon (UL), *Reitoria – Despacho n.º 3855/2017 (in Portuguese)*. Diário da República, 2017. 88(2.ª série – 8 de maio de 2017): p. 8628-29, Retrieved from [URL](https://www.re publica.pt/en/2017/05/07/despa-cho-n-3855-2017-in-portuguese)

17. University of Lisbon (UL), *Edital 655/2017 (in Portuguese)*. Diário da República, 2017. 173(2.ª série – 7 de setembro de 2017): p. 19764-7, Retrieved from [URL](https://www.republica.pt/en/2017/05/07/edital-655-2017-in-portuguese)

18. University of Lisbon (UL), *Edital 688/2017 (in Portuguese)*. Diário da República, 2017. 178(2.ª série – 14 de setembro de 2017): p. 20201-3, Retrieved from [URL](https://www.republica.pt/en/2017/05/07/edital-688-2017-in-portuguese)

19. University of Lisbon (UL), *Edital 662/2017 (in Portuguese)*. Diário da República, 2017. 174(2.ª série – 8 de setembro de 2017): p. 19856-8, Retrieved from [URL](https://www.republica.pt/en/2017/05/07/edital-662-2017-in-portuguese)

20. Ostravská univerzita v Ostravě. *Open position for a postdoctoral fellow*. 2018 [cited 2018 September 19, 2018]; Available from [URL](https://www.oxfordjournals.org/doi/abs/10.1093/oxj/abu003)

21. Muthamilarasan, M. and M. Prasad, *Impact of impact factor in quantifying the quality of scientific research*. Current Science, 2014. 107(8): p. 1233-1234.
22. Ferreira, R., F. Antoneli, and M. Briones, *The hidden factors in impact factors: a perspective from Brazilian science*. Frontiers in Genetics, 2013. 4(130).

23. de Meis, L., M.S. do Carmo, and C. de Meis, *Impact factors: just part of a research treadmill*. Nature, 2003. 424: p. 723.

24. UMET Universidad Metropolitana. *Senescyt establece lineamientos para categorización de investigadores (in Spanish)*. 2018 [cited October 12th, 2018]; Available from [URL](https://www.umet.edu.ec/index.html).

25. Universidad Tecnica Particular de Loja. *Fase de méritos*. 2018 [cited October 12, 2018]; Available from [URL](https://www.utec-loja.edu.ec/index.html).

26. Universidad y Sociedad: Foro Ecuador. *El model de evaluacion de las universidades Ecuatorianas - Apuntes criticos para el debate*. 2014 [cited October 12, 2018]; Available from [URL](https://www.utecequador.org/index.html).

27. American Society for Cell Biology. *The San Francisco Declaration on Research Assessment (DORA)*. 2018 [cited 2018 September 13th 2018]; Available from: [https://sfdora.org/](https://sfdora.org/).

28. Curry, S., *Let's move beyond the rhetoric: it's time to change how we judge research*. Nature, 2018. 554(February): p. 147.

29. Hallberg, L., *Can the impact factor measure the quality of research?* International Journal of Qualitative Studies on Health and Well-being, 2012. 7(1): p. 19772.

30. Campbell, P., *Escape from the impact factor*. Ethics in Science and Environmental Politics, 2008. 8(1): p. 5-7.

31. Tregoning, J., *How will you judge me if not by impact factor?* Nature, 2018. 558(June): p. 345.

32. van Noorden, R., *Metrics: A profusion of measures*. nature, 2010. 465(June): p. 864-866.

33. Glänzel, W., et al., *Springer Handbook of Science and Technology Indicators*. 2018, Cham, Switzerland: Springer International Publishing.

34. Van Diest, P.J., et al., *Impactitis: new cures for an old disease*. Journal of Clinical Pathology, 2001. 54(11): p. 817-819.

35. Norris, M. and C. Oppenheim, *Citation counts and the Research Assessment Exercise V: Archaeology and the 2001 RAE*. Journal of Documentation, 2003. 59(6): p. 709-730.

36. Oppenheim, C., *Do citations count? Citation indexing and the Research Assessment Exercise (RAE)*. Serials, 1996. 9(2): p. 155-161.

37. Sample, I. *Nobel winner declares boycott of top science journals* 2013 [cited September 20th 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2013-nobel-prize-in-science-and-technology-for-top-science-journals).

38. Nobel Media AB. *Impact factors are 'a substitute for judgement' – Nobel Laureate Michael Brown (YouTube video)*. 2011 [cited September 21, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2011-nobel-prize-in-science-and-technology-for-top-science-journals).

39. Nobel Media AB. *Michael S. Brown – Facts* 2018 [cited September 17, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2018-michael-s-brown-facts).

40. Nobel Media AB. *Impact factors? What counts is the data – Nobel Laureate Joseph Goldstein (YouTube video)*. 2014 [cited September 21, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2014-impact-factors-what-counts-is-the-data-joseph-goldstein).

41. Nobel Media AB. *Joseph L. Goldstein – Facts* 2018 [cited September 17, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2018-joseph-l-goldstein-facts).

42. Nobel Media AB. *Nobel Laureate Peter Doherty: Impact factors are skewing science (YouTube video)*. 2012 [cited September 21, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2012-nobel-laureate-peter-doherty-impact-factors-are-skewing-science).

43. Nobel Media AB. *Peter C. Doherty – Facts* 2018 [cited September 17, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2018-peter-c-doherty-facts).

44. Nobel Media AB. *Sir Paul M. Nurse – Facts* 2018 [cited September 17, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2018-sir-paul-m-nurse-facts).

45. Nobel Media AB. *There's a lot of rubbish published in high-profile journals' Paul Nurse, Nobel Laureate (YouTube video)*. 2014 [cited September 21, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2014-there-is-a-lot-of-rubbish-in-high-profile-journals-paul-nurse-nobel-laureate).

46. Nobel Media AB. *'I can categorically say I hate impact factors!' Nobel Laureate Martin Chalfie (YouTube video)*. 2014 [cited September 21, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2014-impact-factors-i-hate-impact-factors-martin-chalfie).

47. Nobel Media AB. *Martin Chalfie – Facts* 2018 [cited September 17, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2018-martin-chalfie-facts).

48. Nobel Media AB. *Bruce A. Beutler – Facts* 2018 [cited September 17, 2018]; Available from [URL](https://www.nobelprize.org/en/press/press-announcements/2018-bruce-a-beutler-facts).
49. Nobel Media AB. *The impact factor really isn't that important' Bruce Beutler, Nobel Laureate (YouTube video)*. 2015 [cited September 21, 2018]; Available from [URL](https://www.nobelprize.org/press/press_knowledge/press_knowledge_2015_09_01_00000000000000000000000000000000.html) (archived at [URL](https://web Archive.WW祈安网/20150901000000000000000000000000.html))

50. Nobel Media AB. *Brian K. Kobilka - Facts* 2018 [cited September 17, 2018]; Available from [URL](https://www.nobelprize.org/press/press_knowledge/press_knowledge_2017_09_01_00000000000000000000000000000000.html). © 2018 by Abambres et al. (CC BY 4.0)

51. Nobel Media AB. 'I am very troubled by the importance of impact factors' Brian Kobilka, Nobel Laureate (Youtube video). 2016 [cited September 21, 2018]; Available from [URL](https://www.nobelprize.org/press/press_knowledge/press_knowledge_2016_09_01_00000000000000000000000000000000.html) (archived at [URL](https://web Archive.WW祈安网/20160901000000000000000000000000.html))

52. Nobel Media AB. *Impact factors are taking us in the wrong direction* - Brian Kobilka, Nobel Laureate (Youtube video). 2016 [cited September 21, 2018]; Available from [URL](https://www.nobelprize.org/press/press_knowledge/press_knowledge_2016_09_01_00000000000000000000000000000000.html) (archived at [URL](https://web Archive.WW祈安网/20160901000000000000000000000000.html))

53. Kisspng.com. *Cartoon Scientist Illustration* – Cartoon scientist material – Unlimited Download. 2018 [cited September 22, 2018]; Available from [URL](https://www.kisspng.com/cartoon-scientist/). © 2018 by Abambres et al. (CC BY 4.0)

54. Randall, L. and R. Sundrum, *Large Mass Hierarchy from a Small Extra Dimension*. Physical Review Letters, 1999. 83(17): p. 3370-3373.

55. Li, H., et al., *The Sequence Alignment/Map format and SAMtools*. Bioinformatics, 2009. 25(16): p. 2078-2079.

56. Kisspng.com. *Assistant Professor Essay Saint Joseph's College, Colombo Thesis statement* – doktor cartoon - Unlimited Download. 2018 [cited September 22, 2018]; Available from [URL](https://www.kisspng.com/). © 2018 by Abambres M et al. (CC BY 4.0)

57. Abambres, M. and P. Arab, *Citation Indexes Accounting for Authorship Order in Coauthored Research—Review and New Proposal*. Science & Technology Libraries, 2016. 35(4): p. 263-278, doi: [10.1080/0194262X.2016.1242450](https://doi.org/10.1080/0194262X.2016.1242450). © 2018 by Abambres et al. (CC BY 4.0)

58. Cockriel, W.M. and J.B. McDonald, *The influence of dispersion on journal impact measures*. Scientometrics, 2018. 116(1): p. 609-622.

59. Kisspng.com. *Black Nerd Geek Female Person of color* – *others* – Unlimited Download. 2018 [cited September 22, 2018]; Available from [URL](https://www.kisspng.com/). © 2018 by Abambres M et al. (CC BY 4.0)

60. Davis, P. *Impact Factor Denied to 20 Journals For Self-Citation, Stacking* 2018 [cited September 29th 2018]; Available from [URL](https://www.davispress.org/). © 2018 by Abambres M et al. (CC BY 4.0)

61. Kisspng.com. *Mad scientist Science Laboratory Despicable Me* – *scientist* – Unlimited Download. 2018 [cited September 22, 2018]; Available from [URL](https://www.kisspng.com/). © 2018 by Abambres M et al. (CC BY 4.0)

62. Kisspng.com. *Nerd Glasses Geek Clip art* – *nerd* – Unlimited download. 2018 [cited September 22, 2018]; Available from [URL](https://www.kisspng.com/). © 2018 by Abambres M et al. (CC BY 4.0)

63. Kisspng.com. *Professor Utonium Scientist Science Cartoon* – *Mad Scientist* – Unlimited Download. 2018 [cited September 22, 2018]; Available from [URL](https://www.kisspng.com/). © 2018 by Abambres M et al. (CC BY 4.0)

<figure><img src="image_9.png" /></figure>