Quantifying the evolution of individual scientific impact
Authors: Albert-László Barabási
Chaoming Song
Dashun Wang
Roberta Sinatra
Pierre Deville
Journal: Science (2016)
Discipline: Multidisciplinary
Keywords: Quantitative indicators; professional impact; scientist; impact; productivity
Affiliations: Center for Complex Network Research and Physics Department, Northeastern University, Boston, MA 02115, USA., Center for Network Science and Math Department, Central European University, Budapest, Hungary., Kellogg School of Management, Northwestern University, Evanston, IL 60208, USA., Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL 60208, USA., Department of Applied Mathematics, Université catholique de Louvain, Louvain-la-Neuve, Belgium., Department of Physics, University of Miami, Coral Gables, FL 33124, USA., Center for Complex Network Research and Physics Department, Northeastern University, Boston, MA 02115, USA. ., Department of Medicine, Brigham and Women's Hospital, Harvard Medical School, Boston, MA 02115, USA., Center for Network Science, Central European University, Budapest, Hungary., Center for Cancer Systems Biology, Dana-Farber Cancer Institute, Boston, MA 02115, USA.

Abstract:  Despite the frequent use of numerous quantitative indicators to gauge the professional impact of a scientist, little is known about how scientific impact emerges and evolves in time. Here, we quantify the changes in impact and productivity throughout a career in science, finding that impact, as measured by influential publications, is distributed randomly within a scientist's sequence of publications. This random-impact rule allows us to formulate a stochastic model that uncouples the effects of productivity, individual ability, and luck and unveils the existence of universal patterns governing the emergence of scientific success. The model assigns a unique individual parameter Q to each scientist, which is stable during a career, and it accurately predicts the evolution of a scientist's impact, from the h-index to cumulative citations, and independent recognitions, such as prizes. 

Retracted: No

PubPeer comments:
null, 
 The paper exaggerates the predictive power of their citation-based Q-factor on getting a Nobel Prize. ![file] The Receiver Operating Characteristic (ROC) in Figure 6(A) of [1] (reproduced above) is a misleading way to represent the data. When you look at the plot, it may seem that the classifier works well. At certain rank threshold every Nobel Prize winner in the sample gets in the sieve. The True Positive Rate (TPR) is 100%. The False Positive Rate (FPR) at this rank threshold is merely 25%. But wait a second, Nobel winners are a tiny fraction of the scientists. So, there should be a lot more of them in the sieve even with the 25% FPR. To proceed we need to know n_N, the number of Nobel winners, and n_O, the number of other scientists in the sample. I could not find these numbers in Figure captions in [1]. I also could not get them from the authors. So, I had to extract the data from the plots using a plot digitizer. By matching the data from Fig. 6A and Fig. S45 of [1], I could compute the wanted numbers. My estimate is n_N = 25 and n_O u2248 2900. This means that at the rank threshold R u2248 760 when TPR = 1 and FPR u2248 0.25 we get 25 Nobel winners and over 700 other scientists in the sieve. The classifier is poor. Note that if, for example, we had n_O = 28 and the same ROC plot then at TPR = 1 and FPR = 0.25 we would get 25 Nobel winners and 7 other scientists in the sieve. This means that the ROC plot is not the best way to look at the data since it does not separate these drastically different situations. Fig. S45 in [1] has the plot of the precision (the ratio of the number of Nobel winners in the sieve to the total number of the scientists in the sieve) as a function of rank threshold. For R = 1 the precision is 0, meaning that the scientist with the highest Q-factor did not get a Nobel Prize. The next data point is at R=11 where the precision is 0.27. This means that out of 11 highest Q-factor scientists 3 are Nobel winners. It seems strange that the second data point is at the odd value of 11 rather than at a natural value of 10. My guess is that this is because precision reaches maximum at 11. This would imply that the second and third highest-Q scientists are not Nobel winners, for otherwise the maximum of precision would be at one of those thresholds. With all that said Fig. S45 from Supplementary Information is much more useful than the Figure 6(A) in the main text for it clearly shows that the precision of the classifier is low. An alternative way to analyze the data is to compute a correlation coefficient. To each scientist we can attribute a Q-number which is 1 if they get in the sieve at given rank threshold and 0 otherwise. And an N-number which is 1 if the scientist got a Nobel Prize and 0 otherwise. We can interpret the N-number as a rating on a 2-point scale. Now we can compute a correlation coefficient, r, between these vectors. You can find the equation for r in [2]. For the natural rank threshold R = n_N = 25 we get r u2248 0.19 (we would get r = 1 if the classifier was perfect). This is on par with low correlation coefficients between popularity of books and movies and their quality ratings reported in Figure S2 of [3]. ![file] I also would like to comment on another issue. The blue curve (R-model) in Figure 3 (D) of [1] (reproduced above) is wrong. In the Q-model a higher citation rate for the bulk of the work implies a higher Q and thus a higher citation rate of the highest cited paper compare to the R-model. Thus, the red curve (Q-model) should be above the blue curve (R-model) at the high average impact end. By a similar argument, the red curve should be below the blue curve at the low average impact end. When we look how they actually compute the R-model curve in Eq.(S9) we see that they average citations for the bulk of the work for fixed number of citations to the highest cited paper. They should have fixed the number of citations per paper for the bulk of the work and compute the average number of citations to the highest cited paper. I conclude that the ROC plot in Figure 6(A) of [1] over-represents the accuracy of the classifier. It is not the right way to look at the data when the ratio of positives and negatives in the sample is so small. The correlation between the Q-factor and Nobel prize winning is small. There are a couple of comments on the article in question in PubMed. Vincent Detours wrote about  scientifically unremarkable managers dominating science. Donald Forsdyke wrote about  the triumph of marketing over ability. These comments have some relation to reality. 1. Sinatra, R., Wang, D., Deville, P., Song, C., Barabasi, A.-L. (2016) Quantifying the evolution of individual scientific impact. Science 354(6312), 5239 2. Simkin, M.V. (2020) Success in creative careers depends little on product quality, thttps: arxiv.org abs 2005.07349 3. Janosov, M., Battiston, F. Sinatra, R. (2020) Success and luck in creative careers. EPJ Data Sci. 9, 9. https: doi.org 10.1140 epjds s13688-020-00227-w ,