Fitting Lotka's law: Some cautionary observations on a recent paper by Newby et al. (2003)

I believe that a recent paper in which Newby et al. (2003) explore the parallel between software development and publishing, with programmers corresponding to authors, is seriously flawed. I shall just consider the analysis of their first data set in detail; similar comments apply to the others. These particular data were obtained from the metadata repository Linux Software Map (LSM). Full details are given in the original paper. The first three columns of Table 1 give the observed productivity distribution of authors across LSM entries together with “predicted” values for a Lotka distribution of index 2.82 as reported by Newby et al. (2003), which the authors claim to give the best fit as measured by total squared prediction error. Note that productivities greater than 10 were not given so we must omit them in what follows just as they were eliminated from the original analysis. (This will lead to slight inaccuracies but will not be of major concern.) The basis for the calculation of the predicted numbers is incorrect. When comparing an empirical distribution with a fitted theoretical distribution, the predicted—or expected—frequencies are given by the simple binomial formula where N is the total number of observations, here 3332, and pr is the probability of productivity r according to the fitted model. This ensures that the sums of observed and expected frequencies are the same. In particular, for a fitted Lotka distribution of order n, we have where c(n) is the normalizing constant. Alternatively, we can write which corresponds to Equation (1) of Newby et al. (2003). However, in the third column of Table 1 we find that they have actually used where O1 is the observed frequency of single-producers. If we correct for this by standardizing so that the total expected number is 3332, rather than 3379, we get the fourth colEr O1 r , Er E1 r , Er Nc(n) r , Er Npr umn of Table 1. The problem now is that there is no guarantee that this fitted distribution is the best one according to the original criterion of minimizing total squared prediction error, since the normalizing constant c(n) has been ignored in the original minimization procedure. To illustrate this, in column five of Table 1 we give the standardized expected values for a Lotka distribution of order 2.9. The total squared error for n 2.9 is 1205.4, whereas that for n 2.82, supposedly the best fit, is 2142.5. The point is that, having standardized the fitted distributions, we are necessarily minimizing within a different set of distributions, or at least adopting a slightly different minimization criterion. One reason why n 2.9 works so well—but we are not claiming it to be the “best”! —is that it makes the expected value for the single-producers very close to the observed value and it is this category that dominates the distribution. Indeed, a very small proportionate difference between the observed and expected differences for this category can lead to a very large squared absolute difference. Indeed, looking at the sum of squares of absolute differences to assess goodness-of-fit is not a good idea—the procedure does not allow for the different category sizes. To allow for different category sizes, the usual procedure would be to consider not the simple sum of squares of absolute differences (Or Er), but one that is appropriately weighted. The natural weighting to allow for different sizes is (Or Er) Er which is of course the well known x2 statistic. However, there does not seem to be an easily implemented procedure for minimizing the x2-statistic for a Lotka distribution. Indeed, it is the presence of the normalizing constant c(n), related to the Riemann zeta function, which makes the quest for a “best-fitting” Lotka distribution difficult by whatever criterion is adopted. It would seem that the best straightforward fitting method currently available is the program given by Rousseau and Rousseau (2000) for estimation of the Lotka index by the maximum likelihood (ML) method. For purposes of comparison we have included the standardized fitted distribution using the ML estimate n 3.053 in column five of Table 1.