Response to Letter from Löser et al.

Although the use of citations to develop data sets for analysis has a lengthy history, Loser et al. (2008)xLoser, P., Guhr, A., Kurtz, A., and Wobus, A.M. Cell Stem Cell. 2008; 3: 129–130Abstract | Full Text | Full Text PDF | PubMed | Scopus (5)See all ReferencesLoser et al. (2008) are correct to point out—as I acknowledged—that these data sets are inherently noisy. In this case, I made a deliberate choice to use citation linkages to define my data sets because of the opportunity these linkages offered to use an identical methodology to develop data sets not just for hESC- and RNAi-related research but also for research related to 50 other influential biomedical research articles. In my view, this ability to make comparisons between fields and countries outweighed the loss of precision present in hand-collated data sets in previous studies (Guhr et al., 2006xGuhr, A., Kurtz, A., Friedgen, K., and Loser, P. Stem Cells. 2006; 24: 2187–2191Crossref | PubMed | Scopus (66)See all References, Owen-Smith and McCormick, 2006xOwen-Smith, J. and McCormick, J. Nat. Biotechnol. 2006; 24: 391–392Crossref | PubMed | Scopus (38)See all References).I did not claim, nor intend to imply, that my data set consisted entirely of experimental work involving hESCs. Rather, throughout my paper, and indeed in its title, I referred to my data sets as research related to hESCs and RNAi, and I clearly articulated the importance of this phrasing. This distinction is not mere semantics. Many of the non-hESC articles included in my hESC-related data set report primary research using embryonic stem cells in animal models or other human stem cells. I do not consider these articles false positives. Many important advances in hESC science have been based on just this sort of related research. Furthermore, scientists starting hESC research programs may opt to work with mouse cells initially and transition to hESCs over time. Although not all articles in related fields cite the initial hESC article, I believe those that do are disproportionately likely to report relevant work. My data set also includes some review material that summarizes existing research or introduces hESC science to related fields. Though not primary research, these articles serve an important scientific purpose, clearly related to hESC science. I do agree that articles on the ethical and legal implications of hESC research are false positives. Due to time limitations, I chose not to manually review each of ∼22,000 articles in my three data sets to systematically remove these articles. A preliminary search of the titles of articles in my hESC-related set for keywords relevant to ethical, legal, and policy issues identified 24 papers (∼2%) divided among nine countries. Excluding these articles has essentially no impact on my identification of over- and underperforming countries in research related to hESCs.Loser et al.'s letter highlights a point that should have, in retrospect, been drawn out more clearly. The inclusion of research related to, but not explicitly using, hESCs into my data set reduced the magnitude of the signal my study aimed to detect because the distribution of this related research is more likely to mirror the control set. The conservative nature of this signal means my approach will tend to underestimate the magnitude of underperformance in countries, such as Germany, France, and Japan, that have relatively strong developmental biology research communities.If anything, given the different methodologies, I interpret Loser et al.'s analysis as confirmatory rather than contradictory. Indeed, if their data on country shares of experimental hESC research are compared to my biomedical control set, the set of seven overperforming countries are the same, as are four of the top five underperforming countries (the only change is the substitution of Germany for Switzerland). Thus, although the order of the countries and the magnitude of the under- and overperformance differs, both analyses paint broadly similar pictures of the development of this field.Incorporating measures of research quality would be an interesting extension to my analysis of country-level research output. However, because citation distributions are skewed to the point that a journal's impact factor is not particularly representative of any individual article (Garfield, 2006xGarfield, E. JAMA. 2006; 295: 90–93Crossref | PubMed | Scopus (839)See all References, Seglen, 1997xSeglen, P.O. BMJ. 1997; 314: 498–502Crossref | PubMedSee all References), country-level weighted average journal impact factors may not be an appropriate strategy to accomplish this task. Furthermore, the journal selection and publication process is influenced by social as well as scientific factors, particularly given the single-blind review system common in the life sciences. Loser et al. should not discount the possibility that the differences they find reflect, in part, the social structure of science that may make it easier for western and English-speaking scientists to gain access to high-impact journals (Link, 1998xLink, A.M. JAMA. 1998; 280: 246–247Crossref | PubMed | Scopus (148)See all References, Ross et al., 2006xRoss, J.S., Gross, C.P., Desai, M.M., Hong, Y., Grant, A.O., Daniels, S.R., Hachinski, V.C., Gibbons, R.J., Gardner, T.J., and Krumholz, H.M. JAMA. 2006; 295: 1675–1680Crossref | PubMed | Scopus (70)See all References).I commend Loser et al. for their efforts developing a verified data set of confirmed experimental hESC articles—a hugely valuable resource. Policy analysis is most effective when it builds upon multiple data sources, and I believe both approaches can contribute to the growing understanding of the influence of public policy on the development of hESC and related research.