What do altmetrics counts mean? A plea for content analyses

Sir, Altmetrics is one of the hot topics in scientometrics. There are several reasons for this: (a) Science policy wants to measure the broad impact of science, and altmetrics may be an option. (b) Altmetrics allows an impact measurement shortly after the publication of a paper. Traditional citations can be used to measure the impact of publications that were published at least 3 years before. (c) There are large data sets available with counts of tweets, mentions in blogs, number of bookmarks, etc., which can be empirically analyzed with regard to the impact of publications. Most of the studies that have investigated the meaning of altmetrics calculated correlations between altmetrics and traditional citations. As the results of Bornmann (2015) show, “the correlation with traditional citations for micro-blogging counts is negligible (pooled r = 0.003), for blog counts it is small (pooled r = 0.12) and for bookmark counts from online reference managers, medium to large (CiteULike pooled r = 0.23; Mendeley pooled r = 0.51)” (p. 1123). However, the correlations are difficult to interpret. An absent correlation of micro-blogging counts (Twitter counts) with citations could mean that Twitter counts measure something different from citations. An alternative explanation is that Twitter counts have little value. They could be generated completely in a process without any relation to quality aspects and this would result in an absent correlation with citations. Because correlation analyses do not really reveal the meaning of altmetrics, I would like to plead for content analyses of tweets and blogs (for bookmarks, content analyses are not possible). I could only find the study of Thelwall, Tsou, Weingart, Holmberg, and Haustein (2013) investigating a small sample of n = 270 tweets with content analysis techniques. As the literature overview of Bornmann and Daniel (2008) shows, content analyses of traditional citations could reveal interesting insights into the process of citing. Studies have worked out different types of citations: affirmational, assumptive, conceptual, contrastive, methodological, negational, perfunctory, and persuasive. For example, the results of content analyses showed that citations of the negational type (citing work disputes some aspects of cited work; citing work corrects/questions cited work; citing work negatively evaluates cited work) are less frequent than citations of the other types. Hence, these results could counteract scruples about the view of citations as a measure reflecting the intellectual and cognitive impact of research. As an example of a content analysis, I investigated tweets citing a paper by Hirsch (2005). In that paper, the h-index was introduced. On August 4, 2015, I used Scopus (Elsevier) to analyze the tweets of the paper for the time period June 17, 2011 to June 16, 2015. Earlier and later data were not available in Scopus at that time. With 698 tweets and 1 Facebook post, the paper is in the 99th percentile compared to other papers of the same age and document type in the multidisciplinary subject category (as displayed by Scopus for this paper). That means the paper belongs to the 1% most mentioned papers compared to similar papers. I could not include 19 tweets (28%) in the content analysis, because they were written in Ukrainian, Catalan, Spanish, or Japanese. Of the total of 49 tweets, 35 (71%) were simple retweets (repetitions of tweets from other people); 6 (12%) tweets only mention the paper of Hirsch (2005) as follows: “An index to quantify an individual’s scientific research output,” “Hirsch (2005) An index to quantify an individual’s scientific research output,” or “h-INDEX: This is the PNAS paper that started everything. ” In 2 tweets (4%) personal h-index values are reported (“Finally made it to H = 30 . . . Can I retire now? ” and “I said I would go up for Full Professor at an h-index of 15. I just hit 14 today! Getting closer”). Further, 2 tweets (4%) mention the paper of Hirsch (2005) because it is used in two other studies on new metrics. Most of the retweets refer to these initial two tweets. Only 4 tweets (8%) discuss the paper in more detail: “The h index instead of impact factor? ”, “An analysis that checks Hirsch’s claim that self-cites aren’t a big deal would be good,” “Or, similarly, correcting for network effects (e.g., cites from past co-authors self-citing),” and “I wonder if Hirsch realized the monster he would unleash when he put this on Arxiv back in 2005. ” If we categorize the tweets relating to the paper of Hirsch (2005) to the different types of citations (see above), then 83% (71% + 12%) of the tweets are perfunctory, 8% (4% + 4%) are methodological, and 8% are affirmational or assumptive, respectively (that means citing work is strongly influenced by the paper). In addition, the analysis of tweets around the paper of Hirsch (2005) shows that obliteration by incorporation (Merton, 1968) is not only relevant to traditional citations but also to tweets: I found many tweets that mentioned the h-index, but did not cite Hirsch (2005).