Georgia Institute of Technology
From the SelectedWorks of Diana Hicks

2023

Seeing impact: genres referencing journal articles
Diana Hicks

Available at: https://works.bepress.com/diana_hicks/58/

Seeing Impact: genres referencing journal
articlesA prior version of this manuscript was published as:Hicks, Diana (2023). “Seeing impact: genres referencing journal articles”. Profesional de la información, v. 32,
n. 2, e320212. https://doi.org/10.3145/epi.2023.mar.12
Nota: Este artículo se puede leer en español en:
https://revista.profesionaldelainformacion.com/index.php/EPI/article/view/XXXXX

Diana Hicks
School of Public Policy
Georgia Institute of Technology
dhicks@gatech.edu
version 2: August, 2023
version 1 was posted April 2023

Abstract
This paper examines the societal impact of research from the perspective of interconnected genres.
Information reaches professionals outside academia through many different types of documents. Those
documents often connect with scholarship by referencing academic work, mentioning professors, or
publishing articles authored by scholars. Here patterns of referencing journal articles are comparedacross professional genres. Such citation counts make visible societal impacts to the extent that a field
engages a genre, and different genres favor different fields. Biomedical sciences are most visible inpatent citation counts. News and social media most often reference medicine. Policy documents makeheavy use of social science. Ulrich's indexing of trade journals, magazines, and newspapers suggests
social sciences engage heavily with the professions through trade press. However, caution is warranted
when using citations to indicate societal impact. Engagement with scholarship occurs not only throughreferencing but also through authorship and mentions. Not all citations indicate substantive
engagement, particularly in social media. Academic literature is but one of many types of sourcesreferenced in professional genres. And scholarship engages with many genres beyond those currently
indexed, most notably trade press. Nevertheless, understanding citation patterns across heterogeneous
professional genres offers a promising frontier for information sciences to provide a foundation for the
analysis of scholarship’s societal impact.

Keywords
Societal impact, patents, Twitter, Facebook, newspapers, genre, policy, Overton, PlumX, citations

1

Introduction
Increasingly, researchers, universities, and funders are interested not just in the scholarly impact of
research but also the broader societal impact. Bibliometricians have responded by analyzing how often
research articles are cited in genres other than journal articles. Mendeley uploads and Tweets linking tojournal articles have been counted. References to journal articles have been counted in: trade press(Van Steijn andRip, 1988; Nederhof and Meijer, 1995), blogs (Bornmann, 2015), newspapers (Begum etal., 2016), regulatory impact analyses (Desmarais and Hird, 2014), policy documents (Bornmann et al.,2016; Pinheiro et al., 2021; Szomszor & Adie, 2022; Vilkins & Grant, 2017), clinical trials documents(Thelwall and Kousha, 2016) a drug information database (Thelwall et al. 2017) and clinical practiceguidelines (Grant et al. 2000; Kryl et al. 2012; Lewison & Sullivan, 2008; Thelwall and Maflahi, 2016).
Counts of references in these documents are used to signify broader societal interest in research output.
These analyses tend to examine one type of source, or genre, using one database and explore
constructing indicators of broader impact using that source. The number of genres explored acrossthese studies suggests a bigger picture awaits exploration. Shifting focus from the counts to the genres
highlights the many different types of documents through which information reaches professionals
outside academia. Those documents often engage with scholarship and record that engagement usingreferences. This paper examines the pattern of referencing across genres as well as evidence of thecomplexity of genre intertextuality. That is, I examine the societal impact of research from theperspective of interconnected genres. Professional information genres are the frame of reference, andtheir interconnection with scholarship is the phenomenon of interest.

BackgroundSources of information have proliferated over the past century, with ever more scientific journals being
published, and ever more newspapers, magazines, and patents appearing. The advent of digitization
about twenty years ago accelerated expansion. The internet challenged existing periodicals to adapt
and build a digital presence. As well, new sources and new genres multiplied. In the dental trade press,
print magazines were joined by digital forums, commercial news websites, news aggregators, andindependent bloggers (Hicks, Isett & Melkers, 2019). Facebook and Twitter were established andbecame another way for professionals to share technical information with the added possibility of
conversation and engagement with a broader audience. Ulrich's indexes trade press journals andmagazines, and Figure 1 displays the number of trade press journals, newspapers, and magazines
established in each decade since the 1960s. The trade literature expanded every decade, with thestrongest growth in the 2000s, more than double the growth rate in the 1970s and 2010s, suggesting
digitization took off during those years.

2Figure 1 Number of US trade journals established in recent decades

Source: Ulrich’s web 1
The internet has revolutionized the accessibility of every genre. Of course, Twitter and Facebook were
established as platforms to post content accessible to all. Subscription trade press and newspaper
articles are now findable and often readable one at a time without paying. Ad-supported news sites,
both trade press and mass media, are open to all. The patent database used to be accessible tospecialists who acquired physical copies of the tapes containing the database. Now the patent databaseis online, searchable through Google, and readable by everyone. The National Academies removed the
paywall from their reports in 2011, and now about half of report use traces to the general public, i.e.
those outside teaching and university research (Hicks et al., 2022). Governments and think tanks post
their policy reports online for anyone to read for free. The Overton database of policy documentsshows strong growth in the number of documents indexed over time likely because digitization has
made policy documents much more accessible 2. The expansion in numbers and genres of professional
information sources and the reduction in barriers to accessing them has revolutionized professional
information systems, which are much more complex, accessible, and used than even 20 years ago.
Professional information genres are not self-contained. Referencing is primarily associated with journal
articles; indeed, almost all scholarly journal articles contain references. Most links between documents
are to other documents of the same genre. Thus, journal articles primarily reference other journal
articles, and patents primarily reference other patents. Therefore, most analytical attention is devoted
to characterizing the networks these links establish between documents. However, referencing is foundthroughout professional genres. Though not all items contain references, some do, and among those,some contain references to scholarly journal articles.
Therefore, the proliferation of professional information genres in recent decades offers ever-expanding
opportunities to analyze the links between documents, though analysis is challenging in the absence ofcomprehensive indexes. Of course, well-established, high-quality databases - Scopus, Web of Science,
Dimensions, and PubMed -track the expanding scholarly literature. And patents have always been
1

Search for: Status:("Active") Serial Type:("Journal" "Magazine" "Newspaper") Content Type:("Trade") Language of
Text:("English") Format:("Print" "Online") Country of Publication:("United States")

Therefore, it is impossible to discern the contribution to perceived growth originating in the number of reports
produced versus the increased posting of reports on the internet (Szomszor & Adie, 2022).

2

3

indexed, so our understanding of the impact of scholarship is heavily shaped by the analysis of patent
referencing. The expanding digitization underpinning the proliferation of genres also enables indexing.
Databases such as Altmetrics, PlumX, and Overton were founded to index references in social media andpolicy documents, and the picture they provide of research impact differs. The indexing of references
to journal articles originating in other media opens up analytical opportunities to identify the research ofmost interest to different audiences. Taking advantage of these indexes, this paper compares
referencing to journal articles in trade press, policy reports, news, blogs, Twitter, and patents. These
patterns are compared with two other indicators of public interest in research, downloads of journal
articles and UK REF impact cases.

Genres referencing journal articles
Many genres reference research, but genres’ goals differ, and therefore they reference for different
reasons. Genres also differ in their pattern of referencing across scientific fields, indicating thataudience interests vary. In what follows, I characterize professional genres referencing scholarship andtheir pattern of referencing journal articles.
To represent each genre, I provide sample titles. The titles concern a single topic, a dental imaging
technique called cone beam computed tomography (CBCT). In CBCT, an X-ray source rotates around thepatient's head, obtaining hundreds of distinct images, which software compiles into a three-dimensional
image. The first CBCT scan was taken in 1994, and the first dental CBCT paper appeared in 1998. In
2001 the FDA approved the first CBCT scanner for the US market. Use in US dentistry took off only in
2006-07 (Schulze 2015), marked by the first sessions on CBCT at the American Dental Association
national conference (Hicks, Melkers & Isett, 2019).
First, I set a baseline with the pattern of referencing in Scopus which is then contrasted with the patternof referencing in news and social media, patents, policy documents, and trade press. Counts report
citations to/mentions of papers published in 2018 or after. Cited fields are aggregated into five highlevel categories: life sciences, including agriculture and molecular biology; health, including medicine
and allied health professions; physical sciences, including chemistry, engineering, computer sciences,and environmental sciences; social sciences, including economics and psychology; and multidisciplinary
journals.

Journal articles
Journal articles exist to communicate research results to the scholarly community and serve as theoutput of research projects. Although a blog could serve the same purpose, authors value the additional
services journals provide which include peer review to certify that an article is worth reading, editorial
oversight to certify for readers that these complex documents meet minimum standards of legibility,
findability in indexes and search, registration or independently associating a discovery with an author
and a time, and keeping the article available in perpetuity. Journals are, in addition, being asked tocertify an expanding list of article characteristics on behalf of readers: that the authors actually
performed the research, that text isn't plagiarized, that images are not manipulated, that research was
conducted ethically, that the underlying data are available to interested readers, and so on.
Journals vary in their relationship to non-scholarly use of research. This is easily seen in a research area
closely connected to professional practice, dentistry. Table 1 shows titles from four dental journals. The
first two specialist journals serve dental researchers and are indexed in the Web of Science. In contrast,
4

General Dentistry and Journal of the American Dental Association (JADA) also serve dentists in practiceand are indexed in PubMed only. Titles in the two specialist journals exhibit precision and technicalcomplexity in their language use, in this case anatomical vocabulary, concern with technique –sialography - and with measurement. Using CBCT to obtain measurements of variable jaw geometry andascertain ranges in the population was a prominent topic in the academic CBCT literature. In contrast,
General Dentistry and JADA use anatomical vocabulary that overlaps with general vocabulary – forexample, "teeth" and address their readers' concern with care – diagnosis and treatment.
Table 1 Journal article sample CBCT titles

Sample Article Title
CBCT and SimPlant Materialize Dental Software Versus Direct
Journal of Oral and
Measurement of the Width and Height of the Posterior Mandible: An
Maxillofacial Surgery
Anatomic Study
JournalDentomaxillofacial
Radiology

CBCT sialography of Stafne bone cavity

General Dentistry

CBCT for diagnosis and treatment planning of supernumerary teeth.

JADA

CBCT and the ortho-surgical management of impacted teeth.

The distribution of references in journal articles across scientific fields depends on both the number of
papers in a field and how long reference lists are in a field. Figure 2 displays the distribution of scholarly
papers and citations across fields, with the total number of papers and citations recorded in the lower
right corner. The physical sciences have the largest presence in Scopus. Health and biomedical sciences
together would be comparable to physical sciences

5Figure 2 Academic pattern of referencing: the distribution of (1) papers indexed in Scopus, (2) the
citations to those papers, and (3) the blog mentions referencing those papers across disciplines.

Note: Total counts of the entities are included. Publications and citations/mentions may be applicable
to multiple disciplines.
Interpretation of the pies also depends on the structure of the classification system. The next level ofthe hierarchical field classification scheme contains 27 fields. If the pies displayed 27 fields, medicine,with 14% of Scopus citations, would account for the largest share, because at the 27 field level medicine
accounts for 88% of the five fields comprising health sciences (the others are nursing, healthprofessions, veterinary, and dentistry). Oncology, infectious diseases, epidemiology, 49 subfields in total(more than double the number of any other field) are not split out at this level. In contrast, engineering,
the largest component of physical sciences, accounts for only 18% of physical sciences because materials
science, physics, chemistry, and computer science are split out at the 27 field level. The same is true ofall the pies discussed below; that is if the pies displayed 27 fields, medicine would account for thelargest share.
The pattern of referencing from blogs is also shown in Figure 2, as it is almost identical to the pattern inthe scholarly literature, with physical sciences accounting for the largest share of citations from blogs,
followed by life and health sciences. Medicine accounts for 13%. This might suggest that blogs
referencing scholarly literature are primarily written by academics.

Public interest
Newspapers seek to entertain, educate and inform the public about current events. Items are news to
the extent they have a big impact, involve conflict, happen nearby, involve well-known people, anddeviate from everyday happenings. Current research advances sometimes meet these criteria,
providing newsworthy items. For example, CBCT met these criteria once for The New York Times. The
headline in Table 2 shows why; there was something to worry about. The article questioned theincreasingly widespread use of CBCT, particularly in orthodontics for children, because CBCT delivers ahigher radiation dose than traditional dental x-rays (Bogdanich and McGinty, 2010).

6

Table 2 News and social media sample CBCT article titles
Sample article title

Genre

Publication

Newspaper

New York Times The radiation boom: radiation worries rise with 3-D dental images

Social media

Twitter

CBCT-based #root-canal length measurements are accurate andreliable when compared with a gold standard. [URL] #endodontic[URL] CBCT X-rays should not be done on every patient. At this point
there is too much radiation. Impactions yes #majeroni

Advances in medicine are most often seen as newsworthy, garnering themost coverage, and accounting for over 34% of citations from news
sources to journal articles (with 1% more coming from the other healthsciences fields), Figure 3. Physical and life sciences each account forover 20% of news citations, with social and multidisciplinary eachaccounting for slightly more than 10%. Papers in multidisciplinary
journals received outsize attention from newspapers as multidisciplinary
journals such as Nature, Science, and PNAS publish and publicize themost dramatic scientific advances with the broadest implications. In
addition, journalists often use academics as sources to provide context
and depth to coverage of current events.
Facebook and Twitter exhibit different patterns of interest. Twitter is used to learn about fast-moving
news, build awareness of information andbuild celebrity. In this environment,
current research advances provide
newsworthy and comment-worthy
content. Of tweets referencing CBCT
journal articles, 70% simply provided thepaper's title and URL, which accuratelyreflects tweeting about dental journal
articles in general (Robinson-Garcia et al.,2017). Table 2 lists other tweets that
more usefully convey the conclusions ofpapers. Figure 4 shows that on Facebook
and Twitter attention seems to be more
evenly distributed across the fourscientific areas, with health sciences garnering the most interest. Medicine accounts for 38% ofreferences to journal articles on Twitter and 30% on Facebook. On Twitter life sciences, which is closely
linked to health sciences, follow whereas on Facebook social sciences, which include psychology, is thesecond most referenced scientific area.

7

Figure 5 provides another view on public interest as revealed bycomments left by people using gmail and ISP email accounts when
downloading journal articles from the Harvard Dash repository.
Here we see heavy interest in social sciences, primarily economics,
as well as humanities. The fields portrayed here make up 99% of
downloads but only 50% of papers in the repository (Doshi, 2023,
table 5.1)

Patents
The classic measure of research use is referencing from patents. Patents protect inventions by awarding
property rights to inventors. In return, they reveal knowledge of how an invention works. The patentoffice mandates that patent titles are short, accurate descriptions of the invention useful for indexing,classifying, and searching. Patent examiners will remove certain words from titles, including new,improved, and novel 3. Although inventors may not wish to reveal their technical advance in the widely
viewed title, patent titles are specific, technical, and convey the purpose of the invention, Table 3.
Table 3 Patent and policy document titles
Genre
PatentPolicy Report

3

Title
Methods and apparatus for super resolution scanning for CBCT system and conebeam image reconstruction
Method for teeth segmentation and alignment detection in CBCT volume
Cone beam CT for dental and maxillofacial radiology: evidence-based guidelines.
(Directorate-General for Energy - European Commission)
The Use of Cone Beam CT in Dental, Oral, and Maxillofacial Surgery, andOtolaryngology Settings (Canadian Agency for Drugs and Technologies in Health)Compliance Guide for Dental Radiology including Dental Cone Beam CT (New
Zealand Ministry of Health)
References in patents identify prior art, serving to establish therequired novelty of the invention given prior art. Patent citations aretaken to represent use of research in innovation. USPTO patents exhibit
the highest rates of referencing to journal articles, and the first pie in
Figure 6 reports referencing to journal articles in USPTO patents issued
2018-2020. The pattern of referencing from patents to scholarly
literature is similar to that of referencing between journal articles in
Scopus with physical sciences accounting for the greatest share,
followed by life and then health sciences. The presence of social scienceis much reduced in patents compared to the journal literature. Thus lifeand health sciences are more visible, accounting for more than half ofpatent citations.

https://www.uspto.gov/web/offices/pac/mpep/s606.html

8

Policy documents
Policy documents are written primarily for or by policy-makers and are intended to influence legislation,
regulation, or other policymaking. 4 Policy documents are a heterogeneous genre system containing, forexample: blog posts, forms, infographics, legislation, meeting agendas, meeting minutes, memos, policy
briefs, posters, press releases, slide decks, speeches, and testimony. Policy reports help prepare policymakers for legislating or regulating by teaching them about an issue and possible approaches togoverning it. They help shape agendas and narratives around issues of concern.
Policy reports aim to cover technical topics for non-specialist audiences comprehensively. Therefore,
their contents cannot be summarized in a title, and report titles are reduced to conveying topic areas.
Table 3 shows three CBCT-related policy report titles. The titles are generic, establishing the association
with CBCT and a concern with regulation but little else. In fact, readers may need to know the source ofthe document to motivate reading, so Table 3 also lists the authoring agency. The EU doesn't regulate
dentists, so it aims to provide guidelines consistent with minimizing radiation exposure and a guide to
future research, which the EU funds. The Canadian report establishes background knowledge to informthe regulation of manufacturers. The New Zealand document advises clinicians and manufacturers on
meeting the requirements of radiation protection legislation.
Policy reports are written by university-trained writers. Presumably,
universities trained writers in referencing, suggesting referencing inreports serves the classic purposes of avoiding plagiarism, providing
resources to readers, and enhancing the text's credibility. Figure 7
suggests that policy-makers' attention is directed very differently thanthe public's. Policy documents, likely mostly reports, most heavily
reference social science literature, which includes economics andaccounts for 37% of references. Health and physical sciences follow
with 24% each. Life sciences attract less interest from policy
documents than from any other genre.

Trade journals, news & blogs
The professional press provides news relevant to the practice of a profession and, in many fields,
continuing education opportunities. The distinction between journals and trade magazines is not always
clear cut. Two of the trade magazines in Table 3, Compendium and Dentistry Today, are partiallyindexed in PubMed. Professional magazines contain articles analogous to those in peer-reviewed
journals as well as articles on topical subjects, news items, reviews, and surveys. Magazines and newssites are supported by advertisements targeted to those in the field. Unique to professional media is
the product announcement. New or improved materials and equipment are introduced continuously
and announced in press releases. Professional media undertake to inform their readership of thesedevelopments. Professional channels also seek to inform readers of upcoming conferences and reporton highlights of recent conferences for those who could not attend.

4

https://help.overton.io/article/whats-your-definition-of-a-policy-document/

9

Table 4 Professional magazine, news, and blog sample CBCT titles
Genre

MagazineNewswebsiteBlog

Publication

Sample Article Title

Compendium

CBCT in endodontics: are we there yet?

Inside Dentistry

CBCT: A Clinician's Perspective

Dentistry Today

Utilizing Digital Imaging to Enhance the Team Approach to Implant
Treatment

Dr Bicuspid

Conebeam and multislice CT measurements found equally accurate
CBCT findings raise liability questions

Endo blog

Uses of CBCT in Endodontics
CBCT in Endodontics to Treat Difficult Anatomy, Preserve Teeth

dentaltown

Cone Beam Imaging is Great, But What Am I Looking At?

Flucke blog

Thanks to Everyone Who Attended my Ultradent 3D Course Yesterday
We've installed the Gendex CB500

Table 4 shows that in contrast to journal articles, patents, and policy reports, articles in professional
media have a more informal style and cover less technical sides of practice. In comparison with thegeneral dentistry journals, the magazine article titles in Table 4 deepen the concern with practice,
explicitly taking the perspective of one who is a team manager as well as a clinician and asking if theinnovation is ready for application. Magazine titles offer enticements to read – promising to answer aquestion or sharing the reader's perspective.
Professional news sources aim to deliver practical, trustworthy, and relevant material to professionals to
help them improve their practice and profitability. They rely on experts, cover widely discussed issues
about which there is disagreement, as well as professionally relevant new social and technologicaltrends. In contrast to magazine article titles, the news titles provide the takeaway up front with thearticle furnishing details for those interested in learning more. The sample titles from professional news
source Dr. Bicuspid report findings in the journal literature on measurement accuracy and discussliability, a business issue.
Most professional blogs do not reference research. The blogs in Table 4 vary in their approach with thefirst being more professional, almost magazine-like, and the others being extremely informal and chatty.
Blog titles reveal an even stronger practitioner focus, firmly grounded in the dentist's point of view.
Professionally oriented blogs, news sites, and magazines differ in publishing models and content.
Publications source articles differently, value different types of information and vary in theirpresentation. Important values such as technical sophistication, the credibility of peer review,grounding in the realities of clinical practice, and being attuned to shifting pressures in dental care areaccommodated to differing degrees in different channels. Each channel disseminates information topracticing dentists about advances in knowledge and information about the profession andmanagement of a practice. Although their reliability is not held in high regard, empirical analysis

10suggests that information in professional media may be as accurate as peer-reviewed literature where it
overlaps (Hicks et al. 2019).
Figure 8 reports the distribution of trade journals, magazines, andnewspapers published in the United States as indexed in Ulrich's. In
contrast to the other pies, social sciences account for 64% of trade
journals, with business & economics alone accounting for 27%. The
references in peer-reviewed articles in trade journals indicate that theauthor may have read the journal article, and it shaped their thinking, i.e.
knowledge flow. References in the trade press are not indexed, but theauthor’s analysis of trade press indexed in Scopus suggested that the
distribution across fields of references from the trade press mirrors the
distribution of titles across fields. That is, business and economics trade
journals likely cite mostly business and economics journal articles. Scopus does index some trade press
content, though the distribution differs with half of Scopus-indexed trade journals and magazines inengineering. 28% of trade press articles indexed in Scopus contain references, with articles averaging
15.6 references, half of which are indexed in Scopus (Elsevier, 2022).

UK REF impact cases
A final genre, not usually thought of as such, are the case studies written for the UK university
assessment of societal impact. Like other genres examined here, these documents reference journal
articles. However, they are unusual in being authored by university researchers to make visible thesocietal impact of their work. In 2021, one of these cases mentions CBCT. Its title is: Bayesian
approaches to X-ray imaging for material inference, dose reduction and improved image quality. Again
we detect concern with the higher radiation dose of CBCT compared to other types of dental x-ray.
Figure 9 Field distribution of UK REF impact cases
Each case is assigned to a field so that it can be read by theappropriate field committee. Figure 9 shows the distribution acrossfields of 6,000 cases written for the 2021 REF assessment. As withtrade journals above, this is not a distribution of citations but a
distribution of documents. Because every UK university
department is required to participate in this exercise and thenumber of cases they submit is proportional to their size, this is the
only source that provides a comprehensive mapping of societal
impact across disciplines. In figure 9 social and professional fields
comprise the largest slice. Most unusual is the appearance of thearts and humanities. It is even possible to see the societal impact of philosophy by reading REF impact
cases.

Whose impact do we see?
Journal articles, patents, trade press, policy reports, newspapers, and social media provide information
for different purposes and more or less often use references to journal articles to support those goals.
From a researcher's perspective, citations from other genres suggest their work is helpful to people
outside academia, hinting at societal impact. Therefore counting citations from other genres has
11attracted increasing attention in recent years, and developing databases to make such counts possible
has become a viable business proposition.
Such citation counts make visible societal impacts to the extent that a field impacts a genre. Different
fields are relevant to different parts of society. For example, while some industries may rely heavily on
trade secrecy, chemistry and pharmaceuticals are very well served by the patent system, and advancesin these technologies are quite closely related to advances in research. Therefore, biomedical sciences
are most visible in patent citation counts. Patent-to-paper citation counts have been available forseveral decades, having first been analyzed in 1997 (Narin et al., 1997). When only patent citation
counts were available, only biomedical sciences and, to a lesser extent, physical sciences seemed to be
applicable outside academia. In 2011 Plum Analytics and Altmetrics.com were founded, making it
possible to count citations from news and social media sources. These sources quantitatively confirm
the intuitively obvious public interest in medical advances. Only in 2019, with the founding ofOverton.io, did it become possible to see the heavy use of social science outside academia in acomprehensive, analytical fashion. An unindexed pool of references remains in the trade press. Ulrich's
indexing of trade journals, magazines, and newspapers suggests even more engagement occurs here
between social sciences and society, specifically with professionals, and would be visible in referencing
patterns. Figure 10 summarizes this argument by compiling all the pie charts into one diagram.
Figure 10 Summary of reference distribution across fields in each genre

Most visible fields

Genre

12

The visibility accompanying citation counts helps researchers and universities being evaluated on theirsocietal impact. But beyond that, such visibility supports arguments that the money government spends
on a field helps society. It is important to remember that field visibility varies across genres and that wehave blind spots corresponding to unindexed genres. Even beyond that, there are societal engagementsthat do not register in documents of any type. Such impact can only be seen comprehensively incollections of case study narratives like those provided by the UK REF exercise. Citation indexes makecitation counts possible across many more genres, but those numbers do not account for everything.

Caveat: Interaction is more than just referencing
Genres interact not only through referencing but also through authorship and mentions. Academics arelisted as inventors on some patents. They host blogs, tweet, and post professional information on
Facebook. Academics also author policy reports and white papers. Particularly in the social sciences,enlightenment literature can be considered one of the four kinds of literature through which knowledge
is advanced and disseminated (Hicks, 2004). Enlightenment literature is a term used to denoteperiodicals dedicated to knowledge transfer to non-specialists, i.e. non-scholarly literature. Social
science builds on enlightenment literature; 67% of references in social science articles are to literature
outside Scopus (Elsevier, 2019, Figure 4). Studies have found that one-quarter of publications of social
science university departments were in the enlightenment literature, or periodicals not indexed in thesocial science citation index (Hicks, 2004). Therefore, academics will author articles in the New YorkTimes (Hicks & Wang, 2013) and in the trade press. In dentistry, several prominent authors with morethan 100 papers indexed in Medline authored articles in dental magazines and on news sites. Professors
who also write for the professional literature can serve a valuable role in diffusing state-of-the-art
knowledge into practice (Hicks, Melkers & Isett, 2019).
Figure 11 Appearances by academics in professional media vary with editorial policies

13

Newspapers and trade press quote professors to provide context and depth to coverage of current
events. In addition, professors' work is occasionally newsworthy enough to merit an article discussing ajust published peer-reviewed journal article. The New York Times mentioned academics in 24% ofarticles in 2011 5. Similarly, Inside Dentistry quoted professors in one-quarter of its articles. Combinedwith the 16% of articles authored by professors, almost 40% of articles involved academics. The news
website Dr. Bicuspid quoted professors in 16% of articles (Hicks, Melkers & Isett, 2019). Figure 11
illustrates the extent of multiplex interaction with scholarship across professional dental literature,
involving not just referencing but also authorship and quotation/mentions, and suggests that differing
editorial policies influence interaction.

Caveat: Citations should be weighted
If we interpret appearance in another genre as use of research, it behooves us to consider how muchthe encounter in another literature engages a reader with the research. Perhaps the most substantial
engagement is offered by articles about a research advance. Newspapers such as the New York Times orthe Economist will write articles explaining recent discoveries of broader interest. As Figure 11 shows,
professional news sites will also write articles about recent advances reported in the scholarly literature.
In this way, broader audiences learn a great deal about a recent advance. Scholars who author articles
in enlightenment or professional literature convey knowledge informed by their research to wider
audiences. If use of research includes advancing public understanding of the world we live in, then suchbroader coverage achieves use.
Referencing also implies a substantive knowledge flow, if not between the scholarship and the broader
audience, then between the author of the enlightenment or professional article and the scholarship.
Similarly, references from patents to papers establish substantive use. Patent references legallydelineate prior art, meaning that the invention offers a novel advance beyond what was reported in thereferenced paper. Such references are taken to indicate knowledge flow between the researcher andthe inventor. References in enlightenment and professional literature might suggest that the article's
author read and used knowledge learned from the referenced paper; therefore, such references could
indicate knowledge flow.
Mentions of professors in news or professional articles seem to be a weaker link. The article's author
presumably contacted the professor to provide context and background to the issue discussed. This
seems less like knowledge flow than an acknowledgment of the professor's credibility as a pundit in the
topic area. Similarly, close reading of tweets in dentistry suggested only a small percentage involved
substantive engagement with the paper by the tweeter or offered the reader substantive information
about the paper beyond metadata such as title or URL. Advocates claimed that tweets wereconversational, reflecting discussion beyond disciplinary boundaries (Priem & Costello, 2010). However,a conversation would require a human behaving like a human on both sides of the transaction. Many
dental tweets turned out to be less than human in being generated by easily automated processes suchas hitting the retweet button to send a paper title and URL. Hitting a button to generate a tweet of
metadata is hardly conversational. And the high-frequency activity so generated leads to information
overload on the side of readers, prompting withdrawal instead of engagement, again not aconversational behavior (Robinson-Garcia et al., 2017). Mentions and tweets, though countable, convey

5

mentioning university, professor or study (Hicks & Wang, 2013)14

less and therefore probably should carry lower weight than other indicators of more substantial use andinteraction between scholarship and broader audiences.

Caveat: Many sources of information are used
Figure 12 Types of material referenced in policy reports
Analysis of references to journal articles risks
creating a blind spot because journal articles are
not the only information source referenced inmost genres and may not even be the mostcommon source of information. Patents, forexample, reference many more patents thanjournal articles. Policy reports reference a broad
range of material. Drawing on a study of US statelevel policy development in autonomous vehicles,
Figure 12 reports the distribution of references in
US state reports about autonomous vehicles.
Although many policy documents are short andunreferenced, most reports contain references.
Here, reports are defined as sophisticated documents containing evidence and analysis to influence orlay the groundwork for decision-making, extending to at least ten pages, and written by or for policymakers. Of 76 state reports found through internet searching, 56 contained references, and the 2,635
references of 54 of these are displayed in Figure 12 6. Reports produced by federal and statetransportation agencies are the most commonly referenced source of information in state reports.
Government reports accounted for 28% of references, while academic sources - journal articles,
conference papers, and books - accounted for 18%. Media, at 15% of references, was almost as heavily
used as academic literature. Reports produced by nonprofits (11%), university transportation research
centers (7%), and consultants (7%) together accounted for one-quarter of the information sources. The
National Academies of Sciences, Engineering and Medicine (NASEM) is prominent in the transportation
information space because it houses the Transportation Research Board, which runs the National
Cooperative Highway Research Program, an annual conference, and a journal (Transportation Research
Record). Together these accounted for 6% of the citations in state AV reports. Corporate information,such as websites of car companies or Waymo gathered 5% of citations from state AV reports.
Figure 12 establishes that academic literature is one of many different sources informing writers ofpolicy reports. Media proves especially useful to policy-makers attuned to current events. Consulting
companies are often employed to write policy reports, which are then cited in later reports. Most policy
areas are a focus for research centers, both nonprofit and based in universities. They also seek tocontribute to the conversation by writing reports, which in turn serve as sources for authors ofsubsequent reports. Policy-makers are understandably attuned to what other governments andagencies are thinking and so most heavily reference other policy-makers' reports. Among these many
players, academic research has a place, though by no means the dominant position in informing policy
Two of these reports were very unusual in their referencing pattern, having three times the number of referencesof any other report and skewing highly academic. They were excluded from figure 12.

615

deliberations as exhibited in reports produced by and for US state governments considering how togovern autonomous vehicles on their roads.

Caveat: Usage can be indirect.
The US National Academies of Science, Engineering and Medicine (NASEM) produces reports
commissioned by federal government departments and so read by federal agencies. Being referenced in
one of these reports is taken as having impact on policymaking. However, other people read them too
meaning their impact is much broader than just federal policymaking. When downloading a report,
readers are asked how they intended to use the report. Anaysis of 1.6 million comments left by US
downloaders of National Academies reports found half of report use to be academic - research, teachingor studying. The other half revealed many users beyond the Federal government. Adults across thecountry seek the high quality information in National Academies’ reports to improve how they do theirjob, to help family members, to satisfy their curiosity and to learn (Hicks et al., 2022).

Caveat: Scholarship engages many genres
Journal articles build on information beyond the journal literature. One out of four references indexedin Scopus is to material outside Scopus, which can be taken to be references to genres other thanscholarly journals. Referencing to non-scholarly material, ranges from about 20% in biochemistry,
molecular biology, immunology, microbiology and neurology to 67% in social sciences and 80% in artsand humanities (Elsevier, 2019, Figure 4).
Newspapers are one genre not indexed in Scopus but sometimes referenced by journal articles. Among
newspapers, the New York Times is by far the most commonly referenced. Close examination ofreferences to the New York Times revealed that use of the New York Times in journal articles was
growing over time and had several motivations. The New York Times is referenced by papers studying
the New York Times or New York City; or when a topic's importance is established with reference topublic interest, as evidenced by press coverage. Roughly half the time, a reference to the New YorkTimes brings into a journal article a quote from a famous person or information about an event, either
very recent or historical. Sometimes references to the New York Times are indistinguishable fromreferences to journal articles, the most famous case being an article reporting how two journalists broke
the anonymization of an AOL data file. Academics sometimes publish in the New York Times or its
magazine, Paul Krugman being the most prominent example, and journal articles will reference thesepieces (Hicks & Wang, 2013).
Another blind spot created by counting references in indexed genres, especially single genres, is thatpublicly engaged research works across many genres, and the genres differ with the type of societal
impact. The UK REF exercise provided an opportunity to see this. This national university researchevaluation required departments to submit narratives describing cases in which research had a societal
impact. References must corroborate not just that the research was published but also the statements
describing the societal impact. This was relatively easy in, for example, inorganic chemistry, wherereferences supporting impact were to patents, corporate websites, and letters from company managers
testifying to their use of the technology.
The field facing perhaps the most difficult challenge in establishing societal impact was philosophy.
Examining the publicly available REF cases in philosophy revealed that philosophers do have a variety ofways to engage the public including: public dissemination, issuing provocations, exploring the16philosophy of everyday items such as wine or information technology, or engaging with people such as
prisoners, teachers, the court system, or doctors and helping them address their problems (Hicks &Holbrook, 2020). Examining the references supporting the impact statements in these cases reveals thateach case touched multiple genres, and the genres involved were highly heterogeneous. They included:
blogs, podcasts, radio and television shows, advertisements, newspapers, trade press, exhibits, movies,
policy reports, white papers, policy organization meeting agendas and minutes, and nonprofit,
government, event, and video websites. Perhaps the only certainty is that if there is a genre of culturalexpression, academics have engaged with it. Undue focus on counts of tweets risks under-appreciating
scholarship's full cultural impact.

Conclusion
Tracking the interactions between genres can provide a window into the use of knowledge throughout
society. Such analysis has become more available with digitization, but beyond that, high-quality
indexing is required to facilitate analysis. Resources such as Altmetric, PlumX, and Overton are central
to allowing analysis to extend beyond the scholarly literature indexed in WoS and Scopus or patents,
indexed in patent office databases. This frontier in bibliometric analysis of societal impact offers many
avenues to explore. What role do other genres play, and how does drawing on research support that?
Do authors reference for the classic reasons of avoiding plagiarism, providing resources to readers, orenhancing the credibility of a text? Or are there additional purposes served by referencing in nonacademic settings? Can we comprehensively track references, mentions & authorship?
Citation counts create visibility for societal impact of research. This visibility can help to support
arguments for research funding because it visualizes the connection between research and societal
benefit. However, some fields benefit more than others, not perhaps because their societal impact isgreater, but because more of their connections result in indexed citations. Therefore, there are blind
spots corresponding to unindexed genres and to unreferenced connections. Only expensive exercises
such as constructing comprehensive sets of narratives or building further citation indexes can overcome
the blind spots. In the meantime, analysts of societal impact should remain aware that there exists richand complex cultural engagement that they are unable to see.

Acknowledgements
I am grateful to Elsevier's ICSR Lab for providing high level publication, citation and mentions from Scopus and
PlumX. This work was partially supported by National Science Foundation Award No. 2001455 and NIH grant U19DE-22516.

References
Begum, M., Pallari, E., & Lewison, G. (2016). European cancer research: From bench to bedside and to
breakfast table. Ecancermedicalscience, 10, ed60.
Bogdanich, Walt; McGinty, Jo Craven (2010). The radiation boom: radiation worries rise with 3-D dental
images. New York Times. 2010; November 22; A1.
Bornmann, Lutz (2015). Alternative metrics in scientometrics: A meta-analysis of research into threealtmetrics. Scientometrics, 103(3), 1123–1144.

17Bornmann, Lutz; Haunschild, R., & Marx, W. (2016). Policy documents as sources for measuring societal
impact: How often is climate change research mentioned in policy-related documents? Scientometrics,
109(3), 1477–1495.
Desmarais, B. A., & Hird, J. A. (2014). Public policy’s bibliography: The use of research in US regulatory
impact analyses. Regulation & Governance, 8(4), 497–510.
Doshi, Ameet (2023). Public Use Of Open Access Research: Evidence From The National Academies And
Harvard Dash Repository. Doctoral dissertation, Georgia Institute of Technology.
Elsevier (2019). Research Metrics Guidebook. Elsevier B.V.
Grant, J., Cottrell, R., Cluzeau, F., & Fawcett, G. (2000). Evaluating “payback” on biomedical researchfrom papers cited in clinical guidelines: Applied bibliometric study. BMJ, 320(7242), 1107–1111.
Hicks Diana; Holbrook, J. Britt (2020). A cartography of philosophy's engagement with society, Minerva
58(1), 25-45, DOI: 10.1007/s11024-019-09384-3
Hicks, Diana (2004). The Four Literatures of Social Science. Chapter 21 in Handbook of Quantitative
Science and Technology Studies, edited by Henk Moed, W. Glänzel and U. Schmoch. Dordrecht: Kluwer
Academic, 473-496.
Hicks, Diana; Wang, Jian (2013). The New York Times as a Resource for Mode 2, Science Technology and
Human Values, 38(6), November, 850-876, doi:10.1177/0162243913497806.
Hicks, Diana; Isett, Kimberley; Melkers, Julia (2017). Evolving Dental Media: Implications for Evidence
Based Dentistry, Journal of Evidence-Based Practice for the Dental Hygienist, Summer 3(2). DOI:
10.11607/ebh.86
Hicks, Diana; Melkers, Julia; Isett, Kimberley R. (2019). A characterization of professional media and its
links to research, Scientometrics, DOI:10.1007/s11192-019-03072-5
Hicks, Diana; Michael Melkers, Julie Barna, Kimberley R. Isett, Gregg H. Gilbert (2019). Comparison of
the accuracy of CBCT radiation effective dose information in peer-reviewed journals and dental media.
General Dentistry, May/June, 38-46.
Hicks, Diana; Matteo Zullo, Ameet Doshi, Omar Asensio (2022). Widespread use of National Academies
report by the American Public, Proceedings of the National Academy of Sciences, 119 (9) e2107760119;
DOI:10.1073/pnas.2107760119
Kryl, D., Allen, L., Dolby, K., Sherbon, B., & Viney, I. (2012). Tracking the impact of research on policy andpractice: Investigating the feasibility of using citations in clinical guidelines for research evaluation.
British Medical Journal Open, 2(2), e000897.
Lewison, Grant; & Sullivan, R. (2008). The impact of cancer research: How publications influence UK
cancer clinical guidelines. British Journal of Cancer, 98(12), 1944–1950.
Narin, Fran; Hamilton, Kim S.; Olivastro, Dominic (1997). The increasing linkage between US technology
and public science. Research Policy, 26(3), 317-330.

18

Nederhof, A.J., Meijer, R.F. (1995). Development of bibliometric indicators for utility of research to users
in society: Measurement of external knowledge transfer via publications in trade journals.
Scientometrics 32, 37–48. https://doi.org/10.1007/BF02020187
Pinheiro, H., Vignola-Gagné, E., & Campbell, D. (2021). A large-scale validation of the relationshipbetween cross-disciplinary research and its uptake in policy-related documents, using the novel Overton
altmetrics database. Quantitative Science Studies, 2(2), 616-642.
Priem J, Costello KL. How and why scholars cite on Twitter. Proc Am Soc Inf Sci Technol. 2010; 47(1): 14.
Robinson-Garcia, Nicolas; Costas, Rodrigo ; Isett, Kimberley; Melkers, Julia; Hicks, Diana (2017). The
Unbearable Emptiness of Tweeting – about journal articles, PLOS ONE 12(8): e0183551.
DOI:10.1371/journal.pone.0183551.
Szomszor, Martin, & Adie, Euan (2022). Overton--A bibliometric database of policy document
citations. arXiv preprint arXiv:2201.07643.
Thelwall, Mike; Kousha, K. (2016). Are citations from clinical trials evidence of higher impact research?
An analysis of ClinicalTrials.gov. Scientometrics, 109(2), 1341–1351.
Thelwall, Mike; Maflahi, N. (2016). Guideline references and academic citations as evidence of theclinical value of health research. Journal of the Association for Information Science and Technology,
67(4), 960–966.
Thelwall, Mike; Kousha, K.; Abdoli, M. (2017). Is medical research informing professional practice more
highly cited? Evidence from AHFS DI Essentials in drugs.com. Scientometrics, 112(1), 509–527.
Van Steijn, F., Rip, A. (1988). The role of trade literature in the communication system. Scientometrics
13, 81–91. https://doi.org/10.1007/BF02017176
Vilkins, S.; Grant, W. J. (2017). Types of evidence cited in Australian Government publications.
Scientometrics, 113(3), 1681-1695.

19