Georgia Institute of Technology
From the SelectedWorks of Diana Hicks

2016

Narrative visualization of the outcomes of Federal
investments in research
Diana Hicks, Georgia Institute of Technology - Main Campus

Available at: https://works.bepress.com/diana_hicks/30/

Using narrative visualization tocommunicate the outcomes of Federal research
Diana Hicks1*, Tony Breitzman2, Richard Cardona3, Sanjay Arora1, Bill Valdez4
1School of Public Policy, Georgia Institute of Technology, Atlanta GA, 30332-0345

2

1790 Analytics, Haddonfield, NJ 08033

3

Hypno Design, Moorestown, NJ 08057

4

US Department of Energy, Washington, DC, 20585*Corresponding author – dhicks@gatech.edu, phone: 404 385 6015, fax: 404 385 0504
Keywords:
patent bibliometrics
narrative visualization
research outcome
communication
evaluation use
February 2014

Abstract
We offer here a narrative visualization entitled: Technology hot spots and the Office of Science andposition its contribution within discussion of novel forms of communicating research results as an aid tomaximizing use of research evaluation. In this study, patent co-citation analysis was used to systematically
identify emerging high impact technologies in the US technology ecosystem and then to establish that Office of
Science of the US Department of Energy (DOE) funds research that underpins these technology “hot spots.”
We discuss the strengths and weaknesses of this novel form of communicating evaluation results based ourexperience.

Funding
This project was supported by the Office of Science, Department of Energy under project number [DEFG02-01ER30316] as well as the former Advanced Technology Program of the National Institute of Standards
and Technology.

11. Introduction
Disappointing uptake of evaluation results has concerned the evaluation community for several decades.
Thinking about the problem of getting evaluation results used has led evaluators to investigate factors that
correlate with use (Teirlinck et al., 2013), to expand their perspective on what counts as use and to encourage
participatory evaluation to induce stakeholder engagement. Though somewhat lower in profile, communication
requires attention from those committed to maximizing use of their evaluation results (Shulha & Cousins,
Weiss, 1997). Effective communication intertwines with other factors effecting evaluation use by, for example,
engaging stakeholders too busy to participate in the evaluation itself. Weiss ends her landmark discussion ofevaluation use like this:The evaluator has to seek many routes to communication – through conferences, workshops,
professional media, mass media, think tanks, clearinghouses, interest groups, policy
networks, - whatever it takes to get important findings into circulation. And then we have tokeep our fingers crossed that audiences pay attention.
Since Weiss wrote, the discussion of evaluation use has continued, with communication remaining anafterthought. In 2007 Lawrenz et al. searched for articles focusing on dissemination as a pathway to use andfound much in the way of “and the results were disseminated” but few to no articles whose topic wasdissemination itself. They fill this gap with their article “Dissemination: Handmaiden to Evaluation Use” inwhich they compare the strengths and weaknesses of several methods of dissemination. Even the bookEvaluation Strategies for Communicating and Reporting is seen as exemplifying evaluation as change
consulting maximizing organizational learning rather than the foundation of the study of communication for
maximizing use (Torres, Preskill & Piontek, 2005, cited in Shulha & Cousins, 1997).
Arguably, the communication problem has become if anything more urgent. In many areas, methods
and data have become more sophisticated, meaning the substance of what needs to be conveyed is more arcaneand more difficult for stakeholders to understand. Anne-Marie Slaughter, former dean of Princeton’s Woodrow
Wilson School of Public and International Affairs was recently quoted in a Washington Post blog saying the
research coming out of public policy schools is “less and less accessible to the lay reader. The jargon has
become more and more specialized” (Piereson and Riley, 2013). As the Post notes, if policymakers can’t
understand policy school research, what can professors hope to accomplish? Evaluation, as part of the broader
enterprise of public policy research, partakes of these problems.
In comparison with evaluation of education or social programs, research evaluation has been lessengaged with broader audiences. This parallels the larger science and technology policy arena, which in the
U.S. benefits from broad public and political support but often fails to stimulate active interest, input, andunderstanding from the general public (Neal et al., 2008, p. 400; Morgan and Peha, 2003). The engagement
problem is further compounded by an exclusive discourse that often restricts the dissemination of evaluation
results. Commissioning stakeholders as well as those evaluated tend to share advanced educational
backgrounds and a vocabulary, as well both are familiar with the programs evaluated, evaluation methods andresults. This simplifies the communication task as evaluators can discuss results with stakeholders and witheach other in much the same way (Torres, Preskill & Piontek, 2005, p. 18). Thus communication methods havenot merited much attention. However, the increasing emphasis on achieving broader societal impacts fromresearch suggests an increasing need to communicate and justify the value of research to broader societalgroups. Communicating with diverse audiences presents difficulties whose solution will require more explicit
attention to and valuing of varied communication strategies to meet the different needs of different audiences(Torres, Preskill & Piontek, 2005, pp. 264-5).
Partly in response to these issues, novel forms of communicating evaluation results are beginning to gain
attention (MacNeil, 2000). In the 2005 second edition of their book, Torres et al. added a section on innovative
forms of reporting including photographs, cartoons, poetry and drama. Innovative formats “offer exciting2

options for increasing the accessibility of findings to a wide variety of audiences” (Torres, Preskill & Piontek,
2005, p. xiii). Johnson et al. recently reported on their experimental use of novel reporting formats in their
STEM education evaluations. They used diagrams, drama, multiple program theories and poetry to align theirpractice with their values-engaged approach to education evaluation (Johnson et al., 2013).
For those wanting to experiment with novel forms of communicating evaluation results, digitaltechnology has greatly expanded the communication repertoire. In particular, data visualization holds great
promise for enhancing evaluation use through engaging more stakeholders more easily. Complex data andideas can be grasped more quickly when presented visually. Visual presentation can also allow viewers tointeract with the data. For both reasons, understanding is enhanced (Card et al., 1999; Spence 2000; Tufte
1990). A recent special issue on the use of data visualization in evaluation begins:At the core of our profession is the need to communicate complex ideas to an array of
stakeholders who may vary in their level of knowledge, interest, and familiarity with theevaluand and methods of measuring. (Azzam & Evergreen, 2013, p. 3)
Azzam and Evergreen believe in the promise of data visualization to assist evaluators in their complex
communication tasks. At present, data visualization is thought about mostly as providing tools for data analysis(Segel & Heer, 2010), this being the agenda of the computer scientists working in the area. Visualization
techniques used not for analysis but for communication are developing quickly and are becoming pervasive inareas outside of academic social science to the extent that "communications technologies are displacing theprinted page from its dominant role as the medium of scientific communication" (Boulton, 2012).
The data visualization agenda prioritizes analysis - facilitating user interaction with the data to facilitate
exploration and discovery. From the evaluators’ perspective, analysis is but one stage of the process. The
chapters in Azzam and Evergreen’s edited monograph explore the use of data visualization at all stages:
understanding the program, stakeholders and context; collecting data; analysis and communicating findings.
Evaluators seeking to use data visualization to enhance communication of findings will have a different set ofpriorities than computer scientists. For example, rather than pure interactivity, evaluators will seek a balance
between telling their story and some interactivity to customize the presentation to the user’s context toencourage engagement (Segel & Heer, 2010). Because visual appeal is engaging, aesthetics will be important.
This dimension can sometimes be neglected by computer scientists (Lau and Moere, 2007; Moere and Purchase,2011). Explaining the results of analysis has always involved using static visualizations, i.e. graphs. Digital
media promises much more through the use of emerging techniques that Segel and Heer (2010) call narrativevisualization. In this paper we present an experiment with a novel form of communicating evaluation results
using narrative visualization.
In this study, patent citation analysis was used to systematically identify emerging high impact
technologies in the US technology ecosystem and then to establish that Office of Science of the US Departmentof Energy (DOE) funds research that underpins these technology “hot spots.” This big data analysis marshalled
extensive data infrastructure and sophisticated analytical techniques, which required a great deal of lengthy
explanation in the final report. The traditional final report was produced, but read by few to none and enjoyedby nobody. The narrative visualization in contrast, has been in use over many years by one of us (Valdez)
because it is unequaled in its impact on a science policy audience. The visualization is an Adobe Flashexecutable file that functions as an enhanced slide show incorporating animation and sound effects to help
describe the methods and convey the results. In its original form, the story is moved forward by the presenter
who narrates live. The end point allows interactive drill down by the presenter and the path can be varied basedon audience interest. As presented here, a narration has been recorded, and the interactive element removed dueto the restrictions of the new format.
One might assume that understanding is enhanced only because over-simplified explanations of verybroad issues aimed at the lowest common denominator are presented. This visualization sought to follow best3

practice guidelines in accurately reflecting the information contained in the data (Azzam et al., 2013, p. 9). The
illustrations were carefully designed to reflect proportions and quantities in the data and results.
As is common in narrative visualization, crafting this presentation required a diverse set of skills (Segel& Heer, 2010). Expertise in data science (Breitzman), analysis (Hicks), audio recording and post-production(Arora), artistic ability (Cardona), and the vision to push beyond standard methods (Valdez) were all required tobring the project to fruition. Although the visualization was produced in 2003, the context in which to discuss it
did not develop until recently. Therefore wider dissemination has been delayed until now.
At this point, the argument for the video has been made and the reader is invited to watch the video here:[URL to be provided]. To facilitate comparison between traditional reporting and narrative visualization, in thefollowing sections we provide a traditional literature review of the techniques used and describe the method andresults.

2. Literature review
In this evaluation, patent co-citation analysis was used to systematically identify emerging high impact
technologies in the US and then to establish that Office of Science (DOE) funds research that underpins thesetechnologies. Co-citation analysis of scientific papers was pioneered by Henry Small at ISI in 1973 and has
been in constant development ever since (Small, 1973; Small, 2003; Klavens and Boyack, 2005). Widely usedin the study and evaluation of science, co-citation also forms the basis of ISI/Thomson-Reuters identification of
Research Fronts, or clusters of highly cited papers. In the 1990s the number of studies using co-citation
analysis increased as computational power became more widely available. In 1999 the first patent appearedcontaining the term "co-citation analysis". By 2011, the term "co-citation analysis" had appeared in over 50
patents assigned to Lexis-Nexis, IBM and Xerox, among others (Halevi, 2011).
From the beginning, visualization has been used to present the results of co-citation analysis, in that theresults are displayed as metaphorical maps of the scientific landscape. The "map" presents a projection into twodimensions of the position in n-dimensions of scientific areas in relation to each other. On the map are circles
representing clusters of papers and lines between clusters that are strongly linked by citations in common1. The
maps are interpreted as the location of specific fields within the realm of science as well as the "distance"
between specialties (Rip, 1988.) Using VxInsight software, the metaphor has been extended into a third
dimension with height representing another metric, such as number of citations to the papers in a cluster (forexample see Boyack and Borner, 2003). Today, Thomson-Reuters provides research front maps at
http://www.sciencewatch.com/dr/rfm/ and some of the most beautiful and sophisticated current maps areindexed in the knowledge networks section of www.visualcomplexity.com. Maps, beautiful as they can be, are
not the only way to use the results of co-citation analysis. We hope to demonstrate this here by focusing
attention and exploration on important individual clusters, rather than on the relationships between all clusters.
Most often patent citation analysis focuses on the cited patent and how many citations it receives(forward citations). This makes sense because research has found that highly cited patents are associated withtechnological and commercial success. Many studies have linked highly cited patents with other measures ofsuccess such as invention awards, increases in sales and profits, expert opinion and licensing revenues (Albert etal., 1991; Breitzman & Mogee, 2002; Carpenter et al., 1981; Hicks et al., 1986; Narin et al., 1987). Co-citation
has not been applied to patents very often. Mogee and Kolar (1999) used the technique to identify thetechnology fronts pursued by Eli Lilly. Lai and Wu (2005) proposed using co-citation to develop a patentclassification scheme. Co-citation analysis is also offered as part of the on-line patent analysis tool kit at1

In 2003, a selection of this type of work was published in in special section of JASIST edited by Chaomei Chen (Chen,

2003).

4

www.acclaimip.com. This paper hopes to encourage wider use of co-citation on patents by revealing the utilityof the technique.
The funding acknowledgements on papers are another little used analytical resource. In 1997, Narin etal. included analysis of funding acknowledgements in a paper examining all US authored papers cited in US
patents. Narin et al. found that NIH was the most acknowledged source of funding on biomedical papers citedin patents while NSF lead in chemistry, physics and engineering. Cronin and Shaw (1999) examined four
information science journals and found that a paper's citedness was associated with journal of publication and
an author’s nationality, but not with funding. Harter and Hooten (1999) examined nine volumes of JASIS andfound no relationship between whether an article was funded and citation count. Using grant proposals
containing lists of publications and their respective funding sources, Butler examined research funded by the
National Health and Medical Research Council of Australia (Butler, 2001). Exploring the accuracy with which
researchers report links between publications and the grants from which they emanate, Butler found that
acknowledgments data appear to accurately reflect a funding body's total research output. Boyack and Borner(2003) used data on individual grants to track paper output at the finest possible level arguing that betterdatabases should be constructed to facilitate this type of analysis. Wang and Shapira (2011) examined thepattern of funding acknowledgments on international nanotechnology papers. They found that if funding is
widely distributed across many recipient organizations, then citation counts are relatively higher. National
science foundations and research councils fund relatively more international collaborations than other types offunders. In this paper we continue the work of exploring the utility of funding acknowledgements by using
them to position a Federal agency's funded research within the US technology landscape.

3. Method
In this study, patent co-citation analysis was used to systematically identify emerging high impact
technologies in the US technology ecosystem and then to establish that Office of Science of the US Departmentof Energy (DOE) funds research that underpins these technology “hot spots.” (The Office of Science funds
basic research, especially in the physical sciences.) The results are derived from a comprehensive analysis of
the US science and technology ecosystem using bibliometric databases and techniques. The resources used in
the study were these:
1. The US patent database, with assignee names cleaned up to organizations
2. The database of US publications
3. A database of papers cited in US patents with associated institutional affiliations and funding
acknowledgements.
4. Co-citation clustering of patents
The advantage of co-citation analysis is currency and the ability to move beyond the patent level, to thelevel of technologies, without having to use pre-existing categories which can be too general to be helpful.
With co-citation clustering, patents that inventors consider to be similar are identified and the analyst can
deduce the technological theme being developed. The co-citation technique leverages the networked structureof the knowledge ecosystem to provide a very good way of sorting through the several million patents issued
over the past several decades to identify technologies that are spurring current developments.
In this project, current developments were identified by using only references from recent patents (20012002, analysis conducted in 2003). In this set of references we identified all U.S. patents cited at least 10 times.
The cited patents might be 1 year old or 20 years old. In this step, we eliminated 95% of the cited patents notcited 10 times by the 2001-2002 patents. We kept the top 5% patents having a high impact on emerging
technology. Next we eliminated patents with long records of high citation. Out of over two million older
patents, 16,000 high impact patents were left or 2% of US patents issued 1995-1999. Next we grouped together
any highly cited patents that are co-cited by 2001-2002 patents into clusters and call these clusters “hot-spots.”
5

The co-citation clustering produced 423 hot spot clusters with at least 5 patents in them. These clusters contain
10,000 or 62.5% of the high impact, emerging technology patents. The hot-spots and the next generation ofpatents citing them identify all points in the U.S. technological space around which a confluence of inventive
activity is developing.
Hot spot patents differ from highly cited patents. Not all highly cited patents will be hot spots, and not
all hot spot patents will be highly cited. For example, in 1978 Xerox was awarded a very important patent
related to the development of the ethernet entitled: "Multipoint data communication system with collision
detection". As of 2003, this patent had been been cited 210 times. However, only 3% of those citations werefrom recent patents (issued 2000-2001). This patent is no longer central to current developments. On the other
hand, in 1990 American Home Products was awarded a patent entitled: "Novel benzyl-3H-1,2,3,5oxathiadiazole 2-oxides useful as antihyperglycemic agents". This patent has been cited 16 times, which is not
unusual for a patent of its age and technology class. This patent is not highly cited. But 15 of these 16 citations
were from recent (2000-2001) patents, which is very unusual. This patent was in a hot-spot.
Note also, not all patented innovations and so not all hot spots are in glamorous high-tech areas likeelectronics, drugs, chemicals or software. A 1989 patent with the simple title “Scooter” was cited only twice inthe 10 years after issue, but then things got interesting and the patent gathered 11 more citations in little morethan a year. Patents citing the scooter in the early 2000s included features found in the foldable “razor”
scooters.
Technological development tends to be cumulative in that innovations build on prior technology. The
hot-spot methodology takes advantage of this fact. In fact, if an innovation were to emerge without any
precedents, the hot spot method would miss it. A time machine built from scratch would be an exciting
technological development but would not be seen in a hot spot analysis.
Despite this potential problem, which would bedevil any citation analysis, hot-spot analysis is a
powerful tool to map out current areas of innovation activity. The tool allows us to get beyond the patent levelto look at technology areas, yet it permits us to do this without having to fly up to 30,000 feet and talk in
generalities such as organic chemicals or semiconductors. Using the hot-spot approach, we identify specifictechnologies in which innovation is concentrated. It enables us to see technologies of current interest hidden
among the three million or so U.S. patents issued since 1975. See [ref blinded] for a more detailed
methodological explanation including validation exercises.

4. Results
In part the results of the project are the demonstration that the analysis can be presented in video format.
In this section we briefly repeat in text the analytical results described in the video.
The analysis identified 16,000 US hot spot patents, or patents whose impact on recently issued patents isparticularly strong. Co-citation clustering organized 10,000 of these patents into 423 clusters containing five or
more patents. The DOE presence among these patents is stronger than expected. For patent issue years 19951999, 2% of all US patents are in hot spots but 4.5% of DOE patents are in hot spots. Thus DOE-linked patents
are found in hot spots more than twice as often as expected. In addition, DOE research underpins more than
five times the expected number of hot spots, given the size of their research funding. DOE accounted for 1.6%of US civil R&D in 2000, and we detected DOE contributions in 9% of hot spots (37 of 423).2
With the hot spot structure in place, we can identify clusters supported by DOE research. A hot spot
cluster is said to benefit from DOE research if there is more than one DOE-linked patent in it and DOE-linked
patents are at least 10% of the hot spot – a 10%/1+ patent rule. A DOE linked patent is one that arose from2

Patents resulting from DOE research or patents citing DOE research accounted for 1.2% of US patents in 2000

6

DOE research or that cites DOE research papers as prior art. DOE supported research is well represented in thescience that underpinned hot spots in US innovation in 2001-2002, having an especially strong impact inbiotechnology, advanced materials and electronics.
DOE’s impact ranges broadly across the technology landscape. Table 1 lists technology areas and theshare of clusters that are DOE-linked. Because the hot spot technique identifies technologies at a very detailed
level, we can drill down to precisely identify technologies to which DOE is contributing. Table 2 lists DOElinked hot spot clusters. Technology areas in which DOE has a big impact are: biotechnology; assays in arrays,
biochips, microfluidics; microelectronic fabrication technology; microelectromechanical technology; energy
storage/fuel cells, battery; polymers synthesis and polymer fabrication technologies and medical devices – inparticular medical imaging.
Table 1 - Number of DOE and non-DOE clusters in each broad technology area
Broad area
Advanced Material/Chemicals
Biotechnology
Electronics/Computer Hardware/Communications
Information Technology
Manufacturing
Medical
Other

7

DOE
9
14
9
1
2
3

Other DOE %
49
16%
5
74%
122
7%
53
2%
47
4%
77
4%
32

Table 2 – List of DOE linked hot spot clusters
Broad area
Advanced Material/Chemicals
Batteries With Polycarbon Sulfide
Fuel Cells
Metal-Air Battery
Microsensor Mass & Chemical
Molybdenum & Iron Catalyst Production Of Syndiotactic 1,2 Polybutadiene
Nanocrystals/Quantum Dots Flourescence Probes
Olefin Polymerization & Related Catalysts
Olefin Polymers
Biotechnology
Affymax/metrix Biological Chip
Caliper Lab On Chip
Combinatorial Chemistry
Remotely Programmable Matrices With Memories
DNA Amplification & Detection Including Rolling Circle Amplification
DNA Shuffling And Recombination Directed Evolution
Fiber Optic Biosensor Array
Gene Chip Manufacturing esp. Binding & Detection
Mass Spectrometry & DNA
Microfluidic Devices
Nucleic Acid Chips
Nucleic Acid Detection Using Electric Currents
Plant Genetic Engineering - Monsanto Co-citing
Sorting & identifying polynucl. using oligonuc. tags
MedicalHyperpolarized Noble Gases For Medical Imaging
Imaging Tissue with Autofluorescence
Inactivating Viral And Bacterial Blood Contaminant
Electronics/Computer Hardware/Communications
Extreme UV Lithography
IBM Magnetic Tunnel Junction Device
Method Of Semiconductor Fabrication – Hynix
Microelectromechanical Devices esp. Actuators
Microelectromechanical Fluid Ejection
OLED's Including Thin Flexible Encapsulation
Semiconductor Fabrication Alkaline Earth Oxide
Semiconductors/ Metal Thin Films In Capacitors Largely Micron Co-citing
Short Range Radar Object Locator
Information Technology
Digital Security - Steganography, Watermarking
Manufacturing
Liquid Carbon Dioxide Dry Cleaning
Stirling Engines, Heat Engine

8

5. The importance of the broader audience“The mission of the Energy Department is to ensure America’s security and prosperity by addressing its
energy, environmental and nuclear challenges through transformative science and technology solutions” (USDepartment of Energy). One way of estimating the success of DOE's research is to measure the extent to which
the DOE funds transformative science and technology using techniques such as hot-spot identification andanalysis. However, it can also be argued that the success of DOE – or any agency in the US for that matter – isa function of its ability to lucidly communicate its accomplishments to the public and decision makers. This ischallenging because DOE's research portfolio touches on many complex technical developments, because theanalysis relies on techniques understood by a small specialist community, and because stakeholders cannot becompelled to devote the hour required to read the report. Here, we show that new forms of media can be used
to report the results of complex indicator based analyses. Our video "report" included the traditional
explanatory elements of background, context, methodological explanation and results. Yet compared to
traditional text reports, the form has the potential to enhance the dissemination and reporting of science andtechnology evaluation for greater impact on policy discourse.
Our argument in favor of narrative visualization needs to be balanced with consideration of the limits ofthe form that surfaced during informal and formal presentations of this project. Evaluation results need to becommunicated to multiple audiences - internal and external - for a variety of purposes – budgeting, program
management, public affairs etc. Different forms of communication suit different audiences. Therefore bestpractice is to deploy multiple formats (Lawrenz et al. 2007; Johnson et al. 2013) as thought out in advance in an
evaluation reporting plan (Torres et al. 2005). Torres et al advocate devising effective communication plans
using their structured, 50 point analysis of every stakeholder. This type of analysis will help evaluators decide
for whom and when a narrative visualization might fit into their broader portfolio of reporting options.
Unfortunately, the necessity of using multiple forms of communication works against the viability ofvisualization in a resource constrained environment. Visualizations, attractive though they are, will not suit theneeds of every stakeholder, but because visualizations are relatively expensive, their production might exhaust
resources, leaving nothing for other communication formats. This was a problem here.
Another difficulty with this format is its standalone nature. The tables and figures in traditional reports
can be reworked or copied into new documents. The ability of stakeholders to transform evaluation results andcombine them with other information to produce new documents and presentations enhances the usefulness of
results to sponsors. That flexibility is sacrificed with this presentation format.
A final problem perversely arises from the very effectiveness of narrative visualization in enabling
broader communication of complex analyses. If an analysis goes no further than frequencies, trends andcorrelations, traditional tables and graphs will be used because they are adequate to the job. Assume then thatcomplex and innovative analyses is conveyed with visualization. The direct sponsor, most closely engaged in
the project, will understand the motivation and rationale for the work, the data and the main lines of analysis.
With this close involvement, the sponsor understands and benefits from a report as well as a narrativevisualization. The sponsor will be able to reach more people with the narrative visualization, who will engage
with the results in consequence. The strength of the technique will be its ability to engage and persuade nontechnical stakeholders. In this visualization may be able to make a unique contribution to the communication ofevaluation results.
However, in research evaluation, scientists are also stakeholders and they too are more likely to engage
with a narrative visualization than with a report. Scientists will not necessarily be persuaded by thevisualization. First, academics in general mistrust visual presentation. Second, scientists may not understand or
trust the techniques. Perhaps this is because they were not taught them in college or because as scientists they
mistrust social science in general. Whatever the reason, their learning the results of the study will not be the9same thing as them believing the results of the study. Therefore, in research evaluation, wider dissemination toscientists may ironically lead to wider dismissal of the value of the work.
Against these limitations are set the unique strengths of visual presentation. Accessible presentation ofevaluation may increase impact. Policymakers are limited in their capacity to absorb and process large amounts
of information, and rely on prepackaged solutions when encountering routine choice situations (Jones, 2003). In
this context, innovative presentations that compact and smooth the delivery of complex information with thegoal of making policy research more easily accessible and comprehensible, might enhance the influence of highquality information on decision making, thus advancing the goals of democratic participation. Such adevelopment would be welcome especially in science and technology policy, where specialist techniques suchas bibliometrics, patent analysis and data mining are used (Coates et al., 2001).
In our concern not just with sound analysis but also with communicative form, we marry two goals of
real-time technology assessment (RTTA), which are: 1) to facilitate understanding about a technology’s
development through text-mining, bibliometrics, interviews, etc., and 2) to promote communication between
researchers and the public to identify expectations and consequences of socio-technical interaction (Guston andSarewitz, 2002). Here we seek to unite the findings of a complex research program with the need to connect
active and latent actors in the policy network in previously untapped ways. While our work is only a small
contribution to this larger goal, we hope that it acts as an exemplar for future work.
The value of visualization is perhaps most striking in the classroom. Students' enthusiastic response to
the material offered an unexpected moment to reflect on the progress of interdisciplinary social science ineducational settings in this new era of rapid evolution in information and communication technologies. Indeed,
previous research in the use of technology for educational purposes has shown that the strategic separation ofgraphics and text into narrative formats increases student understanding and retention (Wiebe and Annetta,
2008). Weibe and Anetta (2008) confirm that when narratives are present, students focus more on thesubstantive content of graphics. The idea that this approach can aid classroom learning comes with at least two
caveats. Firstly, the design of such multimedia tools should consider “complex learning” that combines
technology with traditional lecture and discussion and possible unconventional forms of evaluation (Mandl
cited in Backer, 2004). Secondly, students differ in learning styles, some of which are more or less receptive to
multimedia forms of teaching (Choi et al., 2008).
As the academic community prepares to educate the next generation of technologically sophisticatedstudents, we must ask whether our collective research presentation methods are as inspiring and relevant as
possible. In many cases, the answer is no (Felder and Silverman cited in Choi et al., 2008). It is our opinion
that this research – and more importantly, the novel presentation format – is one step in the right direction to
attract new interest in “fun” ways to present and eventually conduct state-of-the-art empirical evaluation. In aworld where normative claims increasingly supersede substantive evidence, higher-education tools that
overcome complexity without sacrificing theoretical and methodological integrity should be of considerable
interest to scholars across a variety of disciplines.

6. Conclusions
Science is a cumulative enterprise - that is today’s discoveries build upon past work and today’s research
will form the foundations for future innovation. Also, knowledge is structured in a network - that is a discovery
will draw upon a wide range of past efforts, some very similar to itself and others quite different. And today’s
discovery will contribute to a broad range of future efforts, some closely related and others quite distant.
Because knowledge is cumulative and networked in its structure, the contributions to knowledge made by alarge enterprise such as the Department of Energy are not easy to describe. One has to look back in time, as
well as realize that impacts will develop over time, into the future. Also, tracing all impact entails ranging far
and wide over the knowledge landscape. In addition, the audience is often less interested in the impact on
10

knowledge, than in outcomes, or the contribution to the economy through generation of innovative newtechnologies of relevance to US firms. The work presented here was designed to address the problem ofmaking visible the outcomes of the research sponsored by the Office of Science at the Department of Energy.
We do this in a way that pays respect to the cumulative, networked character of knowledge.
We present here a co-citation analysis of the US patent system, with linkages to papers cited in thepatents which are in turn linked to funding agencies through funding acknowledgments on the papers. Cocitation analysis has been widely used on papers, but little used on patents. Funding acknowledgements havebeen not been widely used in analysis. Others use visualization techniques to produce maps of the scientificlandscape based on co-citation results. We produce no maps. Rather our visualization seeks to tell the story adifferent way.
This text accompanies the video: Technology hot spots and the Office of Science. As the video shows,
this analysis generated a high level overview of the US technological landscape in 2001-2002 and identified“hot” emerging technologies within that landscape. We were able to trace back to reveal where Office of
Science research support underpinned the emerging technologies. The Office of Science was shown to havebroad impact across a range of technology areas. U.S. firms citing Office of Science research were identified.
In doing this we also found that the Office of Science has 2.8 times the expected number of hot spot patents.
The analysis suggests a high return to the tax dollars spent on Office of Science research.
The hot spot technique is a powerful one for tracing research outcomes. It takes advantage of the
cumulative nature of technological advance by using references made from current patents to past, predecessor
inventions. Because it works with system level, big data resources it is able to track impacts across theinnovation ecosystem. With the hot spot technique, the complex networks that characterize technologicaladvance are not a barrier to understanding, but a resource. The technique follows the myriad connections in the
networked structure that characterizes the advance of knowledge to reveal the broad impacts of researchfunding.
Mindful of the need to describe complex analyses to non-specialist audiences, we used narrativevisualization to convey the results of this analysis. The strengths of narrative visualization as compared to
traditional reporting are that techniques and results can be conveyed more effectively and quickly than ispossible using text. This is important if the large amounts of money spent on this kind of work are going toprovide value to more than a narrow specialist audience. In policy work in particular, being able to reach abroad and non-specialist audience is crucial to informing decision making. The challenges in using thesetechniques include the cost required to bring onto the team additional skills, the difficulty analysts have inarticulating their work in a visual form, and the cool reception such a presentation will likely receive at present
in traditional academic circles. Over the longer term, if it can be shown that the understanding achieved
through the visualization is comparable to that achieved by reading a long technical report, skepticism may
decrease. We advocate augmenting traditional presentation modes to succeed in a world of increasing audience
diversity, information clutter and extreme competition for the attention of stakeholders.

7. References
Albert, Michael, Daniel Avery, Paul McAllister & Narin, F. (1991) ‘Direct validation of citation counts as
indicators of industrially important patents’, Research Policy, 20: 251–259.
Azzam, Tarek, and Stephanie Evergreen, eds. (2013) ‘Data Visualization: Part 1’, New Directions for
Evaluation, Number 139. John Wiley & Sons.
Azzam, Tarek, Evergreen, S., Germuth, A.A. & Kistler, S.J. (2013) ‘Data Visualization and Evaluation’, New
Directions for Evaluation, 139: 7-32.
11

Backer, Patricia (2004) ‘Using multimedia to teach a class on technology and society’, The Journal ofTechnology Studies, 30/2: 77-79.
Boulton, Geoffry (2012) ‘Open your minds and share your results’, Nature, 486: 441.
Boyack, Kevin W. and Katy Börner (2003) ‘Indicator-assisted evaluation and funding of research: Visualizing
the influence of grants on the number and citation counts of research papers’, Journal of the American
Society for Information Science and Technology, 54/5: 447–461.
Breitzman, Anthony & Mogee, M.E. (2002) ‘The many applications of patent analysis’. Journal of Information
Science, 28/3: 187–205.
Butler, Linda (2001) ‘Revisiting bibliometric issues using new empirical data’, Research Evaluation, 10/1: 5965.
Card, Stuart, Jock Mackinlay & Shneiderman, B. (1999) Readings in Information Visualization, Using
Visualization to Think. Burlington, MA: Morgan Kaufmann.
Carpenter, Mark, Francis Narin & Woolf, P. (1981) ‘Citation rates to technologically important patents’. World
Patent Information, 4: 160–163.
Chang, Connie K.N. and Anthony Breitzman (2009) ‘Using patents prospectively to identify emerging, highimpact technological clusters’, Research Evaluation, 18/5: 357-364.
Chen, Chaomei (2003) ‘Visualizing Scientific Paradigms: An Introduction’, Journal of the American Society for
Information Science and Technology, 54/5: 392-393.
Choi, I., S.J. Lee, & J.W. Jung. (2008) ‘Designing Multimedia Case-Based Instruction Accommodating
Students’ Diverse Learning Styles’. Journal of Educational Multimedia and Hypermedia, 17/1: 5-25.
Coates, V., M. Farooque, R. Klavans, K. Lapid, H. A. Linstone, C. Pistorius, & A.L. Porter (2001) ‘On the
Future of Technological Forecasting’, Technological Forecasting and Social Change, 67/1: 1-17.
doi:10.1016/S0040-1625(00)00122-0
Cronin, B., & Shaw, D. (1999) ‘Citation, funding acknowledgement and author nationality relationships in four
information science journals’, Journal of Documentation, 55/4: 402–408.
Guston, D. H., & D. Sarewitz (2002) ‘Real-time technology assessment’, Technology in Society, 24/1-2: 93109. doi:10.1016/S0160-791X(01)00047-1
Halevi, Gali (2011) ‘The Beginning of a Beautiful Friendship – Social Sciences Research and Industry
Products’, Research Trends, November. Accessed on July 13, 2012 at:
http://www.researchtrends.com/issue25-november-2011/the-beginning-of-a-beautiful-friendship%E2%80%93-social-sciences-research-and-industry-products/ .
Harter, S. P., & Hooten, P. A. (1992) ‘Information science and scientists: JASIS, 1972–1990’, Journal of theAmerican Society for Information Science, 43/9: 583–593.
Hicks, D.M., B.R. Martin and J. Irvine, "Bibliometric Techniques for Monitoring Performance inTechnologically Oriented Research: the Case of Integrated Optics", R&D Management, 16: 211-224,
1986.
Johnson, Jeremiah, et al. (2013) ‘Exploring Alternative Approaches for Presenting Evaluation Results’,
American Journal of Evaluation 34/4: 486-503.
12

Jones, B. D. (2003) ‘Bounded Rationality and Political Science: Lessons from Public Administration and Public
Policy’, Journal of Public Administration Research and Theory, 13/4: 395-412.
doi:10.1093/jpart/mug028
Klavans, Richard, Boyack, Kevin W. (2005) ‘Identifying a better measure of relatedness for mapping science’,
JASIST 57:251-263.
Lai, Kuei-Kuei and Shiao-Jun Wu (2005) ‘Using the patent co-citation approach to establish a new patentclassification system’, Information Processing & Management, 41/2: 313-330.
Lau, Andrea, and Andrew Vande Moere (2007) ‘Towards a model of information aesthetics in information
visualization’, Information Visualization IV'07, 11th International Conference, IEEE.
Lawrenz, Frances, Arlen Gullickson, and Stacie Toal (2007) ‘Dissemination handmaiden to evaluation use’,American Journal of Evaluation, 28/3: 275-289.
MacNeil, Cheryl (2000) ‘The prose and cons of poetic representation in evaluation reporting’, American
Journal of Evaluation, 21/3: 359-367.
Moere, Andrew Vande, and Helen Purchase (2011) ‘On the role of design in information visualization’,Information Visualization, 10/4: 356-371.
Mogee, Mary Ellen and Richard G Kolar (1999) ‘Patent co-citation analysis of Eli Lilly & Co. patents’, Expert
Opinion on Therapeutic Patents 9/3: 291-305.
Morgan, M. G., & Pea, J. M. (2003) Analysis, governance, and the need for better institutional arrangements. In
M. G. Morgan & J. M. Pea (eds.), Science and Technology Advice for Congress, pp. 3–18. Washington,
D.C.: Resources for the Future.
Narin F, Hamilton KS, Olivastro D. (1997) ‘The increasing linkage between US technology and public science’,
Research Policy, 26: 317-330.
Narin, Francis, Elliott Noma & Perry, Ross (1987) ‘Patents as indicators of corporate technological strength’,
Research Policy, 16: 143–155.
Neal, H. A., Smith, T., & McCormick, J. (2008) Beyond Sputnik: U.S. Science Policy in the 21st Century. Ann
Arbor, MI: University of Michigan Press.
Piereson, James and Naomi Schaefer Riley (2013) ‘The problem with public policy schools’, Washington Post
opinion, December 6.
Rip, A. (1988) ‘Mapping of science: possibilities and limitations’. In Van Raan A.F.J. (ed.) Handbook of
Quantitative Studies of Science and Technology. pp. 253-273. North-Holland: Elsevier.
Segel, Edward & Heer, Jeffery (2010) ‘Narrative Visualization: Telling Stories with Data’, IEEE Transactions
on Visualization and Computer Graphics, November, 1139-1148.
Shulha, Lyn M., and J. Bradley Cousins (1997) ‘Evaluation use: theory, research, and practice since 1986,’
American Journal of Evaluation, 18/3: 195-208.
Small, Henry (1973) ‘Co-citation in the scientific literature: A new measure of the relationship between two
documents’, Journal of the American Society for Information Science, 24/4: 265–269.
13

Small, Henry (2003) ‘Paradigms, citations, and maps of science: A personal history’, Journal of the American
Society for Information Science and Technology, 54/5: 394–399.
Smith, K. & Larimer, K. (2009) The Public Policy Theory Primer. Boulder CO: Westview Press.
Spence, Robert (2000) Information Visualization. New York City: ACM Press.
Teirlinck, Peter, et al. (2013) ‘Closing the policy cycle: Increasing the utilization of evaluation findings inresearch, technological development and innovation policy design’, Science and Public Policy 40/3: 366377.
Torres, Rosalie T., Hallie Preskill, and Mary E. Piontek. (2005) Evaluation strategies for communicating andreporting: enhancing learning in organizations, 2nd edition. Thousand Oaks, CA: Sage.
Tufte, Edward (1990) Envisioning Information. Cheshire CT: Graphics Press.
US Department of Energy. Retrieved from energy.gov on July 18, 2012.
Wang, Jue and Philip Shapira (2011) ‘Funding acknowledgement analysis: an enhanced tool to investigate
research sponsorship impacts: the case of nanotechnology’, Scientometrics, 87/3: 563-586.
Weiss, Carol H. (1998) ‘Have we learned anything new about the use of evaluation?’ American Journal of
Evaluation 19/1: 21-33.
Wiebe, E. & L. Annetta (2008) ‘Influences on Visual Attentional Distribution in Multimedia Instruction’,
Journal of Educational Multimedia and Hypermedia, 17/2: 259-277.

14