Technology Assessment Design Handbook
Released: 2019-12-04T00:00:00

Letter

1.1 Reasons to Conduct and Uses of a Technology Assessment

TAs are significant given their increasing importance to policymakers, and the growing effects of S&T on society, economy, and other areas. While technological changes can be positive, they can also be disruptive. Therefore, it is critical for Congress to be able to understand and evaluate these changes, to ensure, for example, national security and global competitiveness. Examples of potential uses of TAs related to enhancing knowledge and awareness to assist decision-making include:

Highlight potential short, medium, and long-term impacts of a

Elaborate on and communicate the risks and benefits associated with a technology, including early insights into the potential impacts of technology

Highlight the status, viability, and relative maturity of a technology

Plan and evaluate federal investments in S&T GAO TAs are most commonly requested by congressional committees, which may use them to, among other things, make decisions regarding allocating or reallocating resources to address research gaps, support updated rulemaking for a regulatory agency, or inform a legislative agenda or the development of a national strategy.

Technologies present opportunities and challenges that may vary, depending in part on the policy context in which they are evaluated. Therefore, part of a TA is considering the policy context surrounding a given technology. GAO may, where appropriate, identify and analyze policy options as part of its TAs, which may also include: clarifying and summarizing policy-related issues and challenges, and providing information that can be used for decision-making. In this situation, policy options can be defined as a set of alternatives or menu of options (including the status quo) that policymakers, such as legislative bodies, government agencies, and other groups, could consider taking. Policy options can be used to articulate a range of possible actions a policymaker could consider in the context of a given technology and policy goal. Policy options do not state what policymakers should do in a given circumstance with a certain technology. Policy options do not endorse or recommend a particular course of action; they are not recommendations or matters for congressional consideration, which GAO makes in its audits. In addition, policy options are addressed to policymakers more broadly, and are not addressed to a specific federal agency or entity.

1.2 Importance of Spending Time on Design

Developing a written TA design helps TA teams agree on and communicate a clear plan of action to the project team and the team’s advisers, requesters, and other stakeholders. Written TA designs also help guide and coordinate the project team’s activities and facilitate documentation of decisions and procedures in the final report. In addition, focusing the TA on answering specific researchable questions can assist teams to define and select the appropriate scope, approach, and type of product, ensuring usefulness of the product to the intended users. More specific reasons for spending time on systematically designing a TA include:

Enhance its quality, credibility, and usefulness

Ensure independence of the analysis

Ensure effective use of resources, including time Data collection and quality assurance of data can be costly and time- consuming. A thorough consideration of design options can ensure that collection and analysis of the data are relevant, sufficient, and appropriate to answer the researchable question(s), and helps to mitigate the risk of collecting unnecessary evidence and incurring additional costs.

This chapter highlights design phases, cross-cutting considerations, and GAO TA design examples for sound technology assessment (TA) design. To ensure that the information and analyses in TAs meet policymakers’ needs, it is particularly useful to outline the phases and considerations involved in sound TA design, while remaining aware of the iterative and nonlinear process of designing a TA. The information presented in this chapter is based on review of results of a literature search, an expert forum, select GAO reports, and experiences of GAO teams and technical specialists. For more information, please refer to Appendix I: Objectives, Scope, and Methodology.

2.1 Sound Technology Assessment Design

Below are questions to consider for a sound TA design. Reflecting on these questions may help teams make important decisions (like selecting an appropriate design) and ensure quality TAs.

Does the design address the needs of the congressional requester?

Will the design yield a quality, independent, balanced, thorough, and objective product?

Will the design likely yield information that will be useful to stakeholders?

Will the design likely yield valid conclusions on the basis of sufficient and credible evidence?

Will the design yield results in the desired time frame?

Will the design likely yield results within the constraints of the resources available?

How will policy options be identified and assessed, if applicable?

2.2 Phases and Considerations for Technology Assessment Design

Figure 1 outlines three phases and seven considerations for TA design. While Figure 1 presents TA design as a series of phases, actual execution is highly iterative and nonlinear. Teams may need to be prepared to re-visit design decisions as information is gathered or circumstances change.

Below are some considerations for the team to think about while designing a TA and throughout the process of performing the TA. This list is not exhaustive, and some of the considerations may not be unique to TAs. of the technology) and context of the technology (such as social, political, legal, and economic factors) circumstances change and new information comes to light, it may be necessary to revisit scope and design.

The initial situational analysis may also be used to: Inform the goal(s), purpose, and objectives (also known as researchable questions) challenges to design and implementation of the TA, such as: (1) possible changes in operating environment; (2) characterizing or quantifying anticipatory factors, uncertainty, and future condition(s); and (3) lack of or limitations with data. See Chapter 3 for more specific examples.

Communication strategy: Consider potential users of the product(s) and how information regarding the TA will be communicated. How results are communicated can affect how they are used, so it is important for TA teams to discuss communication options. statement. TA teams will need to think about whether the initial policy options are appropriate to the size and scope of the TA, as well as whether they are in line with the policy goal and the overall TA purpose and objectives. In keeping with the iterative nature of TA design and execution, any initial policy option list will be revisited, modified, or refined, as needed, as the work progresses and more information is gained. TA teams may also need to plan to include policy analysis and exploration of the ramifications of each policy option during subsequent design and implementation phases.

Phase 2: Develop Initial Design

During this phase, TA teams continue to build on the situational analysis work and gather more background information. In addition, TA teams:

Confirm and validate the scope from phase 1

Reach agreement with stakeholders on the initial design

May perform an “environmental scan” to further highlight limitations, assumptions, divergent points of view, potential bias, and other factors that may help the team select a design Other specific activities that take place during this phase include: Identify and select appropriate design, methodologies, and analytical approaches (refer to the next section of this chapter for example TA design approaches and App. III for examples of TA methods)

Examples of data collection and analytical techniques used in GAO TAs to date include: interviews, literature review, expert forums, site visits, technology readiness assessments, surveys, conceptual models, small group discussion, content analysis such as Delphi, among others. OTA reported using similar methodologies for its TAs (OTA, Policy Analysis at OTA: A Staff Assessment, 1983).

Identify and select appropriate data sources, or the need to gather data Identify, select, and possibly develop appropriate dimensions of analysis, if applicable

Develop possible policy goal(s)

Clarify the possible initial policy options that will be considered and describe how they may be analyzed, if applicable Identify and consult with external experts to inform design and implementation, and assist with external review, as appropriate If policy options are being considered, it is important to determine the relevant dimensions along which to analyze the options. The dimensions will be highly context- specific, vary from TA to TA, and depend on the scope and policy goal statement of the TA.

Phase 3: Implementation of Design

During this phase, the design and project plan are being implemented, potentially while aspects of phase 2 are still underway. It is important to consider changes in the operating context—such as changes in the operating environment, understanding of the issues, and access to information—and review and make changes to the design and project plan accordingly.

We reviewed select GAO products that used policy analysis to present policy options. We found that these products used a variety of data collection and analytical approaches, such as: interviews, literature review, survey, expert forum, site visits, case studies, analysis of secondary data, content analysis, among others.

If an initial policy options list was developed earlier in design, it may be necessary to revisit the list as work progresses. During this phase, TA teams may gather additional information regarding the policy options, further analyze policy options, and present the results of the analysis. Policy options are to be presented in a balanced way, including presentation of opportunities and considerations, and not resulting in a single overall ranking of policy options.

2.2.1 GAO Technology Assessment Design Examples

We found that GAO TAs used a variety of design approaches and methodologies to answer various categories of design objectives (researchable questions). GAO TAs generally include one or more of the following categories of design objectives, which are not mutually exclusive: (1) describe status of and challenges to development of a technology; (2) assess opportunities and challenges arising from the use of a technology; and (3) identify and assess cost-effectiveness, other policy considerations, or options related to the use of a technology. Provided below are example questions, design approaches, and GAO TAs, for each of these categories of objectives. GAO TA examples were used given our familiarity with GAO products, though numerous non-GAO TA design examples exist. This is not intended to be a comprehensive list of design examples. For more examples of methodologies, please refer to App. III.

Describing the status and challenges to the development of a technology. Table 2 provides example questions, design approaches, and GAO TAs, for design objectives related to describing the status and challenges to the development of a technology. Questions may address, for example, what the current state of the technology is, and may involve identifying and describing the status of the technology, which GAO TAs have done using a variety of methods.

Assessing opportunities and challenges that may result from the use of a technology. Table 3 provides example questions, design approaches, and GAO TAs, for design objectives related to assessing opportunities and challenges that may result from the use of a technology. Questions may address, for example, what are the expected or realized benefits of the technology, and may involve gathering and assessing evidence on the results from using the technology, which GAO TAs have done using a variety of methods.

Assessing cost-effectiveness, policy considerations, or policy options related to the use of a technology. Table 4 provides example questions, design approaches, and GAO TAs, for design objectives related to assessing cost-effectiveness, policy considerations, or policy options related to the use of a technology. Questions may address, for example, what are the economic trade-offs of a technology, and may involve gathering and analyzing evidence related to cost, which GAO TAs have done using a variety of methods.

This chapter describes select challenges regarding technology assessment (TA) design and implementation, as well as possible strategies to mitigate those challenges. The information in this chapter is based on review of results of a literature search, an expert forum, select GAO reports, and experiences of GAO teams and technical specialists. The tables provided below are not intended to be a comprehensive list of challenges or strategies. For more information, please refer to Appendix I: Objectives, Scope, and Methodology.

3.1 Ensuring Technology Assessment Products are Useful for Congress and Others

To be useful, TA assessment products must be readable and timely, among other things, which may present a challenge for numerous reasons. Table 5 provides examples of potential mitigation strategies to address these challenges.

3.2 Determining Policy Goals and Measuring Impact

Another challenge in TA design arises from determining policy goals and policy options, and estimating their potential impacts. Many of the effects of policy decisions may be distant, and policy outcomes may be uncertain at the time of the TA. Table 6 provides examples of potential mitigation strategies to address these challenges.

3.3 Researching and Communicating Complicated Issues

TAs are complex and interdisciplinary, and emerging technologies are inherently difficult to assess. Table 7 provides examples of potential mitigation strategies to address these challenges.

3.4 Engaging All Relevant Stakeholders

An additional challenge in conducting TAs is engaging all relevant internal and external stakeholders, ensuring none are overlooked. Table 8 provides examples of potential mitigation strategies to address this challenge.

Appendix I: Objectives, Scope, and Methodology

This handbook identifies key steps and considerations in designing technology assessments (TAs). Below is a summary of methodologies used for all chapters of the handbook.

Review of GAO Documents

We reviewed GAO documents, including:

Designing Evaluations (GAO-12-208G) select GAO products utilizing policy analysis approaches to identify and assess policy options We reviewed and analyzed 14 GAO TAs, including their designs and considerations, using a data collection instrument that contained fields regarding each report’s purpose, methodologies, and key considerations for each methodology used (such as strengths and weaknesses). The data collection instrument also contained fields regarding whether policy considerations were presented or if specific policy options were identified and assessed in each TA report, what methodologies were used to identify and assess policy options, and key considerations associated with the methodologies used.

We also reviewed GAO reports from non-TA product lines that utilized policy analysis approaches to assess policy options. An initial pool of 56 GAO reports was generated based on a keyword search of GAO’s reports database. Of the 56 GAO reports, 12 were selected for review based on the following criteria: (1) the reports were publicly released after January 1, 2013 and (2) the reports included identification and assessment of policy options (not solely a presentation of agency actions related to policy options or general policy considerations). Testimonies and correspondence were excluded. We analyzed each of these selected GAO reports according to a data collection instrument that contained the following fields regarding policy options in the report: purpose, methodologies, and key considerations for each methodology used (such as strengths and weaknesses). A list of GAO documents reviewed is provided below.

GAO Documents Reviewed for Preparing this Handbook

Retirement Security: Some Parental and Spousal Caregivers Face Financial Risks. GAO-19-382. Washington, D.C.: May 1, 2019.

GAO Science Technology Assessment, and Analytics Team: Initial Plan and Considerations Moving Forward. Washington, D.C.: April 10, 2019.

Retirement Savings: Additional Data and Analysis Could Provide Insight into Early Withdrawals. GAO-19-179. Washington, D.C.: March 28, 2019.

Critical Infrastructure Protection: Protecting the Electric Grid from Geomagnetic Disturbances. GAO-19-98. Washington, D.C.: December 19, 2018.

Postal Retiree Health Benefits: Unsustainable Finances Need to Be Addressed. GAO-18-602. Washington, D.C.: August 31, 2018.

Data Collection Seminar Participant Manual. Washington, D.C.: March 2018.

Artificial Intelligence: Emerging Opportunities, Challenges and Implications. GAO-18-142SP. Washington, D.C.: March 28, 2018.

Chemical Innovation: Technologies to Make Processes and Products More Sustainable. GAO-18-307. Washington, D.C.: February 8, 2018.

Federal Regulations: Key Considerations for Agency Design and Enforcement Decisions. GAO-18-22. Washington, D.C.: October 19, 2017.

Medical Devices: Capabilities and Challenges of Technologies to Enable Rapid Diagnoses of Infectious Diseases. GAO-17-347. Washington, D.C.: August 14, 2017.

U.S. Postal Service: Key Considerations for Potential Changes to USPS’s Monopolies. GAO-17-543. Washington, D.C.: June 22, 2017.

Internet of Things: Status and Implications of an Increasingly Connected World. GAO-17-75. Washington, D.C.: May 15, 2017.

Flood Insurance: Comprehensive Reform Could Improve Solvency and Enhance Resilience. GAO-17-425. Washington, D.C.: April 27, 2017.

Flood Insurance: Review of FEMA Study and Report on Community- Based Options. GAO-16-766. Washington, D.C.: August 24, 2016.

Medicaid: Key Policy and Data Considerations for Designing a Per Capita Cap on Federal Funding. GAO-16-726. Washington, D.C.: August 10, 2016.

Municipal Freshwater Scarcity: Using Technology to Improve Distribution System Efficiency and Tap Nontraditional Water Sources. GAO-16-474. Washington, D.C.: April 29, 2016.

GAO Memorandum: Quality Assurance Framework Requirements for Technology Assessments. Washington, D.C.: April 6, 2016.

Biosurveillance: Ongoing Challenges and Future Considerations for DHS Biosurveillance Efforts. GAO-16-413T. Washington, D.C.: February 11, 2016.

Social Security’s Future: Answers to Key Questions. GAO-16-75SP. Washington, D.C.: October 2015.

Water in the Energy Sector: Reducing Freshwater Use in Hydraulic Fracturing and Thermoelectric Power Plant Cooling. GAO-15-545. Washington, D.C.: August 7, 2015.

Nuclear Reactors: Status and Challenges in Development and Deployment of New Commercial Concepts. GAO-15-652. Washington, D.C.: July 28, 2015.

Veterans’ Disability Benefits: Improvements Needed to Better Ensure VA Unemployability Decisions Are Well Supported. GAO-15-735T. Washington, D.C.: July 15, 2015.

Debt Limit: Market Response to Recent Impasses Underscores Need to Consider Alternative Approaches. GAO-15-476. Washington, D.C.: July 9, 2015.

Temporary Assistance for Needy Families: Potential Options to Improve Performance and Oversight. GAO-13-431. Washington, D.C.: May 15, 2013.

Private Pensions: Timely Action Needed to Address Impending Multiemployer Plan Insolvencies. GAO-13-240. Washington, D.C.: March 28, 2013.

Designing Evaluations: 2012 Revision. GAO-12-208G. Washington, D.C.: January 2012.

Neutron Detectors: Alternatives to Using Helium-3. GAO-11-753. Washington, D.C.: September 3, 2011.

Climate Engineering: Technical Status, Future Directions, and Potential Responses. GAO-11-71. Washington, D.C.: July 28, 2011.

Technology Assessment: Explosives Detection Technologies to Protect Passenger Rail. GAO-10-898. Washington, D.C.: July 28, 2010.

Technology Assessment: Protecting Structures and Improving Communications during Wildland Fires. GAO-05-380. Washington, D.C.: April 26, 2005.

Technology Assessment: Cybersecurity for Critical Infrastructure Protection. GAO-04-321. Washington, D.C.: May 28, 2004.

Technology Assessment: Using Biometrics for Border Security. GAO-03-174. Washington, D.C: November 15, 2002.

Review of Experiences of GAO Teams and Technical Specialists

We spoke with and gathered input from GAO teams that are in the process of or have successfully assessed and incorporated policy options into GAO products. In addition, to augment our understanding of TA design and implementation challenges, we collected input from GAO staff who had provided key contributions to GAO TAs. Specifically, we asked for their thoughts regarding: (1) the strengths and limitations of TA methodologies and (2) challenges they faced, and strategies to address those challenges.

Review of Select Office of Technology Assessment Reports

A GAO librarian performed a search for relevant Office of Technology Assessment (OTA) reports, using keyword searches. From this initial list of OTA reports, we selected 17 reports to review that were frameworks, guides, models, or other compilations. We also reviewed the methodologies of the OTA reports selected for review. A list of OTA reports reviewed is included below.

Office of Technology Assessment Reports Reviewed for Preparing this Handbook

Office of Technology Assessment. Insider’s Guide to OTA. Washington, D.C.: January 1995.

Office of Technology Assessment. Policy Analysis at OTA: A Staff Assessment. Washington, D.C.: May 1993.

Office of Technology Assessment. Research Assistants Handbook. Washington, D.C.: June 1992.

Office of Technology Assessment. Strengths and Weaknesses of OTA Policy Analysis. Washington, D.C.: 1992.

Office of Technology Assessment. The OTA Orange Book: Policies and Procedures of the Office of Technology Assessment: Communication with Congress and the Public. Washington, D.C.: February 1986.

Office of Technology Assessment. What OTA Is, What OTA Does, How OTA Works. Washington, D.C.: March 1983.

Office of Technology Assessment. Draft: An OTA Handbook. Washington, D.C.: June 7, 1982.

Office of Technology Assessment. Draft: A Management Overview Methodology for Technology Assessment. Washington, D.C.: February 2, 1981.* Office of Technology Assessment. Draft: Technology Assessment in Industry: A Counterproductive Myth. Washington, D.C.: January 30, 1981.* Office of Technology Assessment. Draft: Technology Assessment Methodology and Management Practices. Washington, D.C.: January 12, 1981.* Office of Technology Assessment. Draft: Technology Assessment in the Private Sector. Washington, D.C.: January 9, 1981.* Office of Technology Assessment. Draft: A Process for Technology Assessment Based on Decision Analysis. Washington, D.C.: January 1981.* Office of Technology Assessment. Draft: Technology as Social Organization. Washington, D.C.: January 1981.* Office of Technology Assessment. A Summary of the Doctoral Dissertation: A Decision Theoretic Model of Congressional Technology Assessment. Washington, D.C.: January 1981.* Office of Technology Assessment. Report on Task Force Findings and Recommendations: Prepared by the OTA Task Force on TA Methodology and Management. Washington, D.C.: August 13, 1980.

Office of Technology Assessment. Phase I Survey Results: Draft Papers Prepared for the Task Force on TA Methodology and Management. Washington, D.C.: April 10, 1980.

Review of Select Congressional Research Service Reports

We identified a pool of 29 Congressional Research Service (CRS) reports to consider reviewing that were technology assessments or included an analysis of policy options, based on a keyword search of CRS’s website. We also interviewed CRS officials. Of the initial 29 CRS reports we identified, we selected six CRS reports to review, based on the following criteria: (1) published within the past 15 years (2004-2019) and (2) if a review of technology (technology assessment) and/or policy options was included. Reports were excluded based on the following criteria: (1) for technology assessment related reports—if they represented a summary of a technology assessment that was included in our review or (2) for policy options related reports—the report did not indicate how CRS arrived at the policy options (no methodology to review or analyze). A list of CRS reports reviewed is included below.

Congressional Research Service Reports Reviewed for Preparing this Handbook

Congressional Research Service. Advanced Nuclear Reactors: Technology Overview and Current Issues. Washington, D.C.: April 18, 2019.

Congressional Research Service. Drug Shortages: Causes, FDA Authority, and Policy Options. Washington, D.C.: December 27, 2018.

Congressional Research Service. Policy Options for Multiemployer Defined Benefit Pension Plans. Washington, D.C.: September 12, 2018.

Congressional Research Service. Shale Energy Technology Assessment: Current and Emerging Water Practices. Washington, D.C.: July 14, 2014.

Congressional Research Service. Carbon Capture: A Technology Assessment. Washington, D.C.: November 5, 2013.

Congressional Research Service. Energy Storage for Power Grids and Electric Transportation: A Technology Assessment. Washington, D.C.: March 27, 2012.

Review of Literature

A GAO librarian performed a literature search based on keyword searches for two areas—TA and policy options. For TA literature, the team selected 29 documents to review that were frameworks, guides, models, or other compilations, based on a review of the literature titles and abstracts. In general, we excluded specialized types of TAs, such as health-related TAs, as we focused on TA design more broadly. For policy options literature, the team selected 14 documents to review that were frameworks, guides, models, or other compilations and focused on policy options related to science and technology. We also asked experts we consulted to suggest literature for our review; these suggestions confirmed the literature list noted below. A list of literature reviewed is included below.

Literature Reviewed for Preparing this Handbook

Grunwald, Armin. Technology Assessment in Practice and Theory. London and New York: Routledge, 2019.

Armstrong, Joe E., and Willis W. Harman. Strategies For Conducting Technology Assessments. London and New York: Routledge, 2019.

Noh, Heeyong, Ju-Hwan Seo, Hyoung Sun Yoo, and Sungjoo Lee. “How to Improve a Technology Evaluation Model: A Data-driven Approach.” Technovation, vol. 72/73 (2018): p. 1-12.

Larsson, A., T. Fasth, M. Wärnhjelm, L. Ekenberg, and M. Danielson. “Policy Analysis on the Fly With an Online Multicriteria Cardinal Ranking Tool.” Journal of Multi-Criteria Decision Analysis, vol. 25 (2018): p. 55-66.

Nooren, P., N. van Gorp, N. van Eijk, and R. O. Fathaigh. “Should We Regulate Digital Platforms? A New Framework for Evaluating Policy Options.” Policy and Internet, vol. 10, no. 3 (2018): p. 264-301.

Smith, A., K. Collins, and D. Mavris. “Survey of Technology Forecasting Techniques for Complex Systems.” Paper presented at 58th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Grapevine, TX (2017).

Ibrahim, O., and A. Larsson. “A Systems Tool for Structuring Public Policy Problems and Design of Policy Options.” Int. J. Electronic Governance, vol. 9 , nos. 1/2 (2017): p. 4-26.

Christopher, A. Simon. Public Policy Preferences and Outcomes. 3rd ed. New York: Routledge, 2017.

Weimer, David L., and R. Aidan Vining. Policy Analysis Concepts and Practice. 6th ed. London and New York: Routledge, 2017.

Mulder, K. “Technology Assessment.” In Foresight in Organizations: Methods and Tools, edited by Van Der Duin, Patrick, 109-124, 2016.

Coates, Joseph F. “A 21st Century Agenda for Technology Assessment.” Technological Forecasting and Social Change, vol. 113 part A (2016): p. 107-109.

Coates, Joseph F. “Next Stages in Technology Assessment: Topics and Tools.” Technological Forecasting and Social Change, vol. 113 (2016): p. 112-114.

Mazurkiewicz, A., B. Belina, B. Poteralska, T. Giesko, and W. Karsznia. “Universal Methodology for the Innovative Technologies Assessment.” Proceedings of the European Conference on Innovation and Entrepreneurship (2015): p. 458-467.

Sadowski, J. “Office of Technology Assessment: History, Implementation, and Participatory Critique.” Technology in Society, vol. 42 (2015): p. 9-20.

Larsson, A., O. Ibrahim. “Modeling for Policy Formulation: Causal Mapping, Scenario Generation, and Decision Evaluation.” In Electronic Participation: 7th IFIP 8.5 International Conference, 135-146, Springer, 2015.

Moseley, C., H. Kleinert, K. Sheppard-Jones, and S. Hall. “Using Research Evidence to Inform Public Policy Decisions.” Intellectual and Developmental Disabilities, vol. 51 (2013): p. 412-422.

Calof, J., R. Miller, and M. Jackson. “Towards Impactful Foresight: Viewpoints from Foresight Consultants and Academics.” Foresight, vol. 14 (2012): p. 82-97.

Parliaments and Civil Society in Technology Assessment, Collaborative Project on Mobilization and Mutual Learning Actions in European Parliamentary Technology Assessment. The Netherlands: Rathenau Instituut, 2012.

Blair, P. D. “Scientific Advice for Policy in the United States: Lessons from the National Academies and the Former Congressional Office of Technology Assessment.” In The Politics of Scientific Advice: Institutional Design for Quality Assurance, ed. Lentsch, Justus, 297-333, 2011.

Paracchini, M.L., C. Pacini, M.L.M. Jones, and M. Pérez-Soba. “An Aggregation Framework to Link Indicators Associated With Multifunctional Land Use to the Stakeholder Evaluation of Policy Options.” Ecological Indicators, vol. 11 (2011): p 71-80.

Roper, A. T., S. W. Cunningham, A. L. Porter, T. W. Mason, F. A. Rossini, and J. Banks. Forecasting and Management of Technology, 2nd ed. New Jersey: Wiley, 2011.

Lepori, B., E. Reale, and R. Tijssen. “Designing Indicators for Policy Decisions: Challenges, Tensions and Good Practices: Introduction to a Special Issue.” Research Evaluation, vol. 20, no. 1 (2011): p. 3-5.

Russel, A. W., F. M. Vanclay, and H. J. Aslin H.J. “Technology Assessment in Social Context: The Case for a New Framework for Assessing and Shaping Technological Developments.” Impact Assessment and Project Appraisal, vol. 28, no. 2 (2010): p. 109-116.

Shiroyama, H., G. Yoshizawa, G., M. Matsuo, and T. Suzuki. “Institutional Options and Operational Issues in Technology Assessment: Lessons from Experiences in the United States and Europe.” Paper presented at Atlanta Conference on Science and Innovation Policy, Atlanta, 2009.

Tran, T.A., and T. Daim T. “A Taxonomic Review of Methods and Tools Applied in Technology Assessment.” Technological Forecasting and Social Change, vol. 75 (2008): p. 1396-1405.

Brun, G., and G. Hirsch Hadorn. “Ranking Policy Options for Sustainable Development.” Poiesis Prax, vol. 5 (2008): p. 15-31.

Tran, T.A. “Review of Methods and Tools applied in Technology Assessment Literature.” Paper presented at Portland International Conference on Management of Engineering and Technology, Portland Oregon, 2007.

Burgess, J., A. Stirling, J. Clark, G. Davies, M. Eames, K. Staley, and S. Williamson. “Deliberative Mapping: A Novel Analytic-Deliberative Methodology to Support Contested Science-Policy Decisions.” Public Understanding of Science, vol. 16 (2007): p. 299-322.

Decker, M., and M. Ladikas. Bridges Between Science, Society and Policy: Technology Assessment — Methods and Impacts. Berlin: Springer-Verlag, 2004.

Guston, D. H., and D. Sarewitz. “Real-time Technology Assessment.” Technology in Society, vol. 24 (2002): p. 93-109.

Rip, A. “Technology Assessment.” In International Encyclopedia of the Social & Behavioral Science, vol. 23, edited by Smelster, N. J. and B. P. Baltes, 15512-15515. Amsterdam: Elsevier, 2001.

Van Den Ende, J., K. Mulder, M. Knot, E. Moors, and P. Vergragt. “Traditional and Modern Technology Assessment: Toward a Toolkit.” Technological Forecasting and Social Change, vol. 58 (1998): p. 5-21.

Wood, F. B. “Lessons in Technology Assessment: Methodology and Management at OTA.” Technological Forecasting and Social Change, vol. 54 (1997): p. 145-162.

Janes, M. C. “A Review of the Development of Technology Assessment.” International Journal of Technology Management, vol. 11, no. 5-6 (1996): p. 507-522.

Hastbacka, M. A., and C. G. Greenwald. “Technology Assessment - Are You Doing it Right?” Arthur D. Little – PRISM, no. 4 (1994).

Rivera, W. M., D. J. Gustafson, and S. L. Corning. “Policy Options in Developing Agricultural Extension Systems: A Framework for Analysis.” International Journal of Lifelong Education, vol. 10, no. 1 (1991): p. 61-74.

Lee, A. M., and P. L. Bereano. “Developing Technology Assessment Methodology: Some Insights and Experiences.” Technological Forecasting and Social Change, vol. 19 (1981): p. 15-31.

Porter, A. L., F. A. Rossini, S. R. Carpenter, and A. T. Roper. A Guidebook for Technology Assessment and Impact Analysis, vol. 4. New York and Oxford: North Holland, 1980.

Pulver, G.C. “A Theoretical Framework for the Analysis of Community Economic Development Policy Options.” In Nonmetropolitan Industrial Growth and Community Change, edited by Summers, G. and A. Selvik, 105-117. Massachusetts and Toronto: Lexington Books, 1979.

Ascher, W. “Problems of Forecasting and Technology Assessment.” Technological Forecasting and Social Change, vol. 13, no. 2 (1979): p. 149-156.

Majone, G. “Technology Assessment and Policy Analysis.” Policy Sciences, vol. 8, no. 2 (1977): p. 173-175.

Berg, M., K. Chen, and G. Zissis. “A Value-Oriented Policy Generation Methodology for Technology Assessment.” Technological Forecasting and Social Change, vol. 4, no. 4 (1976): p. 401-420.

Lasswell, Harold D. A Pre-View of Policy Sciences. Policy Sciences Book Series. New York: Elsevier, 1971.

Consultation with External Experts

We held a forum to gather experts’ opinions regarding TA design. An initial list of experts was prepared based on a review of GAO TA reports, literature, and referral by other experts. Experts were selected based on their knowledge and expertise in the subject, including: (1) prior participation on a National Academy of Sciences panel or other similar meeting; (2) leadership position in one or more organizations or sectors relevant to technology research and development implementation or policy; and (3) relevant publications or sponsorship of reports. Care was also taken to ensure a balance of sectors, backgrounds, and specific areas of expertise (e.g., science, technology, policy, information technology, and law). We also asked the experts to suggest literature for our review; these suggestions confirmed the literature list noted above. A list of external experts consulted is included below.

External Experts Consulted for the Handbook

Dr. Jeffrey M. Alexander, Senior Manager, Innovation Policy, RTI International Dr. Robert D. Atkinson, President, Information Technology and Innovation Foundation Mr. David Bancroft, Executive Director, International Association for Impact Assessment Mr. Duane Blackburn, S&T Policy Analyst, Office of the CTO, MITRE Dr. Peter D. Blair, Executive Director, Division of Engineering and Physical Sciences, National Academies of Sciences, Engineering, and Medicine Ms. Marjory Blumenthal, Acting Associate Director, Acquisition and Technology Policy Center; Senior Policy Researcher, RAND Corporation Mr. Chris J. Brantley, Managing Director, Institute of Electrical and Electronics Engineers, Inc., USA Dr. Jonathan P. Caulkins, H. Guyford Stever University Professor of Operations Research and Public Policy, Carnegie Mellon University Mr. Dan Chenok, Executive Director, Center for The Business of Government, IBM Dr. Gerald Epstein, Distinguished Research Fellow, Center for the Study of Weapons of Mass Destruction, National Defense University Dr. Robert M. Friedman, Vice President for Policy and University Relations, J. Craig Venter Institute Mr. Zach Graves, Head of Policy, Lincoln Network Ms. Allison C. Lerner, Inspector General, National Science Foundation Mr. Mike Molnar, Director of Office of Advanced Manufacturing, National Institute of Standards and Technology Dr. Michael H. Moloney, CEO, American Institute of Physics Dr. Ali Nouri, President, Federation of American Scientists Dr. Jon M. Peha, Professor, Engineering and Public Policy; Courtesy Professor, Electrical and Computer Engineering, Carnegie Mellon University Dr. Stephanie S. Shipp, Deputy Director and Professor, University of Virginia, Biocomplexity Institute and Initiative, Social and Decision Analytics Division Dr. Daniel Sarewitz, Co-Director, Consortium for Science, Policy & Outcomes Professor of Science and Society, School for the Future of Innovation in Society, Arizona State University Ms. Rosemarie Truman, Founder and CEO, Center for Advancing Innovation Dr. Chris Tyler, Director of Research and Policy, Department of Science, Technology, Engineering and Public Policy (STEaPP), University College London (UCL)

Appendix II: Summary of Steps for GAO’s General Engagement Process

As part of GAO’s Quality Assurance Framework, GAO’s general design and project plan templates contain five phases that are followed in sequential order, with modifications or changes as needed. GAO technology assessments (TAs) use these templates, as applicable. Throughout the phases, the status of the work, including decisions, is communicated to stakeholders and congressional committees that requested the work. Provided below is a summary of the activities GAO staff undertake during each of the phases, and is based on a review of GAO documentation related to engagement phases.

Phase I: Acceptance

Engagement characteristics such as risk level or internal stakeholders are determined at a high-level Engagement Acceptance Meeting.

Engagement teams obtain a copy of and review the congressional request letter(s), as applicable.

Phase II: Planning and Proposed Design

Staff are assigned to the engagement and set up the electronic engagement documentation set folders.

Staff enter standard information regarding the engagement in GAO’s Engagement Management System (EMS), which is used to monitor the status of the engagement throughout the engagement process and regularly updated.

Engagement teams hold an initiation meeting with engagement stakeholders to discuss potential research questions, design options, and stakeholder involvement.

Engagement teams clarify engagement objectives and approach through discussions with the congressional requesters, as applicable.

Engagement teams obtain background information. For example, to gather information about the topic and any work already performed, teams may conduct a literature review, search prior and ongoing GAO work related to the topic, or consult with external stakeholders, outside experts, and agency officials, including the Congressional Research Service, Congressional Budget Office, and Inspectors General of federal agencies.

Engagement teams formally notify agencies of the engagement through a notification letter, and hold an entrance conference, as applicable.

Engagement teams prepare a design matrix, project plan, risk assessment tool, data reliability assessment, and all participants on engagements, including stakeholders, affirm their independence. The design matrix is a tool that describes: researchable questions; criteria; information required and sources; scope and methodology; and limitations. The project plan identifies key activities and tasks, dates for completing them, and staff assigned.

Engagement teams secure approval to move forward with engagement approach at a high-level Engagement Review Meeting.

Phase III: Evidence Gathering, Finalizing Design, and Analysis

Engagement teams finalize design: teams work with internal stakeholders to confirm soundness and reach agreement on proposed initial design. If engagement teams and stakeholders conclude that additional work is needed or the design faces significant implementation challenges, design is reviewed and modified, as needed.

Engagement teams collect and analyze evidence: teams may collect and analyze evidence using a variety of methodologies including document review, interviews, surveys, focus groups, and various forms of data analysis. For example, engagement teams may meet with agency officials and outside experts, as applicable, to gather evidence.

Engagement teams assess evidence and agree on conclusions: teams assess whether the evidence collected is sufficient and appropriate to support findings and conclusions reached for each objective. Once sufficient evidence is collected and analyzed, the team discusses how the evidence supports potential findings and shares these findings with stakeholders, generally in the form of a formal message agreement meeting.

Engagement teams update congressional requesters, as applicable, on the engagement status and potential findings.

Phase IV: Product Development

Engagement teams draft product: after drafting the product, teams send draft to internal stakeholders for review. Teams also send draft to relevant external parties, including relevant agencies, to confirm facts and obtain their views.

Teams identify sources of all information in the draft and an independent analyst (not on the team) verifies the sources through a process called indexing and referencing.

Engagement teams perform exit conferences with agencies, as applicable, to discuss findings and potential recommendations. Agencies and external parties are given the opportunity to comment on the draft, as applicable.

Engagement teams communicate findings and potential recommendations, as well as timeframes for issuing the product, to congressional requesters, as applicable.

The draft product is copy-edited, prepared for issuance, and publicly released on GAO’s website, as applicable.

Phase V: Results

Engagement documentation is closed out.

Engagement teams conduct follow-up, track the results, and prepare reports on the status of recommendations and financial and non-financial benefits, as applicable, using GAO’s results tracking system.

Appendix III: Example Methods for Technology Assessment

This appendix provides examples of methods and analytical approaches that GAO technology assessment (TA) teams can use to examine different types of evidence. Also included in this appendix are considerations of the strengths, limitations, and synergies among evidence types and methods, which can be useful to consider throughout design to ensure that evidence is sufficient and appropriate to answer the researchable questions. Examples from GAO TAs were used given our familiarity with GAO products, though numerous other (non-GAO) examples of TA methods exist. This appendix included a review of GAO reports and select literature, and is not intended to be comprehensive. This is a simplified presentation of methods, and there is variation in the levels of structure of the example methods.

This appendix is divided into several sections, including by evidentiary types: Testimonial, Documentary, and Physical. For each of these types of evidence, example methods are presented with low and high levels of structure, and include examples of considerations (such as general benefits and limitations) that analysts may consider. In general, more highly structured approaches generate increased consistency and comparability of results that allows for stronger quantification. Less structured approaches tend to provide more flexibility and context, and richer illustrative evidence.

Examples of Methodologies for Testimonial Evidence

Testimonial evidence is elicited from respondents to understand their experience, opinions, knowledge, and behavior, and it can be obtained through a variety of methods, including inquiries, interviews, focus groups, expert forums, or questionnaires. Testimonial evidence can be gathered from individuals who may be responding personally based on their own experience in an official capacity to represent agencies or other entities, or groups, who may share individual level responses, or may present a single group response. Group testimony enables interactions that can be used to explore similarities and differences among participants, to identify tensions or consensus in a group, or to explore ideas for subsequent research and collaboration. It is important to evaluate the objectivity, credibility, and reliability of testimonial evidence. Analysts may use a combination of approaches to gather testimonial evidence, depending on the relevant population(s) of respondents, intended analytical approach(es), likely respondent burden, and resource considerations. Table 9 provides more examples.

Examples of Methodologies for Documentary Evidence

Documentary evidence is existing information, such as letters, contracts, accounting records, invoices, spreadsheets, database extracts, electronically stored information, and management information on performance. It is important to evaluate the objectivity, credibility, and reliability of documentary evidence. Analysts may use a combination of approaches to gather documentary evidence, depending on the relevant sources and types of documents, intended analytical approach(es), and resource considerations. Table 10 provides more examples.

Examples of Methodologies for Physical Evidence

Physical evidence is obtained by direct inspection or observation of people, property, or events. The appropriateness of physical evidence depends on when, where, and how the inspection or observation was made and whether it was recorded in a manner that fairly represents the facts observed. Common considerations for physical evidence include the reliability of site selection, intended analytical approaches, and resource considerations. Table 11 provides more examples.

GAO may also rely on agency and other secondary data. Considerations for those secondary data are dependent on the type, source, and collection method, and could include all of the considerations above. Use of secondary data is usually more efficient than collecting new data on a topic, and administrative records (a form of documentary evidence) are generally not as prone to self-reporting biases that may be present in testimonial evidence. However, when secondary data are used, more work may be required to assess whether data are reliable and appropriate for a given purpose. For example, analysts will gather all appropriate documentation, including record layout, data element dictionaries, user’s guides, and data maintenance procedures. Depending on the database, procedures and analysis can be very complex—and it would be important to note assumptions, limitations, and caveats pertaining to the data, which may affect the conclusions that can be drawn based on the analyses.

Examples of Analytical Approaches

Examples of analytical approaches found in the literature to analyze data include: Interpretive structural modeling: shows a graphical relationship among all elements to aid in structuring a complex issue area, and may be helpful in delineating scope.

Trend extrapolation: is a family of techniques to project time-series data using specific rules, and may be helpful in forecasting technology.

Scenarios: is a composite description of possible future states incorporating a number of characteristics, and may be helpful in policy analysis.

Scanning methods, such as checklists: is listing factors to consider in a particular area of inquiry, and may be helpful in identifying potential impacts.

Tracing methods, such as relevance trees: includes identifying sequential chains of cause and effect or other relationships, and may be helpful in identifying potential impacts.

Cross-effect matrices: are two-dimensional matrix representations to show the interaction between two sets of elements, and may be helpful in analyzing consequences of policy options.

Simulation models: are a simplified representation of a real system that is used to explain dynamic relationships of the system, and may be helpful in identifying impacts and forecasting technology.

Benefit-cost analysis: is a systematic quantitative method of assessing the desirability of government projects or policies when it is important to take a long view of future effects and a broad view of possible side effects.

Decision analysis: is an aid to compare alternatives by weighing the probabilities of occurrences and the magnitudes of their impacts, and may be helpful in determining impacts and assessing policy options.

Scaling: is an aid that may include developing a matrix that identifies potential impact related to an activity and stakeholder group, and qualitatively or quantitatively assesses the potential impact, and may be helpful analyzing potential impacts, including of policy options.

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contacts named above, key contributors to this report were R. Scott Fletcher (Assistant Director), Diantha Garms (Analyst-in- charge), Nora Adkins, Colleen Candrl, Virginia Chanley, Robert Cramer, David Dornisch, John De Ferrari, Dennis Mayo, Anika McMillon, SaraAnn Moessbauer, Amanda Postiglione, Steven Putansu, Oliver Richard, Meg Tulloch, Ronald Schwenn, Ben Shouse, Amber Sinclair, Ardith Spence, Andrew Stavisky, David C. Trimble, and Edith Yuh.

GAO’s Mission

The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony Order by Phone

The fastest and easiest way to obtain copies of GAO documents at no cost is through our website. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. You can also subscribe to GAO’s email updates to receive notification of newly posted products.

The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm.

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov.

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs

Strategic Planning and External Liaison

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Please Print on Recycled Paper.