The Joint Institute for Strategic Energy Analysis is operated by the Alliance 

for Sustainable Energy, LLC, on behalf of the U.S. Department of Energy’s 

National Renewable Energy Laboratory, the University of Colorado-Boulder, 

the Colorado School of Mines, the Colorado State University, the 

Massachusetts Institute of Technology, and Stanford University.

Contract No. DE-AC36-08GO28308

Multi-Metric Sustainability 

Analysis

Shannon Cowlin, Donna Heimiller, 

Jordan Macknick, Margaret Mann, and 

Jacquelyn Pless

National Renewable Energy Laboratory

David Munoz

Colorado School of Mines

Technical Report

NREL/TP-6A50-61719

December 2015
The Joint Institute for Strategic Energy Analysis is operated by the Alliance 

for Sustainable Energy, LLC, on behalf of the U.S. Department of Energy’s 

National Renewable Energy Laboratory, the University of Colorado-Boulder, 

the Colorado School of Mines, the Colorado State University, the 

Massachusetts Institute of Technology, and Stanford University.

JISEA® and all JISEA-based marks are trademarks or registered trademarks of the Alliance for 

Sustainable Energy, LLC.

The Joint Institute for

Strategic Energy Analysis

15013 Denver West Parkway

Golden, CO 80401

303-275-3000 • www.jisea.org Contract No. DE-AC36-08GO28308

Multi-Metric Sustainability 

Analysis 

Shannon Cowlin, Donna Heimiller, 

Jordan Macknick, Margaret Mann, and

Jacquelyn Pless 

National Renewable Energy Laboratory

David Munoz

Colorado School of Mines

Prepared under Task Nos. 6A50.1010 and 6A50.3010

Technical Report

NREL/TP-6A50-61719

December 2014
NOTICE

This report was prepared as an account of work sponsored by an agency of the United States government. 

Neither the United States government nor any agency thereof, nor any of their employees, makes any warranty, 

express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of 

any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately 

owned rights. Reference herein to any specific commercial product, process, or service by trade name, 

trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, 

or favoring by the United States government or any agency thereof. The views and opinions of authors 

expressed herein do not necessarily state or reflect those of the United States government or any agency thereof.

Available electronically at http://www.osti.gov/bridge

Available for a processing fee to U.S. Department of Energy

and its contractors, in paper, from:

U.S. Department of Energy

Office of Scientific and Technical Information

P.O. Box 62

Oak Ridge, TN 37831-0062

phone: 865.576.8401

fax: 865.576.5728

email: mailto:reports@adonis.osti.gov

Available for sale to the public, in paper, from:

U.S. Department of Commerce

National Technical Information Service

5285 Port Royal Road

Springfield, VA 22161

phone: 800.553.6847

fax: 703.605.6900

email: orders@ntis.fedworld.gov

online ordering: http://www.ntis.gov/help/ordermethods.aspx

Cover Photos: (left to right) PIX 04135, iStock 22779761, PIX 16933., PIX 15648, PIX 08466, PIX 21205

NREL prints on paper that contains recycled content.
iii

Table of Contents

Introduction..................................................................................................................................................1

Past Approaches to Multi-Metric Sustainability Analysis and Motivation for a New Tool...................1

Proposed Multi-Metric Sustainability Approach ......................................................................................3

Establishing the MMSA Indicators.........................................................................................................5

Energy Affordability......................................................................................................................6

Energy Diversity............................................................................................................................6

Climate Friendliness......................................................................................................................7

Water Conservation .......................................................................................................................7

Safety ..........................................................................................................................................8

Local Employment Impacts...........................................................................................................8

Applying the MMSA Visual to Location and Technology Options .........................................................9

Analysis of MMSA Indicators at Potential Deployment Sites..............................................................11

Energy Affordability....................................................................................................................11

Energy Diversity..........................................................................................................................12

Climate Friendliness....................................................................................................................13

Water Conservation .....................................................................................................................14

Safety ........................................................................................................................................16

Local Employment Impacts.........................................................................................................16

MMSA Analysis and Presentation........................................................................................................17

Discussion..................................................................................................................................................19

References .................................................................................................................................................20

List of Figures

Figure 1. In this proposed visual tool for MMSA, larger ruler lengths represent more positive 

outcomes...................................................................................................................................3

Figure 2. Conceptual comparison using the visual tool for MMSA..............................................................4

Figure 3. Extraction areas used for the multi-metric sustainability analysis...............................................10

Figure 4. Water availability by source for six study regions.......................................................................14

Figure 5. Water consumption by site for utility-scale PV installation ........................................................15

Figure 6. Water consumption impacts of a variety of electricity generating technologies .........................16

Figure 7. MMSA visual showing case study findings.................................................................................18

List of Tables

Table 1. Potential Indicators for Use in Multi-Metric Sustainability Analysis.............................................4

Table 2. Features of the Multi-metric Sustainability Analysis Case Study Sites........................................11

Table 3. Technology Costs Used to Calculate LCOE .................................................................................12

Table 4. LCOE for Onshore Wind and Solar PV at Six Case Study Sites..................................................12

Table 5. Potential Solar and Wind Generation on Case Study Sites...........................................................13

Table 6. Baseline Diversity Indicator for Each Case Study Site and the Percent Change in 

Diversity Indicator under Wind and Solar Project Development Scenarios...........................13

Table 7. Local Employment Impacts from Deployment of Wind and Solar Projects at Six Land 

Parcels.....................................................................................................................................17

Table 8. Summary of MMSA Indicators, Metrics, and Range of Values ...................................................17

Table 9. Metric Values for Each Scenario and Land Parcel........................................................................18
iv

Acknowledgments

The authors thank the Joint Institute for Strategic Energy Analysis for funding this 

research through its Innovative Research Analysis Award Program. They also appreciate 

the support and assistance of Patricia Statwick, Yimin Zhang, Jaquelin Cochran, Scott

Gossett, Kendra Palmer, Karin Haas, and Dean Armstrong. 
1

Introduction

Common metrics for comparing energy technologies—such as the levelized cost of 

energy (LCOE), greenhouse gas (GHG) emission profiles, and criteria air pollutant 

emissions—are instructive, but they are not always sufficient on their own for informing 

policy or investment decisions. For instance, although costs per unit of output (levelized 

costs) can provide a baseline guide and encourage or redirect investment (Gross et al., 

2009), they are less applicable in markets characterized by a diverse range of technology 

options and an uncertain future (Awerbuch, 2004). There are often many other factors to 

consider when seeking to fully explore the potential impacts of an energy project. Policy 

and investment decisions often take into account a combination of social, environmental,

and economic factors that vary in priority by jurisdiction. Use of a readily accessible 

framework that allows for evaluating impacts and comparing tradeoffs among these 

factors in energy policy, expansion planning, and investment decision-making is needed. 

Recognizing this, the Joint Institute for Strategic Energy Analysis (JISEA) funded this 

exploration of multi-metric sustainability analysis (MMSA) to provide energy decision 

makers with a means to make more comprehensive comparisons of energy technologies, 

with a focus on the policymaker’s perspective.

The resulting MMSA tool allows decision makers to simultaneously compare energy 

technologies and potential deployment locations according to several sustainability

indicators. For simplicity, the project team chose six possible indicators—two social, two 

environmental, and two economic indicators. However, the designers intend for this 

visual comparison tool to be scalable and adaptable to unique local needs. Similarly, the 

tool is designed so that trade-offs between technology and locations options can be easily 

interpreted. 

This analysis reviews the state of MMSA, explains the methodology for developing the 

new MMSA tool, tests the concept by evaluating two energy technologies at six different 

U.S. locations, and discusses potential directions for additional work.

Past Approaches to Multi-Metric Sustainability 

Analysis and Motivation for a New Tool

Previous efforts to provide comprehensive assessments of energy development options

have primarily analyzed historic energy installations or system-wide sustainability. One

set of studies assessed the performance of energy projects on various sustainability scales

through retrospective investigation (Begic and Afgan 2007). Evans et al. (2009) 

evaluated the sustainability of renewable power generation technologies by qualitatively 

assessing performance against a range of sustainability indicators, translating those to a 1 

to 4 scale and summing over all categories for a total sustainability ranking. While these 

system-wide assessments can provide insight into the general nature of technologies and 

systems, and they can be perhaps adapted for prospective evaluation, they cannot be 

directly translated into impacts for specific proposed projects.
2

Other research has applied multi-criteria analysis, using normalization and weighting to 

aggregate findings into one metric (Lior 2007; Afgan et al. 2000; Afgan and Carvalho

2002; Pehnt 2006). The resulting ordered ranking of options allows stakeholders to 

choose the “best” options. While the ranked list is straightforward and easy to 

understand, aggregated indicators do not allow for comparisons of trade-offs, nor do they

reflect differing value judgments and prioritization of individual metrics by various

stakeholders. In other words, using a single metric to evaluate energy development 

options does not allow for comprehensive consideration of other potential project costs 

and benefits that may be important, such as environmental, social, and economic impacts, 

and trade-off analysis.

Other approaches to measuring project sustainability have been developed, particularly 

with the use of spider diagrams. For instance, in the mid-1990s, Elkington (1994) built a 

framework for measuring performance in corporate America. This analysis was termed 

the triple bottom line and extended the traditional business performance measures, such 

as return on investment and profits, to environmental and social dimensions. The triple 

bottom line focuses on business-oriented applications. However, a similar approach can 

be used to assess energy project development along environmental, social, and economic 

impacts, and other measurements of concern to policymakers and other stakeholders (i.e., 

local employment impacts, climate friendliness). 

Convertino et al. (2013) propose the use of multi-criteria decision analysis methods to 

select optimal metric sets, but they focus on design and monitoring of sustainable 

ecosystem restorations as opposed to energy technologies.

One sophisticated use of the spider diagram methodology is the Arup SPeAR model

(Arup 2014), which applies the United Kingdom’s Government’s Sustainable 

Development Indicators and the Global Reporting Initiative (GRI) indicators within its 

project appraisal framework. Spear has been modified to be applicable at an international 

scale. The method incorporates a color scheme for varying degrees of indicator 

magnitude, and it offers a unique visual for communicating results. Furthermore, a 

framework for integrated sustainability analysis developed at the University of Sydney 

comprises a range of environmental, economic, and social indicators as a triple bottom 

line analysis illustrated in a spider diagram (Dey n.d.). 

The range of tools developed in this field reflects the complexity associated with 

measuring sustainability. The objective of this research is to propose an option that is 

particularly intuitive—offering the benefit of ease of interpretation. Furthermore, the 

proposed methodology is unique in that it allows the user to readily and consistently

compare technology and location options for a given project. By presenting multiple 

dimensions of sustainability simultaneously and in an easily understandable manner, this 

visual will offer decision-makers an option for quickly evaluating trade-offs, and in a way 

that is adaptable to a wide variety of scenarios and priorities.

The MMSA is designed to offer a readily-accessible and adaptable framework for 

comparing tradeoffs. Therefore, the goal of this pilot analysis is to develop a tool that 

allows policymakers to assess an array of energy project development options within the 
3

context of their unique local needs and priorities, and to demonstrate its application in a 

specific case study.

Proposed Multi-Metric Sustainability Approach

As noted, one objective of this study was to develop a means of presenting multiple 

indicators in a disaggregated format for easy interpretation by a diverse stakeholder 

group. Given that the sustainability indicators may vary in priority from one case to the 

next, a second objective was for the approach to be adaptable to a wide variety of 

scenarios. 

Figure 1 illustrates the design selected for further evaluation. Each indicator is presented 

as a ruler, where a longer ruler indicates a more positive outcome—offering the ease of a 

“bigger is better” type of interpretation. Indicators from each sustainability category 

(social, environmental, and economic) are included, and indicators within categories are 

adjacent and presented with similar color schemes. This graphic allows for straightforward visual comparisons of trade-offs faced when considering specific projects and 

technology options. The graphic could also be used to compare different project scenarios 

at one or many locations, as shown in Figure 2. 

The indicators selected for this analysis were selected based on a policymaker’s 

perspective. For instance, policymakers are likely to be concerned about employment 

effects of a project (and thus it is included as criteria here), however project developers 

may be less likely to consider employment in their project investment assessments. To 

account for varying priorities, a different set of potential indicators can be considered 

when assessing technology options.

Figure 1. In this proposed visual tool for MMSA, longer ruler lengths represent more 

positive outcomes
4

Figure 2. Conceptual comparison using the visual tool for MMSA

Figure 1 presents six sustainability indicators that could have relevance for an energy 

project that is considering tradeoffs among three technologies and two locations. There 

are many more indicators that might be of interest and importance to various 

stakeholders. A broader list of potential indicators is presented in Table 1. Note that some 

of the indicators, such as employment and industry expansion, are categorized as 

economic indicators, but these may be relevant social indicators as well if local impacts 

are considered (rather than strictly macro-economic impacts). 

Table 1. Potential Indicators for Use in Multi-Metric Sustainability Analysis

Economic Environmental Social

Cost of energy

Employment 

Industry expansion 

Trade impacts

Energy imports and security

Market demand

Climate resilience

GHG emissions

Biodiversity 

Water use and impact

Air quality

Land use impacts

Soil health

Health impacts

Education opportunities and 

needs

Gender impacts 

Rural development 

Energy access 

Safety and security

Energy security

Food security

Cultural preservation

The proposed visual presents very different indicators in the same manner, which means 

that each indicator has a customized axis based on the metric chosen to represent that 

indicator. To ensure cross-comparability, axis minimums and maximums are established 

so that they correspond to the worst and best possible expected outcomes, respectively,

for both fossil and renewable technologies. In some cases, the indicators may seem 
5

counter-intuitive: instead of larger indicator values always representing a more favorable 

result, smaller indicator values may represent the best possible expected outcome. For 

example, LCOE values are more favorable for Energy Affordability when these values 

are smaller, indicating a lower cost for a given unit of energy output. Thus the smallest 

possible LCOE value represents the best-expected outcome, and accordingly, is shown as 

the axis maximum. Care must be taken to understand each axis’ underlying metric.

Ideally, all indicators should be assessed quantitatively. However, some variables, 

including some social variables, are inherently more difficult to quantify than others

(Francis n.d.). In these cases, reasonable proxies and quantification estimation methods 

that have been generally applied in the related literature can be used. When that may not 

be possible, users may want to use shading or another visual modification to designate

indicators that are assessed qualitatively.

One advantage of this graphic is that the independent impacts are discretely illustrated, 

and so trade-offs across technology choice and location can be compared in light of 

policymaking priorities (as opposed to one indicator that aggregates all impacts without 

opportunity to prioritize objectives). Additionally, estimates of uncertainty for each 

impact can be expressed as error bars or illustrated intervals on the edge of each ruler.

A potential challenge when using this graphical representation is the sensitivity of results 

and their interpretations to changes in the defined upper and lower bounds on each axis. 

The selection of high and low bounds and using a linear scale inherently weights each of 

the criteria; using drastically different bounds could lead to different relative benefit

interpretations. For instance, if the upper bound on the LCOE is set to be lower than its 

original designation, then the magnitude of that indicator may change relative to the other 

indicators. It is important for the user to keep these implications in mind when 

determining axes bounds. For consistency, the standard adopted in the application here is 

to select bounds based on best and worst possible expected outcomes that are generally 

recognized in the field. 

Establishing the MMSA Indicators

For this pilot of the proposed MMSA visual, the authors limited the scope to six 

indicators relevant for specific sites and technologies. The team chose two indicators 

from each sustainability category: 

• Economic: energy affordability and energy diversity

• Environmental: climate friendliness and water conservation

• Social: safety and local employment impacts.

Table 8, which appears later in this text, summarizes the MMSA indicators and range of 

values. The following discussion addresses the specific indicators chosen and details 

assessment methodologies and determination of maximum and minimum values applied 

in this case study. The specific metrics were selected for illustrative purposes, but the 

model can be customized to include indicators that are particularly relevant for a variety 

of stakeholders.
6

Energy Affordability

The cost of electricity can determine a project’s economic viability, and so it is included 

here as one of the economic indicators as measured by LCOE in U.S. dollars (USD) per 

kilowatt-hours (kWh) to represent energy affordability. The authors assumed that the 

most costly utility-scale renewable energy electricity generation would be the most 

expensive considered, thus defining the worst expected outcome and the axis minimum.

The Intergovernmental Panel on Climate Change (IPCC) Special Report on Renewable 

Energy (SRREN) presents LCOE estimates for renewable technologies using discount 

rates of 3%, 7%, and 10%, respectively (IPCC 2011); this current analysis only considers

estimates produced assuming a 3% discount rate.1 The maximum LCOE value for utilityscale renewable technologies served as the upper-bound for one-axis solar photovoltaics 

(PV). This value of $0.39/kWh is the maximum LCOE value included, corresponding to 

the minimum energy affordability value. The minimum LCOE value was set to 

$0.00/kWh, representing maximum energy affordability. 

These costs do not include the cost of financing, integration, or transmission, nor do they 

include potential revenue streams, such as from the sale of renewable energy certificates 

(RECs), though these could be included, if desired, and data are available.

Energy Diversity

The security of energy supply sometimes drives promotion of specific energy strategies.

A secure energy supply is defined differently by various groups but is generally agreed to 

exhibit the characteristics of availability (physical), accessibility (geopolitical), 

affordability (economic), and acceptability (environmental) (Sovacool 2010). A variety 

of indicators have been established and applied to evaluate these different elements of 

energy security, some individual and others aggregated. 

While import dependency and geopolitical risk might also be important in a regional 

sense, this analysis uses a simple diversity indicator to assess the potential impact of a 

proposed project on a region’s energy security. Because diversity can indicate a system’s 

resilience to supply disruption and price volatility, it is a reasonable proxy for energy

security. 

To measure diversity, the Shannon Diversity Index is used, where pi is the fractional 

share of a utility’s electricity generation off-take from energy source i (Jansen et al. 

2004). Assuming all generation comes from coal, natural gas, petroleum, hydroelectric, 

nuclear, biomass, geothermal, wind, and solar, maximum diversity reflects generation 

that is evenly spread across all nine energy sources (i.e., pi = 0.111), which yields a 

diversity indicator value of 2.20.2 Minimum diversity indicates that all generation comes 

 1 Although a 3% discount rate would be considered low for project analysis from the viewpoint of an 

investor or developer, it is the commonly applied social discount rate and is used in this analysis since a 

policymaker’s perspective is assumed. The normal discount rate selection implications apply to the results 

presented.

2 This calculation comes from the Shannon Diversity Index, which is found as 

H = −� pi ln pi

i=1
7

from one source and yields a diversity indicator of 0. As projects assessed under this 

method are unlikely to result in a complete restructuring of an energy system, the impact 

scale preliminarily is set to range from a 5% decrease to a 5% increase in the diversity 

indicator.

Climate Friendliness 

Because climate change may motivate some clean energy investments, the potential 

impact on climate was chosen as an indicator as measured by life cycle GHG emissions 

in grams (g) of carbon dioxide equivalent (CO2-eq) per kWh. The scale was set based on 

recent GHG harmonization analysis published in the IPCC SRREN (IPCC 2011). 

The range of ‘climate friendliness’ median values ranged from 4 g CO2-eq/kWh for 

hydropower to 1,001 g CO2-eq/kWh for coal. Therefore, the climate friendliness range in 

this case study was set to roughly correspond to these figures with minimum and 

maximum values of 0 and 1,000 g CO2-eq/kWh, respectively. This range includes only 

the span of median values reported and not the entire range of possible values; however, 

current deployments would likely employ modern, high-efficiency designs, and 

associated emissions are unlikely to be higher than the median values. 

Water Conservation

Water availability and competing demands for water resources can be a constraint for 

energy project development. Although the technologies selected for comparison in this 

study have relatively low water requirements compared to other electricity generating 

technologies (Macknick et al. 2011), water conservation was chosen as one of the 

environmental metrics for this MMSA to illustrate its relevance and applicability to a 

wide variety of technologies. Both water resources and water uses are distributed 

unevenly throughout the United States, meaning any assessment of water use at a 

proposed site needs to consider local water availability. 

Water availability can be defined in a variety of ways, depending on what types of water 

are included (e.g., surface freshwater, fresh groundwater, brackish groundwater, 

municipal wastewater) and to what degree state and local regulations and legal 

requirements are considered. In places where freshwater resources are stressed, municipal 

wastewater and brackish groundwater sources can be used for electricity operations, 

though this incurs an additional cost and performance penalty (EPRI 2003). 

For the purpose of this analysis, we define water availability as described by Tidwell et 

al. (2014), which considers physical and legal availability associated with a variety of 

freshwater sources, brackish groundwater, and municipal wastewater, on a U.S. 

Geological Survey (USGS) hydrologic unit code (HUC) 8-level (Seaber et al. 1987). 

Water availability for each site in this study was determined by identifying the water 

availability characteristics of the HUC-8 region in which it is located. The water 

conservation metric used here is defined as the fraction of annually available freshwater 

(surface water and groundwater) that would be required for operations at the energy 

facility. If the facility can offset its freshwater usage by utilizing municipal wastewater or 

brackish groundwater for operations, that non-freshwater usage lowers the impact on 

water availability. 
8

The water conservation range was set from zero to one, with a zero value representing no 

available water required for operations for maximum water conservation; a value of one 

indicates that all available water is required for operations (representing the minimum 

value—the worst expected outcome—for water conservation for the purposes of the 

MMSA visual). 

Safety

Based on the importance of risks to workers and project acceptance by local community 

members, safety was identified by the team as a social indicator that may be of interest to 

policymakers. The specific metric chosen is based on recent research that explored

fatality-based indicators across fossil, nuclear, and renewable technologies for the IPCC 

SRREN (IPCC 2011). Fatalities from severe incidents (>5 fatalities) were selected due to 

data reliability and importance in public perception, which were normalized to electricity 

production in Gigawatt electrical years (GWe-years). 

All Organization for Economic Cooperation and Development (OECD) estimates for 

fossil technologies, the lower-limit total fatalities for Generation II nuclear facilities, and 

the full range across renewable technologies were included. Reported fatality rates ranged 

from 0.000245 fatalities/GWe-yr for PV, to 0.12 fatalities/GWe-yr for coal (IPCC 2011).

In the MMSA visual, these values represent the maximum (best expected outcome) and 

minimum (worst expected outcome) levels of safety corresponding to fatality risk for the 

safety indicator’s axis. 

Local Employment Impacts

Given the importance of local employment opportunities to securing community support 

for energy projects, particularly from a policymaker’s perspective, local employment 

impact is included as a social metric in this pilot analysis. Many utility-scale renewable 

energy projects are developed in rural areas where employment opportunities may be 

scarce, with the potential for increasing urban migration. Though the absolute number of 

jobs associated with one energy development project may be small relative to the overall 

economy, the impact of increased jobs may be significant in some regions.

For this analysis, the local employment impact is defined as the number of nonconstruction direct jobs created in the community divided by the county’s labor force. 

The maximum value would be 1 if a project were responsible for all jobs in the county, 

but as projects will not typically reach across an entire county, the maximum impact 

value (best expected outcome) is set to 1% of all jobs for the purposes of these 

preliminary analyses. The minimum value (worst expected outcome) is set to 0%, 

although it is important to note that macro-level jobs analysis may indicate a net negative 

jobs result and might be a consideration if conducting this MMSA from a macro-level 

perspective. 
9

Applying the MMSA Visual to Location and 

Technology Options

To test how the MMSA visual would work to evaluate potential energy development 

projects, two different technologies were evaluated at six different locations. Areas were 

selected based on a recent study of U.S. technical potential for renewable technologies 

(Lopez et al. 2012). This analysis identified all land area in the United States deemed 

suitable for utility-scale wind and solar development after screening for terrain slope 

restrictions, appropriate land use categories, environmental factors, and minimum wind 

resource intensity at 80 m hub height. 

These lands were overlaid with a geospatial dataset identifying Bureau of Land 

Management (BLM) and tribal-owned lands, and six 18-km2 parcels were identified with 

sufficient area, based on the screening criteria, to support at least 50 MW of wind,

assuming a deployment density of 5 MW/km2. Two potential sites, one BLM and one 

tribal, were chosen in each of Wyoming, Idaho, and Arizona. The potential development 

sites are shown in Figure 3 as “extraction areas.” The areas’ features are presented in 

Table 2, which also lists the specific tribes owning land designated as under the control of 

the Bureau of Indian Affairs (BIA). Also shown in Table 2 is the amount of the land from 

the initial screening area available for PV and wind development. 
10

Figure 3. Extraction areas used for the multi-metric sustainability analysis

The region shown is the southwest United States, with extraction areas in Arizona, Idaho, 

and Wyoming. BIA denotes tribal ownership, and BLM denotes ownership by the federal 

government. The pink extraction areas show the locations selected for analysis.
11

Table 2. Features of the Multi-metric Sustainability Analysis Case Study Sites

Area ID Ownership

Available 

PV Land Area (km2)

PV Average 

Capacity 

Factor

Available Wind 

Land Area 

(km2)

Wind Average 

Capacity Factor

1 BIA/Navajo 1.40 0.26 14.48 0.39

2 BLM 18.00 0.25 17.60 0.32

3 BIA/Duck Valley 2.67 0.22 15.44 0.32

4 BLM 4.16 0.22 17.32 0.34

5 BIA/Wind River 6.11 0.23 16.52 0.41

6 BLM 4.69 0.22 18.48 0.32

BLM and tribal lands are being explored for widespread deployment of renewable 

technologies (BLM 2012).

3 This makes tribal lands attractive for this study, but the local 

benefits may differ due to specific project development characteristics. 

Two deployment scenarios were evaluated on each land parcel: development of a 10-MW 

PV plant and development of wind across the developable land area at a density of 5 

MW/km2. The solar scenario assumed a PV density of 48 MW/km2. The wind scenario 

assumed deployment of International Electrotechnical Commission class 2 wind turbines 

at 80 m hub height in calculating the gross capacity factor.

Analysis of MMSA Indicators at Potential Deployment Sites

Energy Affordability

The LCOE for each project was calculated assuming industry-reported capital and 

operating costs and a discount rate of 3%, following the simplified LCOE methodology 

reported by NREL (2012). The overnight capital costs, fixed operating costs, and variable 

operating costs were obtained from NREL’s Transparent Cost Database (OpenEI 2012).

The median value for the technology-specific cost was selected in most cases; the 

variable operating costs for onshore wind only reported three values, and the intermediate

value was used in the analysis. 

To ensure consistency with values reported previously, these values were compared 

against those reported in a November 2010 report on LCOE (Tidball et al. 2010), which 

asserted a project lifetime of 30 years. This assumption was used in the cost calculations.

The cost data are presented in Table 3, and the calculated LCOE by technology and site 

are presented in Table 4. 

 3 Learn more about these efforts at http://apps1.eere.energy.gov/tribalenergy/,

http://www1.eere.energy.gov/tribalenergy/guide/cs_wind_opportunities.html, and http://www.nrel.gov/docs/fy14osti/61395.pdf. 
12

Table 3. Technology Costs Used to Calculate LCOE

Cost Category Onshore Wind Solar PV

Overnight Capital Costs ($/kW) $1,980.00 $5,200.00

Fixed Operating Costs ($/kW-year) $27.26 $16.00

Variable Operating Costs ($/kWh) $0.012 $0.000

Table 4. LCOE for Onshore Wind and Solar PV at Six Case Study Sites

Area ID Onshore Wind 

LCOE ($/kWh)

Solar PV 

LCOE ($/kWh)

1 $0.049 $0.12

2 $0.058 $0.13

3 $0.058 $0.15

4 $0.055 $0.15

5 $0.048 $0.14

6 $0.058 $0.15

Energy Diversity

In order to compute the baseline Shannon Diversity Indicator and the change in value 

under the proposed deployment scenarios, an appropriate system boundary must be

established. One option explored was to look at the generation profile of the nearest 

likely off-take utility. Assuming that this would be the nearest large utility, the three 

potential off-takers identified were Idaho Power, PacifiCorp, and APS. Delivered 

electricity by energy source was accessible only for Idaho Power as part of their 

Integrated Resource Plan (IRP) (2011). These values were compared with data available 

in the Ventex database, which reports for all three utilities, but this database only reports 

generation by owner, and so it does not capture energy delivered to utility customers as 

part of long-term power purchase agreements. 

In order to capture local diversity issues in the absence of this information, the baseline 

Shannon Diversity Index was calculated for each state based on reported U.S. Energy 

Information Administration (EIA) data (EIA 2010). The value calculated for Idaho, 

0.636, differs somewhat from the value calculated using generation values reported in 

Idaho Power’s 2011 IRP, 0.837. The disparity between Wyoming and PacfiCorp, the 

possible off-take utility for projects in that state, would likely be even greater given that 

PacifiCorp operates across several western states. 

To compute the diversity index under the different deployment scenarios, the expected 

new generation was added to EIA’s 2010 generation values. The expected generation

values, assuming maximum wind deployment and 10 MW of deployed solar PV, are 

reported in Table 5. 
13

Table 5. Potential Solar and Wind Generation on Case Study Sites

Area ID

Potential Solar 

Generation (MWh)

Installed Wind Capacity 

(MW)

Potential Wind 

Generation (MWh)

1 22,545 69.0 238,097

2 21,717 87.0 243,878

3 19,384 76.5 214,445

4 19,369 85.5 255,442

5 20,007 81.0 290,000

6 19,016 91.5 256,493

The solar scenarios assume a 10-MW project, and the wind scenarios show the estimated installed capacity 

and resultant generation estimates.

Table 6 presents the baseline diversity indicator for the states and the percentage change 

in diversity indicator under solar and wind project development scenarios at the six land 

parcels. As the wind projects result in much greater electricity generation, they have a 

greater impact on system diversity. The greatest change in the diversity indicator—

3.13%—was realized in the Idaho wind scenario on land parcel 4, corresponding to a 

reduced share of hydropower generation from 82.2% to 81.6%. In the Wyoming 

scenarios (land parcels 5 and 6), even the highest levels of deployment only reduced the 

share of coal from 91.3% to 91.2%. In Arizona (parcels 1 and 2), the additions of wind 

and solar realized the smallest changes to the already somewhat diverse system.

Table 6. Baseline Diversity Indicator for Each Case Study Site and the Percent Change in 

Diversity Indicator under Wind and Solar Project Development Scenarios

Land Parcel

Baseline 

Shannon 

Diversity Index

Diversity Indicator 

Change Under 

Solar Scenario 

(%)

Diversity Indicator 

Change Under Wind 

Scenario (%)

1 1.25 0.04 0.32

2 1.25 0.04 0.32

3 0.64 0.74 2.65

4 0.64 0.74 3.13

5 0.35 0.37 1.34

6 0.35 0.35 1.19

Climate Friendliness

In the absence of specific emission values for a given project, a prospective look at the 

likely climate impacts could use the most recent, published reports of the life cycle GHG 

emissions normalized to production amounts. In this case, we use the median values by 

technology reported in the IPCC SRREN. For all solar project scenarios, this corresponds 

to a value of 46 g CO2-eq/kWh, and for all wind projects, the GHG emissions are 12 g 

CO2-eq/kWh. 
14

Water Conservation

Considering the example locations in this study, water availability varies greatly from site 

to site (Figure 4). Sites 1, 2, and 6 have relatively little to no availability of surface water, 

and sites 2, 3, 4, and 6 have relatively little to no availability of groundwater. Sites 1 and 

2 have substantial municipal wastewater and/or brackish groundwater resources, which 

could be utilized to offset freshwater demands.

Figure 4. Water availability by source for six study regions

Given projected annual energy output for the two energy technologies considered here (as 

shown in Table 5), we calculate total annual water impacts based on established water 

consumption rates for PV technologies (DOE 2012). Wind has no operational water uses, 

and PV systems’ low operational water uses lead to relatively minor water consumption 

impacts at each site (Figure 5).
15

Figure 5. Water consumption by site for utility-scale PV installation

Water consumption from these two alternatives can also be analyzed in the context of 

other electricity generating technologies. Utilizing average annual energy output for the 

PV systems (20,000 MWh) and wind systems (250,000 MWh) for all sites, we compare 

the water consumption impacts of other electricity technologies that could potentially 

meet that level of generation using established water consumption values (Macknick et 

al. 2011). Overall, wind and PV water uses are negligible compared to other electricity 

generating technologies. Concentrated solar power (CSP) systems and coal plants with 

carbon capture and storage (CCS) technologies that utilize cooling towers have the 

highest relative water consumption impacts (Figure 6).
16

Figure 6. Water consumption impacts of a variety of electricity generating technologies

For these sites the impact on local water availability from PV and wind systems would be 

relatively minor, yet greater impacts could occur from development of other electricity 

generating technologies. 

Safety

For the projects under evaluation here, project safety was measured using the IPCCpublished values for fatalities of severe incidents as a proxy (IPCC 2011).4 The only 

reported value for PV was for crystalline silicon, and this value, 0.000245 fatalities/GWeyear, was assigned to the potential PV projects. Two values were reported for onshore 

wind, one each for Denmark and the United Kingdom. The average of these values, 

0.00415 fatalities/GWe-year, was assigned for the potential wind projects. 

Local Employment Impacts 

To assess local employment impacts, labor force values by county were obtained from 

the U.S. Bureau of Labor Statistics (BLS 2012). The average of the labor force data by 

county reported for the period April 2011 to May 2012 was used. Direct jobs for each of 

the project deployment scenarios were estimated by obtaining operation and maintenance 

job values from a published review of renewable energy jobs studies (Wei et al. 2010). 

 4 Fatality rates consider the full energy chain, noting that accidents can happen during any stage from 

exploration, extraction, and processing for fossil fuels to long distance transport and disposal. Estimates for 

renewable energy technologies are based on available accident statistics, literature review, and expert 

judgment because of limited historical experience and data (PSI n.d.).
17

The values reported for solar PV and wind in Table 2 of Wei et al. (2010) were averaged, 

scaled to the project size being evaluated (10 MW for solar; see Table 5 for wind project 

installed capacity), and rounded to the nearest integer value. The local employment 

impact was calculated by dividing the anticipated direct local jobs by the average labor 

force. Table 7 shows the labor force, expected direct jobs, and job impact for the six land 

parcels under both the solar and wind project scenarios. 

Table 7. Local Employment Impacts from Deployment of Wind and Solar Projects at Six

Land Parcels

Land 

Parcel County

Labor Force; 

Average April 

2011–May 

2012

Direct 

Jobs 

Created 

Solar

Direct 

Jobs 

Created 

Wind

Local 

Emp 

Impact 

Solar (%)

Local Emp 

Impact 

Wind (%)

1 Coconino, AZ 72,139 5 17 0.01 0.02

2 Pinal, AZ 139,144 5 21 0.00 0.02

3 Owyhee, ID 4,540 5 19 0.11 0.42

4 Owyhee, ID 4,540 5 21 0.11 0.46

5 Fremont, WY 19,725 5 20 0.03 0.10

6 Fremont, WY 19,725 5 22 0.03 0.11

MMSA Analysis and Presentation

To facilitate comparisons and trade-off analysis, this section tabulates all of the metrics 

discussed and illustrates the findings with the MMSA visual. Table 8 presents a summary 

of the MMSA indicators, metrics used, and minimum and maximum values applied for 

indicator evaluation. Table 9 shows the metric value calculated for each scenario and land 

parcel. Figure 7 presents the findings applied within the MMSA visual framework for all 

scenarios. 

Table 8. Summary of MMSA Indicators, Metrics, and Range of Values

MMSA 

Indicator

Energy 

Affordability

Energy 

Diversity

Climate 

Friendliness

Water 

Conservation Safety

Local 

Employment 

Impacts

Metric LCOE 

(USD/kWh)

Change in 

diversity 

indicator (%)

GHG emission 

(gCO2eq/kWh)

Available 

water used for 

plant operation 

(fraction)

Fatalities from 

severe 

incidents 

(fatalities/GWyr)

Contribution to 

county 

employment 

(%)

Minimum 0.39 -5 1,000 1 0.120000 0

Maximum 0 5 0 0 0.000248 1
18

Table 9. Metric Values for Each Scenario and Land Parcel

Deployment 

Scenario Land Parcel LCOE (USD/kWh)

Change in 

Diversity 

Indicator 

(%)

GHG 

Emission 

(g-CO2eq/ 

kWh)

Available 

Water Used 

for Plant 

Operations 

(fraction)

Fatalities 

from Severe 

Incidents 

(fatalities/G

W-yr)

Contribution 

to County 

Employment (%)

Solar

1 0.12 0.04 46 0 0.000245 0.01

2 0.13 0.04 46 0 0.000245 0.00

3 0.15 0.74 46 0 0.000245 0.11

4 0.15 0.74 46 0 0.000245 0.11

5 0.14 0.37 46 0 0.000245 0.03

6 0.15 0.35 46 0 0.000245 0.03

Wind

1 0.05 0.32 12 0 0.004150 0.02

2 0.06 0.32 12 0 0.004150 0.02

3 0.06 2.65 12 0 0.004150 0.42

4 0.05 3.13 12 0 0.004150 0.46

5 0.05 1.34 12 0 0.004150 0.10

6 0.06 1.19 12 0 0.004150 0.11

Legend

Figure 7. MMSA visual showing case study findings
19

Discussion

In the MMSA literature, some sustainability domains are relatively well-researched, with 

well-vetted quantification methods, such as GHG emissions and water use. However, this 

is not the case for other metrics, such as the social impacts of energy systems—

quantifying their potential impacts can be complex, and the studies that do exist 

sometimes rely only on expert opinion or general perceptions rather than quantitative 

analytical approaches. For example, one recent study of the European Union (EU) energy 

system (Carrera and Mack 2010) involved a thorough review of the social indicator 

research to identify energy-relevant impact indicators, but it relied on surveys of expert 

opinion to assign values to different social impact categories. 

Working in this multivariable, multidimensional, and relatively new field of sustainability 

science presents significant challenges that one would expect from a frontier field. Solid 

understanding of these challenges is essential to properly apply the proposed MMSA tool 

and present results. One of those challenges relates to the interdisciplinary nature of this 

broad area and the fact that, for the past several decades, each pertinent discipline has 

developed a unique language to efficiently communicate advances in the field among 

their peers. 

This paper presents one potential approach to synthesizing information from disparate 

disciplines and presenting it in a way that is accessible to a diverse set of stakeholders,

and which offers the benefit of a tractable visual interpretation for a wide range of 

audiences. The tool could be further refined to account for uncertainty or to include 

dimensions that are limited strictly to qualitative assessment. It is important to note that 

parameter choices and the determination of maximum and minimum values for each 

indicator affect interpretation of results. For instance, a ruler scaled such that all impacts 

are likely to appear small could be misleading. Clearly detailing the underlying data and 

framework can help to ensure transparency and allow for critical analysis by interested 

stakeholders, as well as further refinements to meet unique project needs. As the

proposed MMSA tool is used and results are presented to diverse audiences, detailing the 

underlying assumptions and data that ultimately drive the implications of results will be 

critical for interpretation purposes. 

Lastly, consideration of the specific priorities and perspective of the decision-maker 

using the tool will likely influence the indicators selected for the analysis as well as the 

prioritization of indicators within the framework. For example, a decision-maker that is 

involved in investment planning or project development might be less concerned with 

climate friendliness or employment benefits relative to a policymaker. While the 

objective of this initial pilot was to demonstrate the proposed tool’s applicability 

specifically from a policymaker’s perspective, the tool was designed to be flexible and 

applicable to a wide range of potential users.
20

References

Afgan, N.H.; Carvalho, M.G.; Havonov, N.V. (2000). “Energy system assessment with 

sustainability indicators.” Energy Policy (28:9); pp. 603–612.

Afgan, N.H.; Carvalho, M.G. (2002). “Multi-criteria assessment of new and renewable 

energy power plants.” Energy (27:8); pp. 739–755.

Awerbuch, S. (2004). Portfolio-Based Electricity Generation Planning: Implications for 

Renewables and Energy Security. London: REEEP Environmental Policy Department. 

Begic, F.; Afgan, N.H. (2007). “Sustainability assessment tool for the decisions making 

in selection of energy system–Bosnian case.” Energy (32:10); pp. 1979–1985.

Bureau of Labor Statistics (BLS). (2012). Labor force data by county, not seasonally 

adjusted. September 2011–October 2012. Accessed March 

2014: http://www.bls.gov/lau/laucntycur14.txt. 

Bureau of Land Management (BLM). (2012). “New Energy for America.” Accessed 

March 2014: http://www.blm.gov/wo/st/en/prog/energy/renewable_energy.html. 

Convertino, M.; Baker, K.M.; Vogel, J.T.; Lu, C.; Suedel, B.; Linkov, I. (2013). “Multicriteria decision analysis to select metrics for design and monitoring of sustainable 

ecosystem restorations.” Ecological Indicators (26); pp;. 76-86.

Dey, C. (undated). Integrated Sustainability Analysis. The University of Sydney. 

Accessed October 2014: http://www.isa.org.usyd.edu.au/research/tblone.shtml. 

Electric Power Research Institute (EPRI). (2003). Use of Degraded Water Sources as 

Cooling Water in Power Plants. Technical Report 1005359. Palo Alto, CA: Electric 

Power Research Institute.

Elkington, J. (1994). “Towards the Sustainable Corporation: Win-Win-Win Business 

Strategies for Sustainable Development.” California Management Review (36:2) pp. 90-

100.

Energy Information Administration (EIA). (2010). Electric Power Sector Consumption 

Estimates, 2010 (Trillion Btu). Accessed March 

2014: http://www.eia.gov/beta/state/seds/data.cfm?incfile=sep_sum/html/sum_btu_eu.ht

ml.

Evans, A.; Strezov, V.; Evans, T.J. (2009). “Assessment of sustainability indicators for 

renewable energy technologies.” Renewable and Sustainable Energy Review (13); pp. 

1082–1088.

Francis, C. (undated). Cobweb Polygons (Spider Diagrams) for Visual Display of 

Sustainability. University of Nebraska Sustainable Agriculture. Accessed March 

2014: http://sustainableag.unl.edu/pdf/sustainability_polygon.pdf
21

Gallego Carrera, D; Mack, A. (2010). “Sustainability assessment of energy technologies 

via social indicators: Results of a survey among European energy experts.” Energy Policy

(38:2); pp. 2030–2039.

Goedkoop, M.; De Schryver, A.; Oele, M.; Durksz, S.; de Roest, D. (2010). “Introduction 

to LCA with SimaPro 7.” Amersfoort, The Netherlands: PRé Consultants.

Gross, R.; Blyth, W.; Heptonstall, P. (2009). “Risks, revenues, and investment in 

electricity generation: Why policy needs to look beyond costs.” Energy Economics (32); 

pp. 786–804. 

Hofstetter, P. (1998). Perspectives in Life Cycle Impact Assessment; A structured 

approach to combining models of the technosphere, ecosphere and valuesphere. New 

York, NY: Kluwer Academic Publishers.

Idaho Power Company. (2011). 2011 Integrated Resource Plan. Accessed September 

2014: http://www.idahopower.com/pdfs/AboutUs/PlanningForFuture/irp/2011/2011IRPF

INAL.pdf. 

Intergovernmental Panel on Climate Change (IPCC). (2011). IPCC Special Report on 

Renewable Energy Sources and Climate Change Mitigation. Prepared by Working Group 

III of the Intergovernmental Panel on Climate Change. Edenhofer, O; Pichs-Madruga, 

R.; Sokona, Y.; Seyboth, K.; Matschoss, P.; Kadner, S.; Zwickel, T.; Eickemeier, P.; 

Hansen, G.; Schlömer, S.; von Stechow, C., eds. Cambridge, United Kingdom and New 

York, NY: Cambridge University Press.

Jansen, J.; van Arkel, W.G.; Boots, M.G. (2004). “Designing Indicators of Long-term 

Energy Supply Security.” ECN-C--04-007. Petten, The Netherlands: ECN. Accessed 

March 2014: http://www.ecn.nl/docs/library/report/2004/c04007.pdf.

Lior, N. (2007). “About sustainability metrics for energy development.” Prepared 

for Sustainable energy development status panel at the NSF Workshop Frontiers in 

Transport Phenomena Research & Education: Energy Systems, Biological Systems,

Security, Information Technology and Nanotechnology, May 17-18, 2007, University of 

Connecticut - Storrs.

Lopez, A.; Roberts, B.; Heimiller, D.; Blair, N.; Porro, G. (2012). U.S. Renewable Energy 

Technical Potentials: A GIS-Based Analysis. NREL/TP-6A20-51946. Golden, CO:

National Renewable Energy Laboratory. 

Macknick, J.; Newmark, R.; Heath, G.; Hallett, K. (2011). 2011: A Review of 

Operational Water Consumption and Withdrawal Factors for Electricity Generating 

Technologies. NREL/TP-6A20-50900. Golden, CO: National Renewable Energy 

Laboratory.

National Renewable Energy Laboratory (NREL). (2012). “Simple Levelized Cost of 

Energy (LCOE) Calculator 

Documentation.” http://www.nrel.gov/analysis/tech_lcoe_documentation.html. 
22

OpenEI. (2012). “Transparent Cost Database.” DOE/NREL. Accessed July 20, 

2012: http://en.openei.org/apps/TCDB/.

Paul Scherrer Institut (PSI). (undated). “Comparative Risk Assessment and the ENSAD 

Database.” Accessed August 2014: http://www.psi.ch/ta/risk-assessment. 

Pehnt, M. (2006). “Dynamic life cycle assessment (LCA) of renewable energy 

technologies.” Renewable Energy (31:1); pp. 55–71.

Seaber, P.; Kapinos, F.; Knapp, G. (1987). Hydrologic Unit Maps. Denver, CO: United

States Geological Survey. Accessed March 2014: ftp://ftp.ftw.nrcs.usda.gov/wbd/.

Sovacool, B.K., ed. (2010). The Routledge Handbook of Energy Security. New York: 

Routledge USA.

Tidball, R.; Bluestein, J.; Rodriguez, N.; Knoke, S. (2010). Cost and Performance 

Assumptions for Modeling Electricity Generation Technologies. NREL/SR-6A20-48595.

Work performed by ICF International, Fairfax, VA. Golden, CO: National Renewable 

Energy Laboratory. 

Tidwell, V.; Moreland, B.; Zemlick, K.; Roberts, B.; Passell, H.; Jensen, D.; Forsgren, 

C.; Sehlke, G.; Cook, M.; King, C.; Larsen, S. (2014). “Mapping water availability, 

projected use and cost in the western United States.” Environmental Research Letters

(9:6). 

United States Department of Energy (DOE). (2012). SunShot Vision Study. DOE/GO102012-3037. Washington, D.C.: U.S. Department of Energy, Office of Energy 

Efficiency and Renewable Energy.

Wei, M.; Patadiab, S.; Kammena, D. (2010). “Putting renewables and energy efficiency 

to work: How many jobs can the clean energy industry generate in the US?” Energy 

Policy (38:2); pp. 919–931.