U.S. Energy Information Administration Independent Statistics & Analysis www.eia.gov

Using R for Advanced Statistical 

Modeling in the National Energy 

Modeling System

Janice Lent, Chief Statistician

U.S. Energy Information Administration

October 24, 2018 | Workshop on Government Advances in Statistical 

Programming, Washington, DC
Overview

2

• Background: EIA’s National Energy Modeling System 

(NEMS)

• Using R in the NEMS

• Applications

–Hurdle models to project residential solar photovoltaic 

system installations

–Bayesian dynamic linear models for spot prices of 

hydrocarbon gas liquids

–Calibrating long-term projections to volatile short-term 

projections

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
EIA’s National Energy Modeling System 

(NEMS)

3

• EIA’s long-term projection system, the NEMS projects 

annual U.S. energy production, consumption, prices, etc. 

for EIA’s Annual Energy Outlook (AEO).

• Inputs to the NEMS

–A vast collection of EIA and non-EIA datasets, including 

global economic and demographic information

–Classic economic assumptions regarding the effects of 

long-term supply and demand pressures

–Specific assumptions regarding macroeconomic conditions 

and technological advances, as specified in NEMS “cases.”

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Examples of NEMS “cases”

4

• Reference case—status quo, assuming moderate GDP 

growth, relative to historical averages

• High and low macroeconomic growth cases

• High and low oil and natural gas price cases

• High and low technological advancement cases

• Special policy-based cases, e.g., cases representing the 

effects of the Clean Power Plan proposed in 2014.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
5

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Using R in the NEMS

6

• The NEMS base code is written in Fortran.

• The NEMS invokes R by sending a call to the command 

prompt.

• R opens and runs the script indicated in the Fortran 

command.

• R reads and writes comma-delimited files to exchange 

data with the NEMS base code.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Projecting the growth of residential solar 

photovoltaic (PV) penetration in the NEMS

7

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Previous NEMS method of projecting 

growth of residential solar PV

8

• Projection strata were defined by cross-classifying housing 

units based on variables from EIA’s Residential Energy 

Consumption Survey (RECS) and data on solar irradiation from 

the National Renewable Energy Laboratory.

• For each projection stratum, a cash flow analysis produced an 

estimated “payoff time” for a residential solar PV installation. 

Projected installations were based on payoff time.

• Projections were driven mainly by projected PV prices and 

failed to reflect social momentum and the effects of 

macroeconomic assumptions in different NEMS cases.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
An econometric modeling approach

9

• Starting with the 2017 version of the NEMS, EIA 

implemented a new econometric approach for modeling 

residential solar PV.

• Reflects social influences and macroeconomic 

assumptions associated with specific NEMS cases.

• Incorporates data from new sources.

• Uses zipcodes as projection strata.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Use of “big data” sources

10

• Project-level data on PV installations

–Some state governments publish project-level databases

–National Renewable Energy Laboratory’s (NREL’s) Open 

PV Project database (voluntarily-reported data on PV 

installations)

• Completeness of the data is uncertain.

• In 2014, EIA began publishing estimates of historical 

residential solar PV electricity generating capacity, 

providing historical data to which projections may be 

calibrated.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Residential solar PV installation data

11

• We aggregate the project-level data to the zipcode level 

for each installation year.

• The observations are zipcode/year combinations.

• We calibrate the model-based estimates to recent 

historical data at the state level.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Model covariates

12

• Because we wish to use the model for long-term 

projections, we limit our model covariates to items that 

are projected in the NEMS.

• Model covariates

–Median income and number of households, by zipcode, from 

the American Community Service (ACS)

–Population density (households per square mile, used as a 

proxy for roof area) from the ACS and Postal Service

–Zipcode-level residential electricity rates, estimated from EIA 

and NREL data

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Model covariates (2)

13

• Model covariates (continued)

–Annual average solar irradiation (kilowatt hours per square 

meter per day) from NREL

–Monthly payment per watt of installed PV capacity, estimated 

using PV price data from the Lawrence Berkeley National 

Laboratory and annual average mortgage interest rates from 

the Federal Reserve Economic Data (FRED) system

–A one-year lagged dependent variable, to represent local social 

momentum.

• The NEMS projects the covariates at the census division 

level. We assume change is constant within divisions.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Model fitting algorithm

14

• The zero-hurdle model comprises a logistic regression 

model combined with a negative binomial count 

model.

• We fit the component models using functions in the 

contributed R package gamlss (Generalized Additive 

Models with Location, Scale, and Shape).

• By default, the gamlss package uses the RigbyStasinopolis (RS) method, a weighted iterative 

maximum likelihood algorithm, to fit the non-linear 

models.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Hurdle model vs. payback model –

reference case

15 Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Hurdle model for macroeconomic cases

16 Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Advantages of the new model

17

• Uses additional data sources, including ACS data and 

project-level datasets of PV installations.

• Reflects social momentum at the zipcode level.

• Reflects macroeconomic and other assumptions 

associated with different NEMS cases.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Projecting prices of hydrocarbon gas 

liquids (HGLs)

18

• Hydrocarbon gas liquids include a range of substances 

produced during natural gas processing and petroleum 

refining.

• Advances in horizontal drilling and hydraulic fracturing 

(“fracking”) technologies have recently led to record high 

levels of HGL production in the United States.

• The “shale gas boom” has affected the market dynamics 

of HGLs.

• The most plentiful HGLs are propane and ethane. 

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Market dynamics for propane

19

• Propane (C3H8)

–Vaporizes at -44° F (-42° C), allowing for easy shipment in 

liquid form.

–Variety of uses, including heating, in industrial, 

commercial, transportation, and residential sectors.

–Domestic demand and exports are expected to expand with 

increasing supply.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Market dynamics for ethane

20

• Ethane (C2H6)

–Vaporizes at -126° F (-88° C) and is usually transported by 

pipeline, limiting export potential (compared to propane), 

though exports by ship begun in recent years.

–Increasing supplies from the Marcellus/Utica shale basin 

have decreased prices.

–Used only by the petrochemical industry for “cracking” to 

produce ethylene.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
The ethane cracking process

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System

21 Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Propane and ethane prices

22

• Prior to horizontal drilling, most domestically produced 

propane came from oil refineries, as a byproduct of 

gasoline and diesel fuel.

• The previous NEMS model for propane prices was an 

ordinary least squares regression model with crude oil 

prices as the sole explanatory variable.

• EIA previously published no projected prices for ethane.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Changing price dynamics of HGLs

23

• Most domestically produced propane and ethane now 

comes from natural gas wells.

• Propane and ethane prices tend to track both oil and 

natural gas prices.

• When oil prices are rising faster (or decreasing more 

slowly) than natural gas prices, the HGL prices tend to 

track natural gas prices more closely, and conversely.

• We needed a model that would reflect the changing 

relative influences of Brent oil prices and Henry Hub 

(HH) natural gas prices on HGL prices.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Goals of hydrogen gas liquids spot price 

projection for the NEMS

24

• Use dynamic models to describe evolving relationships 

between energy spot prices.

• Systematize the use of expert judgment in the long-term 

projections, making it more transparent and more easily 

automated.

• Extend dynamic linear modeling techniques, developed 

for short-term forecasting, to the case of long-term 

forecasting.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System

Bayesian Dynamic Linear Models (DLMs)

26

• DLM computations are performed recursively, 

incorporating new data via a Bayesian updating process.

• For long-term projections, DLMs provide the flexibility 

needed to represent expected changes in energy market 

dynamics.

• We use a “multi-process” DLM, which employs a 

combination of models to represent the complexities of 

the HGL market price dynamics. 

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
NEMS joint propane/ethane price model

27

• Bayesian multi-process model, incorporating four 

component models:

–Three Bayesian DLMs with Brent oil prices and HH 

parameters that change at different rates (slow, moderate, 

and fast) over the projection period

–A random walk DLM with parameters based on historical 

data for the Brent, HH, and the following supply and 

demand variables:

•Ethane production

•Propane production

•Organic chemical demand

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
NEMS joint propane/ethane price model (2)

28

• The component weights change over the projection 

period, gradually reducing the influence of the historical 

data and increasing the influence of the Bayesian 

components.

• Hyperparameters control the evolution of the component 

weights over the projection period and can be customized 

for different NEMS cases (sets of assumptions) 

considered in EIA’s Annual Energy Outlook.

• The joint propane/ethane price model is programmed 

using the dlm package in R. EIA contracted with the 

package author, Giovanni Petris, to assist with the 

programming.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Propane price projections: Reference case

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System 29
Propane price projections: High oil price

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System 30
Advantages of the new model

31

• Through Bayesian priors, the model formalizes the 

incorporation of expert judgment into the long-term 

projections.

• The dynamic model reflects the changing dynamics of the 

HGL markets.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Using R for time-series calibration

32

• In addition to the long-term NEMS projections, EIA 

publishes monthly short-term projections.

• The monthly short-term projection period is one to two 

years. At the end of each year, the projection period is 

extended another year.

• The modeling system is called the Short-Term Energy 

Outlook (STEO) system.

• The annual long-term projections from the NEMS are 

calibrated to agree with the annualized STEO projections 

for the early years of the long-term projection period.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Calibration to short-term projections

33

• Because the historical data series for many energy-related 

statistics are volatile, the short-term projections often 

change substantially from year to year.

• The adjustment factors used to calibrate the long-term 

series are therefore sensitive to the choice of calibration 

year.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Example: Henry Hub Natural Gas Prices (dollars per MMBtu)

34

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Example: Henry Hub Natural Gas Prices – with damping

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System

35
New calibration system

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System 36

• The new calibration methods offer the option of 

calibrating to the trend of the short-term series, rather 

than to a particular value in the series.

• Rather than simply applying ratio adjustments, the new 

calibration methods involve linearly splicing time series, 

i.e., smoothing breaks in the series by inserting lines.

• The methods are implemented in R using the zoo package 

and the Holt-Winters (HW) function in base R.
User inputs to the calibration methods

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System 37

• Users may customize the methods by setting values for 

several parameters.

• Example:

–The HW smoothing involves two parameters, 糠 and 紅, which 

take values in the half-open interval ど, な .

–Users may specify values for 糠 and 紅. Lower values cause earlier 

data points in the series to be weighted more heavily, while 

parameter values near 1 give the more recent data points more 

weight.
Henry Hub NG price with a default 紅

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System 38
Henry Hub NG price with a customized 紅

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System 39
Benefits of the new calibration system

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System 40

• By calibrating to the overall trend of the short-term series, 

the methods reduce the sensitivity of the long-term 

calibrated series to the level of the final year of the shortterm projection.

• Through user-defined parameters, the system allows users 

to customize the calibration process for atypical series 

without revising computer code.

• The code assigns default values to all of the user-defined 

parameters.
Summary

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System 41

• Although the NEMS is written in Fortran, it can easily 

communicate with component programs written in R.

• Bayesian techniques allow systematic incorporation of 

expert judgment into long-term projections.

• By using contributed R packages, EIA is incorporating 

advanced statistical modeling techniques into the NEMS 

without programming the new techniques in Fortran.
References

42

• Bilder, Christopher R. and Loughin, Thomas M. (2015). Analysis of 

Categorical Data with R. CRC Press, New York, pp. 61-140.

• Hernandez, Mari (2013). “Solar Power to the People: The Rise of 

Rooftop Solar Among the Middle Class.” Center for American Progress, 

October 21, 2013.

• Hilbe, Joseph M. (2011). Negative Binomial Regression, 2nd edition. 

Cambridge University Press, UK.

• Keiser, Richard (2012). “Which are the most profitable regions for solar 

in the United States?” PV Tech article available at http://www.pvtech.org/guest_blog/21088. 

• Rothfield, Emily (2010). “Solar Photovoltaic Installation in California: 

Understanding the Likelihood of Adoption Given Incentives, Electricity 

Pricing and Consumer Characteristics.” Trinity College of Duke 

University. Durham, North Carolina.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
References (2)

43

• Harrison, P.J., and Stevens, C. (1971). “A Bayesian approach to shortterm forecasting.” Operations Research Quarterly, 22, p. 341-362.

• Harrison, P.J., and Stevens, C. (1976). “Bayesian forecasting” (with 

discussion). Journal of the Royal Statistical Society, Series B, 38, p. 205-

247.

• Petris, G., Petrone, S., and Campagnoli, P. (2009). Dynamic Linear 

Models with R, Springer, New York.

• Welch, G. and Bishop, G. (2006). "An Introduction to the Kalman Filter." 

Available online at 

http://www.cs.unc.edu/~welch/media/pdf/kalman_intro.pdf. 

• West, M. and Harrison, J. (1997). Bayesian Forecasting and Dynamic 

Models, 2nd edition, Springer, New York.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Contact Information

44

Janice Lent

Janice.lent@eia.gov

202-586-6623

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System
Technical Details

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System

45


Projection method

48

• We use projected covariates from various NEMS modules 

to update the hurdle model covariates. 

• Because the covariates are projected at the Census 

Division level, we assume constant multiplicative change 

factors across zipcodes within each Census Division.

• Apply the hurdle model coefficients to the projected 

covariate data at the zipcode level.

• Aggregate the zipcode level counts to calculate 

installations and penetration rates for larger geographic 

areas.

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System




Weighting the component models

53

• The choice of component weights reflects the following 

expectations:

–Model component 4, whose coefficients are based on 

historical data, will less accurately reflect market conditions 

as the projection horizon increases.

–Of the 絞-based model components, we expect either model 

component 2 (the component with the most moderate 絞

value) or model component 4 (component with the highest 

絞 value) will most accurately reflect market conditions in 

the later projection years. 

Janice Lent, Using R for Advanced Statistical Modeling in the National Energy Modeling System