# Petroleum Marketing Program Evaluation

Program Name:

The Petroleum Marketing Family of Surveys is conducted by the Petroleum
Division of the Office of Oil and Gas within EIA.

**Brief Description of Program**:

The Petroleum Marketing Program collects and publishes data on the
nature, structure, and efficiency of petroleum markets at the national,
regional, and State levels. The Petroleum Marketing Program compiles
data via 11 survey forms, which are listed here in order of their place
in the market chain:

-   Domestic Crude Oil First Purchase Report (EIA-182)---a mandatory
    monthly census of firms that take or retain ownership of domestic
    crude oil leaving the lease on which it was produced within the
    United States including the Outer Continental Shelf. Firms report by
    stream on the volume in barrels and the weighted average cost per
    barrel for purchases they made in the target month.

-   Monthly Foreign Crude Oil Acquisition Report (EIA-856)---a mandatory
    monthly census of two populations: (1) firms that reported data as
    of June 1982 on the Transfer Pricing Report (ERA-51) and (2) firms
    acquiring more than 500,000 barrels of foreign crude oil during the
    reporting month for importation to the United States and its
    territories/possessions. Firms report country of origin, volume
    acquired in barrels, and cost for each acquisition made in the
    target month.

-   Refiners' Monthly Cost Report (EIA-14)---a mandatory monthly census
    of firms who own or control refining operations in the United States
    and its territories/possessions. Firms report for each Petroleum
    Administration Defense District (PADD) on the total volume (in
    thousands of barrels) of crude oil acquired and all costs (in
    thousands of dollars) associated with its acquisition and transport
    to the refinery for domestic and imported crude oil purchases.

-   Refiners'/Gas Plant Operators' Monthly Petroleum Product Sales
    Report (EIA-782A)---a mandatory monthly census of firms who directly
    or indirectly control a refinery or a gas plant located in the
    United States. Firms report by state on their total sales volume (in
    thousands of gallons) and average price per gallon (excluding sales
    taxes) for each petroleum product by sales type by user category.
    The petroleum products are finished motor gasoline (by type/grade),
    No. 2 diesel oil (by sulfur content), No. 2 fuel oil, propane, No. 1
    distillate, kerosene, aviation gasoline, kerosene-type jet fuel, No.
    4 fuel oil, and residual fuel oil (by sulfur content). Retail sales
    of gasoline are partitioned into sales through company-operated
    retail outlets versus sales to other end users, while wholesale
    sales are partitioned into DTW sales, rack sales, and bulk sales.
    Retail sales of No.2 distillates and propane are partitioned into
    residential sales, commercial/institutional sales, industrial sales,
    sales through company-operated outlets, petrochemical sales, and
    sales to other end users. Wholesale monthly state sales of No. 2
    distillates and propane are reported as a group as are retail versus
    wholesale sales of other petroleum products.

-   Monthly Report of Prime Supplier Sales of Petroleum Products Sold
    for Local Consumption (EIA-782C)---a mandatory monthly census of
    petroleum product suppliers who make the first sale of specified
    petroleum products and then deliver that product into a state for
    consumption in that state. Firms report by state on their monthly
    sales (in thousands of gallons) for finished motor gasoline (by
    type/grade), No. 2 diesel oil (by sulfur content), No. 2 fuel oil,
    propane, No. 1 distillate, kerosene, aviation gasoline,
    kerosene-type jet fuel, No. 4 fuel oil, and residual fuel oil (by
    sulfur content).

-   Resellers'/Retailers' Monthly Petroleum Products Sales Report
    (EIA-782B)---a mandatory monthly sample survey of resellers and
    retailers of petroleum products within the United States selling to
    customers within the United States. Using essentially the same form
    as EIA-782A, firms report by state on their total sales volume (in
    thousands of gallons) and average price per gallon (excluding sales
    taxes) for each petroleum product by sales type by user category.
    However, the petroleum products for which reports are requested are
    restricted to finished motor gasoline (by type/grade), No. 2 diesel
    oil (by sulfur content), No. 2 fuel oil, propane, and residual fuel
    oil (by sulfur content). The reporting categories for gasoline and
    No. 2 distillates and propane follow that of the EIA-782A survey.
    Residual fuel oil is reported by retail versus wholesale sales.

-   Annual Fuel Oil and Kerosene Sales Report (EIA-821)---a mandatory
    annual sample survey of companies that deliver or sell to end users
    distillate and/or residual fuel oils or kerosene. Firms report
    annual sales volume (in gallons) by state of destination by product
    type and by energy end use (residential, commercial, industrial, oil
    company, railroad, vessel bunkering, farm, military, on-highway,
    off-highway, electric power, and other uses). Annual sales are
    reported for kerosene, distillates by type of distillate (No. 1, No.
    2 fuel oil, No. 2 low sulfur diesel, No. 2 high sulfur diesel, No.
    4), and residual fuel oil. Every four years, companies are required
    to report their sales of motor gasoline.

-   Winter Heating Fuels Telephone Survey (EIA-877)---a mandatory sample
    survey by telephone of No. 2 heating oil and propane dealers in 24
    Eastern and Midwestern states. Each week during the heating season
    (October 1 to mid-March), sample dealers report their residential
    prices for No. 2 heating oil and propane as of Monday for each of
    the states for which they were sampled. At the beginning of the
    heating season, sampled dealers report their annual sales volume (in
    thousands of gallons) by sampled states for the previous year.

-   Motor Gasoline Price Survey (EIA-878)---a mandatory sample survey by
    telephone of retail gasoline outlets. Each week, sampled outlets
    report the retail pump price of each grade of self service unleaded
    gasoline (including taxes). Once a year, the outlet reports the
    state and local taxes included in the reported price.

-   On-Highway Diesel Fuel Price Survey (EIA-888)---a mandatory sample
    survey by telephone of retail motor vehicle diesel fuel outlets.
    Each week, sampled outlets report the retail pump price of self
    service on-highway motor vehicle diesel fuel (including taxes). Once
    a year, the outlet reports the state and local taxes included in the
    reported price.

-   Petroleum Product Sales Identification Survey (EIA-863)---a
    mandatory census of (1) resellers and retailers of No. 2 distillate,
    propane, and residual fuel oil; (2) resellers of motor gasoline;
    and (3) companies that sell kerosene, No. 1 distillate, crude oil,
    other LPG, No. 4 fuel oil, aviation gasoline, jet fuel, or other
    petroleum products. The census occurs every four years and is used
    to build the frame for the EIA-782B, EIA-821, the EIA-877, and other
    surveys. Firms that report on the EIA-821 in the year the EIA-863 is
    conducted are not required to report on the EIA-863.

Through this sequence of surveys, EIA monitors petroleum volumes and
prices as the commodity moves through the various stages from
production/importation (of the raw material) to refining (to create the
finished products) to transfer/distribution (from the refiner to the
retail outlets) to sales to ultimate consumers.

RECOMMENDATION: Presenting the surveys in this order makes it easier to
understand how these surveys are related to one another and should be
considered for overview presentations. Graphical presentations such as
that given at
http://eia.doe.gov/smg/asa_meeting_2004/fall/petroleummarket_files/slide0005.html
would also be helpful if updated to include all surveys in the Petroleum
Marketing Program.

Program Objectives:

The petroleum marketing program through this family of surveys publishes
high quality statistical data for public and governmental use on
national and regional petroleum marketing activities with an emphasis on
monitoring at national, regional and state levels the quantities of
crude oil and its finished products present at various market stages and
the associated acquisition costs or prices being charged. The Energy
Information Administration is mandated to collect these data under the
Federal Energy Administration Act of 1974 (P.L. 93-275) and the
Department of Energy Organization Act (P.L. 95-91). The petroleum
marketing data are used extensively by the Bureau of Economic Analysis
in calculating various components of the U.S. Gross Domestic Product.
Other Federal agencies use the data to investigate the impact of oil
market events on national or regional economies and as a benchmark for
making economic decisions. The National Highway Traffic Safety
Administration uses petroleum marketing data to assess impacts of motor
vehicle fuel economy standards, for instance. The Joint Committee on
Taxation and the Internal Revenue Service use the data for analytic
studies investigating the potential impact of disruptions in oil supply
or major shifts in oil prices, as well as to validate tax receipts and
make fiscal projections and economic forecasts. The U.S. Administration
of Children and Families\' Division of Energy Assistance uses the data
to determine how to allocate funds during the winter for the Low Income
Home Energy Assistance Program (LIHEAP). The Commodity Futures Trading
Commission uses petroleum marketing data to assess futures trading and
for market surveillance. The data are used for both analytic purposes as
well as descriptive analyses. The data are also used as input for energy
models developed and maintained by EIA. The descriptive reports and
analytic/modeling uses tend to combine data from multiple surveys.

RECOMMENDATION: The Petroleum Marketing Program supports a data system
created through the integration of 11 component surveys. Documentation
needs to be developed for the program as a whole, including a
presentation of the overarching objectives underlying the design of the
data system and how the individual survey components contribute to the
overall design objectives. The present documentation focuses on each
individual survey component as a separate entity without relating it to
the overall system. The objectives for the overall data system should
focus on what EIA wants to be able to accomplish using the data system.
These objectives should clarify the geographic and temporal requirements
for reporting, as well as the type of data that are needed. Geographic
requirements include what constitutes the overall population and
subunits like states or PADDs for which separate estimation is
specified. Temporal requirements refer to the frequency of data
collection, whether weekly, monthly, or annual. Objectives for each
individual survey need to be developed that relate back to the
objectives for the data system and that motivate why each survey is
needed and cannot be replaced by another survey in the data system.
These objectives will also motivate the frequency of data collection and
the data items that are collected.

Target Populations:

The Petroleum Marketing Program collects data from two types of entities
in general. In terms of the market chain, the data system begins by
collecting data from firms that tend to be parent companies that have
complex structures with multiple offices, locations, subsidiaries, and
so forth. These surveys define the target population of firms in terms
of their oil market activities, whether purchasing domestic or foreign
crude oil (EIA-182, EIA-856); refining crude oil into finished petroleum
products (EIA-14); or supplying, reselling, and retailing these finished
products to customers (EIA-782A,B,C; EIA-821, EIA-877). The remaining
surveys focus on individual outlets selling the products to consumers
(EIA-878, EIA-888) and collect price data on a weekly basis. The
Petroleum Product Sales Identification Survey (EIA-863) is used to build
and maintain the frame for most of these surveys. The relationship
between the entities across these survey components is unclear;
population overlaps quite likely occur. The surveys vary in terms of
what constitutes a "firm" for instance and are sometimes vague in
defining the level of the business that should report and for which
reports are needed. The geographic constraints on the target populations
are not clearly specified for some surveys. Some surveys define the
target population to include all of the United States and all its
territories/possessions while other surveys appear to be restricted to
the 50 states and DC only. The reporting requirements for population
members undergoing changes of ownership and control are not addressed
for all surveys, but are addressed for the surveys that include smaller
companies where such events are more common.

RECOMMENDATION: The target populations for the individual surveys need
to be tightened up, particularly in terms of the level of the business
comprising the target population and the geographic and temporal
constraints on the target population. The individual definitions need to
be sufficiently specific that inclusions and exclusions from the
program's overall target population can be completely understood.
Ideally, the target populations across surveys would use a common
definition in terms of geography. Because of its use for economic
reporting, it would be preferable if the target population were defined
to include the U.S. territories and possessions whenever possible. In
defining the target population geographically, distinctions need to be
made between the location of the firm versus the location of its
operations as they affect target population membership. Other terms like
"firm" should also be defined comparably across surveys. For most of the
surveys, the target population should be defined in terms of the entire
business enterprise (the parent company including all locations and all
consolidated and unconsolidated entities). The definitions should
specify the treatment of units experiencing changes during the survey
reporting period in their ownership, control, operation, or in their
market participation characteristics. Any size constraints (or lack
thereof) on target population membership should be described for each
individual survey.

Sampling Frames:

The 11 surveys comprising the Petroleum Marketing Program tend to
develop and update their sampling frames independently. The available
documentation for the surveys usually does not address the quality or
coverage of the frames. The documentation needs to be expanded to
provide further details and a professional assessment of the methods in
use for frame building and maintenance and the extent to which they
assure frame coverage.

The target population definitions are somewhat loose in terms of the
level of the business that should respond and the frames may reflect
this imprecision. The enterprise appears to be the most appropriate unit
for most surveys, except for those that collect data from
establishments. The enterprise is the parent company and the entities
that the parent company owns or controls. An enterprise may have many
establishments\--locations or outlets where petroleum marketing occurs.

Ad hoc sources are used for those surveys such as EIA-14, EIA-182,
EIA-782A, and EIA-856, which census small populations. The frame sizes
for the censuses have been declining over time, and it is unclear to
what extent (if any) the decline represents coverage losses for the
frame. Data collection activities would capture deaths of companies on
the frame as well as changes in ownership and control, but it is unclear
the extent to which births are being captured. The source of the
sampling frames for the censuses is diverse:

-   The frame for the Refiners' Monthly Cost Report (EIA-14) was
    constructed from a list of 206 refiners obtained from the *Oil and
    Gas Journal* in 1983. In 2003, 67 of these 206 refiners were
    currently active and filing the EIA-14 form. The frame is updated
    periodically via information derived from the EIA-782A
    (Refiners'/Gas Plant Operators' Monthly Petroleum Product Sales
    Report) and the EIA-810 (Monthly Refinery Report).

-   The frame for the Domestic Crude Oil First Purchase Report (EIA-182)
    was compiled from the 1974 Federal Energy Administration (FEA) Oil
    and Gas Survey of Producers and Operators. The size of the frame has
    declined from 340 firms in 1978 to 81 firms in 2005. The available
    documentation does not cite how the frame is kept up to date.

-   The frame for the Refiners'/Gas Plant Operators' Monthly Petroleum
    Product Sales Report (EIA-782A) was derived from a list of refiners
    that reported on past EIA surveys and a list of gas plant operators
    derived from the Natural Gas Liquids Operations Report. As of 2003,
    the EIA-782A frame contained 100 companies.

-   The frame for the Monthly Report of Prime Supplier Sales of
    Petroleum Products Sold for Local Consumption (EIA-782C) was derived
    from the frame for its predecessor, the Prime Suppliers' Monthly
    Report (EIA-25), supplemented with firms identified from the
    Petroleum Products Sales Identification Survey (Form EIA-863) and
    other sources. As of 2003, the EIA-782C frame contained 170
    companies.

-   The frame for the Monthly Foreign Crude Oil Acquisition Report
    (EIA-856) is composed of all companies reporting data in June 1982
    on the Transfer Pricing Report (ERA-51) and all companies acquiring
    more than 500,000 barrels of oil in the reference month for
    importation into the United States. The 2003 OMB package notes that,
    "Approximately 400 cargo shipments are reported each month by 37
    companies."

The Petroleum Products Sales Identification Survey (EIA-863) is used for
frame building for (1) resellers and retailers of No. 2 distillate fuel
oil, No. 2 diesel, No. 5 and No. 6 residual fuel oil, propane, and motor
gasoline and gasohol; and (2) companies that sell kerosene, No. 1
distillate, crude oil, other LPG, No. 4 fuel oil, aviation gasoline, jet
fuel, and other petroleum products. EIA-863 is a census whose frame is
constructed by merging the previous EIA-863 census master frame with
state and commercial lists and then deduplicating the resultant
database. The available documentation does not discuss the survey's
response rate.

The current EIA-863 survey results are then used to build the frame for
the Resellers'/Retailers' Monthly Petroleum Sales Report (EIA-782B)
after merging in data from the most recent Annual Fuel Oil and Kerosene
Sales Report (EIA-821). Because the EIA-821 is a sample survey, use of
these data in frame building has the potential for inducing bias in
survey estimates. The OMB package also suggests that data from previous
EIA-782B surveys were used to update to the frame. Again, use of
sample-derived data---this time from previous EIA-782B surveys---has the
potential for inducing bias in survey estimates.

The EIA 821 survey frame is itself derived from the most recent EIA-863
survey results, "supplemented by retailers/resellers and importers of
residual fuel oil who were not identified by the EIA-863 survey."[^1]
The source of this supplementation is unspecified or why these companies
were not included in the EIA-863 survey.

The Monthly Report of Prime Supplier Sales of Petroleum Products Sold
for Local Consumption (EIA-782C) also derives its frame from the EIA-863
survey as does the Winter Heating Fuels Telephone Survey (EIA-877).

The Monthly Gasoline Price Survey (EIA-878) creates a frame of
company/state units based upon the EIA-782 survey of refiners and
resellers/retailers (Is this 782-A or 782-B or 782C or some
combination?). The On-Highway Diesel Fuel Price Survey (EIA-888) also
uses volume data from the EIA-782A and EIA-782B surveys to create a
frame of company/state units for sampling purposes. The EIA-782B data
are derived from a sample survey so the EIA-782B selection probabilities
must be considered in sample selection.

RECOMMENDATION: The frame building and maintenance activities for each
survey need to be documented in sufficient detail that the current
procedures can be evaluated in terms of their ability to maintain frame
quality and coverage. This documentation should describe the procedures
in place to capture births to the target population as well as changes
in ownership and control for frame members. The frame units need to be
evaluated against a precise definition of the target population to
determine whether deviations have occurred. An example might be multiple
listings that are associated with the same parent company, which could
be the source of duplicate reporting. Use of sample-derived data in
frame building should be determined and procedures put in place to
ensure bias is not being introduced in survey estimates.[^2] The
coverage of each survey should be evaluated.

RECOMMENDATION: EIA might consider introducing more data
sharing/coordination across petroleum marketing surveys. A register
could be developed and maintained with entries for all enterprises
engaged in petroleum marketing with their known petroleum activities
recorded, surveys they are obligated to complete, ownership and control
data, and so forth. EIA might also consider expanding the Petroleum
Products Sales Identification Survey (EIA-863) so that it can serve as a
vehicle for updating the frames for all the petroleum marketing surveys,
particularly to investigate new entrants to their target populations.

Program Design:

The Petroleum Marketing Program integrates data across 11 surveys to
measure volume and prices across the various marketing activities. The
available documentation does not present an overall design for the
entire program so the design has to be pieced out of the design
objectives of the individual surveys. The resultant design looks like it
evolved over time rather than in response to a conceptual plan for the
entire program. The overall result looks reasonable but a conceptual
review of the entire system and its survey components is needed to
determine if unnecessary burden is being placed on some businesses and
whether some surveys could be collapsed or replaced with other surveys.

RECOMMENDATION: A conceptual design needs to be developed for the
overall Petroleum Marketing Program describing the program's objectives,
the rationale for the survey components comprising the program, how the
surveys fit within the program's data collection plan, and their
adequacy for their intended role. The design should specify any changes
needed to existing surveys to create a unified, cohesive program design.

The content and the reporting cycles for the 11 Petroleum Marketing
Surveys have been appropriately chosen. Petroleum market conditions can
change quite quickly and frequent monitoring of volumes and prices are
needed to monitor their affect on the nation's economy. The Petroleum
Marketing Program also collects necessary details on crude oil
characteristics that affect demand and cost, about individual petroleum
products and their availability and price changes, and about how these
characteristics vary by region and by state. The bulk of the surveys
require monthly reports, which is appropriate for petroleum market
monitoring and also has the advantage of facilitating data integration
across surveys. The price surveys monitor on a biweekly or weekly basis,
which is needed to capture price swings for these products. There
clearly are population overlaps across survey components and there may
be some unnecessary/unusual burden placed on some population subgroups.

RECOMMENDATION: The overall design of the Petroleum Marketing Program
results in the collection of vital economic data at appropriately
frequent time intervals. Once EIA develops a conceptual design for the
data system, it will be clearer whether some surveys might be redesigned
and/or integrated to reduce reporting burden, particularly for large
businesses. This conceptual design should not only define the roles of
its individual survey components; it should also define the conceptual
relationship between the Petroleum Marketing Program and the Petroleum
Supply Program and the Reserve Survey Program and their associated
survey components.

Five of the 11 surveys are sample surveys (EIA-782B, EIA-821, EIA-877,
EIA-878, and EIA-888), and these surveys include small businesses as
well as large businesses. The Resellers'/Retailers' Monthly Petroleum
Products Sales Report (EIA-782B) has rotated smaller businesses in and
out of the sample with a cycle of about 2 to 2.5 years but the 2003 OMB
package suggests that substitution rather than randomization was used.
(The OMB package is in disagreement with other survey documents so the
rotation approach may now involve randomization.) The sampling
documentation for the remaining surveys does not mention the use of
sample rotation.

RECOMMENDATION: All sample surveys should have procedures put in place
that allow for sample rotation after about two years of reporting and
these procedures should be random so that the probability of inclusion
is known and quantifiable for all sampled cases.

The Resellers'/Retailers' Monthly Petroleum Products Sales Report
(EIA-782B) has developed a creative and innovative solution to the
extremely complex design problem posed by businesses which market
varying petroleum products in varying states and regions. This same
design problem is encountered by other sample surveys such as the Annual
Fuel Oil and Kerosene Sales Report (EIA-821), but the 2003 OMB package
suggests that the same design approach is not being used and it might
improve statistical precision and possibly allow for sample size
reduction.

RECOMMENDATION: The sample designs for the surveys other than EIA-782B
should be reviewed to determine whether they would benefit from sample
redesign using the 782B sampling strategy.

Data Collection and Processing:

Section 13(b) of the Federal Energy Administration Act of 1974 (Public
Law 93-275) specifies that survey participation is mandatory for the 11
surveys in the Program Marketing Program. The 11 surveys are not
consistent in emphasizing the legal mandate to respond. The EIA-14,
EIA-182, EIA-782A, EIA-782B, EIA-782C, and EIA-856 forms contain the
warning that:

This report is mandatory under Public Law 93-275. Failure to comply may
result in criminal fines, civil penalties and other sanctions as
provided by law. For the sanctions and the provisions concerning the
confidentiality of information submitted on this form, see sections 8
and 9 of the instructions. **Title 18 USC 1001 makes it a criminal
offense for any person knowingly and willingly to make to any Agency or
Department of the United States any false, fictitious, or fraudulent
statements as to any matter within its jurisdiction.**

Form EIA-863 has a modified version of this warning:

This report is mandatory under the Federal Energy Administration Act of
1974 (Public Law 93-275). Failure to comply may result in criminal
fines, civil penalties and other sanctions as provided by law. For the
sanctions and the provisions concerning the confidentiality of
information submitted on this form, see instructions. **Title 18 USC
1001 makes it a criminal offense for any person knowingly and willingly
to make to any Agency or Department of the United States any false,
fictitious, or fraudulent statements as to any matter within its
jurisdiction.**

Form EIA-821 has a weakened warning:

**This report is mandatory under Public Law 93-275.** **Title 18 USC
1001 makes it a criminal offense for any person knowingly and willingly
to make to any Agency or Department of the United States any false,
fictitious, or fraudulent statements as to any matter within its
jurisdiction.** For the provisions concerning the confidentiality of
information submitted on this form and sanctions, see Sections 8 and 9
of the instructions.

The remaining forms---EIA-877, EIA-878, and EIA-888---contain an even
weaker warning:

This report is mandatory under Public Law 93-275. For the sanctions and
the provisions concerning the confidentiality of information submitted
on this form, see Sections 8 and 9 of the instructions.

RECOMMENDATION: The mandatory nature of reporting is a "Big Stick" in
encouraging response. All survey forms should contain the same warning
and it should be similar to the more complete warning.

Most surveys in the Petroleum Marketing Program are conducted by mail,
with the exception being the EIA-877, EIA-878, and EIA-888 surveys which
are short telephone surveys designed for fast turnaround collection of
price information for petroleum products. Typically, the mail survey
forms are sent to refiners on an annual basis with business reply
envelops enclosed. (NOTE: this statement from the OMB package for some
surveys is contradicted by the lead letter included in the OMB package
saying that the firm must request that forms and mailing labels be sent
to them.) Other response options EIA makes available to the respondent
include using a PC Electronic Data Reporting Option (PEDRO) or a survey
form in Microsoft Excel Spreadsheet format. The respondent can return
their form by surface mail, by FAX, by e-mail, or through a secure HTTPS
site. Typically, the companies are instructed to report no later than 30
days after the end of each month and to submit the survey form for
months when no activity occurred. Five days after this deadline, a
listing of nonrespondent firms is generated and the companies are
telephoned and asked to submit the form. Noncompliance letters are
mailed to any companies not responding after the telephone prompt
requesting submission by a specified date.

In general, the data collection strategy is quite effective in promoting
response, particularly for the EIA-14, EIA-182, EIA-782A, and EIA-782C
surveys which census small numbers of large businesses on an ongoing,
monthly basis and achieve nearly if not exactly 100 percent response by
the final close out of data collection. The 782B survey is a sample
survey conducted monthly with a sample where smaller businesses rotate
in and out of the sample each year. The 782B survey response rate is
about 90 percent in spite of its mandatory requirement. Another annual
sample survey, the EIA-821 survey of fuel oil and kerosene dealers had a
94 percent response rate in 2001 and an 88 percent response rate in
2003.[^3] The OMB package does not provide response rates for EIA-856,
EIA-863, or EIA-877 so these cannot be evaluated. However, EIA-863
serves as a frame for many surveys so it is critical that it achieve as
close to a 100 percent response rate as possible. The surveys where less
than near perfect response is experienced tend to be sample surveys that
include small businesses, which may go in and out of business. Although
these surveys are mandatory, it is impractical to take action against
these firms. However, allowing nonresponse may educate firms to the fact
that in practice response is not required and lead to increasing
nonresponse over time.

RECOMMENDATION: EIA should examine the procedures it is using for its
mandated surveys experiencing nontrivial amounts of nonresponse to
determine what steps might be taken to encourage response. For instance,
EIA might consider revamping the mailout package for surveys
experiencing problems with response. The mailout packages other agencies
use for voluntary surveys might serve as examples to see how companies
could be motivated to respond. The lead letter sent to respondents could
be revamped to emphasize the importance of the survey and of their
response. Including a brochure addressing "Questions and Answers About
the EIA-xxx Survey" could encourage response as well as provide a
vehicle to educate the respondent by including questions such as, "My
company was sold last year. Do I need to return this form?" Studies have
shown that even the design of the mailout envelop can affect response as
well as details such as stamps versus machine stamping and express
delivery versus regular delivery.

The Petroleum Marketing Survey includes three telephone surveys: the
Winter Heating Fuels Telephone Survey (EIA-877), the Motor Gasoline
Price Survey (EIA-878), and the On-Highway Diesel Fuel Price Survey
(EIA-888). The response rate for EIA-877 is not provided in the 2003 OMB
report but for the other two have response rates of 99.7 percent and 97
percent respectively, which is commendable for surveys such as these
that need to collect price data expeditiously. The questionnaires are
quite short and collect only a handful of items on current petroleum
prices. These surveys collect data that should be readily known by the
respondent and easy to report. The brevity of the survey might make it
feasible for automated collection.

RECOMMENDATION: EIA should consider whether it would be cost effective
to introduce an automated approach for reporting, supplemented with the
current telephone interviews. When reporting is on a regular basis, say
weekly, companies could be asked at close of business on the day before
the reference day to call in to an automated line and enter the price
data for the reference day. Then on the reference day, all nonresponding
businesses could be called to complete the survey and to remind them to
call in next time.

Large businesses quite likely have to complete multiple data collection
forms because they can be expected to have company components that
engage in multiple lines of business:

-   The firm might take or retain ownership of domestic crude leaving
    the lease where it is produced (EIA-182).

-   The firm might also acquire foreign oil for import into the United
    States (EIA-856).

-   The firm might have acquired this domestic and foreign oil to
    support its refining operations so these acquisitions must be
    reported again (EIA-14).

-   Refining operations also have to report their monthly petroleum
    product sales (EIA-782A).

-   The firm may also have divisions that distribute petroleum as a
    prime supplier but they are exempted from EIA-782C if they complete
    EIA-782A.

-   The firm may have retail operations where petroleum products are
    sold (EIA-782B).

-   One or more company retail outlets may be sampled for the motor
    gasoline price survey (EIA-878).

-   These same retail outlets or other outlets may sell diesel fuel and
    be sampled again for reporting (EIA-888).

-   The company may also be asked to complete the Petroleum Product
    Sales Identification Survey (EIA-863).

These same companies may also be completing forms for Petroleum Supply
Surveys and/or the Reserves Surveys or even electric power surveys
(refiners may generate electricity).

RECOMMENDATION: EIA should standardize procedures and methods as much as
possible across surveys. For instance, defining the target population
unit in terms of the same entity whenever possible and avoiding
variations in geographic coverage. EIA should then examine its suite of
petroleum surveys to determine which, if any, surveys could be
integrated or combined. Such integration would reduce survey costs and
improve data quality, while reducing overall burden for large businesses
and even for moderate businesses. EIA should define its reporting entity
to be the business enterprise or parent company and look for a
designated contact person at each business to be responsible for all of
the company's reports. Some government agencies also establish a "Go To"
person within their agency so there is one person the large company can
always approach to address problems or concerns.

The suite of surveys in the Petroleum Market Survey essentially collects
the same data---volume and cost/price---at different stages in the
market sequence from petroleum extraction through ultimate consumption.
Except for the smallest companies, most companies will have to complete
multiple forms. There is a great deal of variation in how volume and
cost/price data are collected across forms:

-   EIA-182 collects first purchase data as number of barrels
    (presumably rounded to nearest integer) and average cost per barrel
    (presumably rounded to the nearest dollar).

-   EIA-856 collects foreign crude oil acquired as number of barrels
    rounded to the nearest integer and average price per barrel in
    dollars and cents.

-   EIA-14 collects crude oil acquired as thousands of barrels and total
    cost in thousands of dollars.

-   EIA-782A collects product sales as thousands of gallons and average
    price per gallon to the tenth of a cent.

-   EIA-782C collects prime supplier sales in thousands of gallons and
    does not collect price data.

-   EIA-821 collects fuel oil and kerosene sales in actual gallons and
    does not collect price data.

-   EIA-877 collects winter heating fuels for a company/state
    combination with only price collected and an average price across
    outlets within the state expressed in dollars to the tenth of a
    cent.

-   EIA-878 collects motor gasoline prices for individual stations with
    only price collected and expressed in dollars to the tenth of a
    cent.

-   EIA-888 collects diesel fuel prices for individual stations with
    only price collected and expressed in dollars to the tenth of a
    cent.

Reporting of average cost per barrel or gallon may deviate from the way
these events are recorded in the company's accounting system and hence
require arithmetic to convert an easily assessable and precise total to
an average. To determine the average price for a region or the nation,
EIA often constructs a total sales amount by multiplying the firm's
average price by total barrels or gallons (which may themselves have
been rounded to thousands). Thus, collecting sales as an average per
gallon/barrel introduces imprecision in survey estimates and may be more
burdensome on the company responding.

RECOMMENDATION: EIA should examine the units used for collection of
volume and price across surveys to make the units conform as closely as
possible to companies' accounting records, to reduce the need for
arithmetic by firms in responding, and to produce more precise
statistics for EIA reports.

For the mail surveys and censuses, the firm is given well crafted
instructions for returning the form by surface mail, by FAX, by e-mail,
and by secure file transfer. The instructions advise that both e-mail
and FAX returns are not secure and the company is urged to use secure
file transfer approach instead. Companies are also instructed where to
obtain a free copy of PEDRO software, which can be used for form
completion. Because PEDRO has data edits built in, the firm is advised
that use of PEDRO should result in the need for fewer resubmissions.
Procedures to ensure confidentiality of completed survey report forms
are not described in the available documentation.

RECOMMENDATION: The Petroleum Marketing Program is to be commended for
easing the burden of reporting by providing alternate vehicles,
particularly for last-minute returns and for its cautions as to data
security. Procedures should be put in place to ensure confidentiality of
forms once they arrive in house or if already in place these procedures
should be documented in the survey's methodology report and noted in the
OMB package.

Forms Design:

The forms for the surveys in the Petroleum Marketing Program have a
common format. The questionnaire is first and is short with tables for
entry of petroleum marketing data. The second component of the package
is composed of detailed instructions for completing the questionnaire.
Finally, there is a list of definitions for key terms used in the
questionnaire and in the instructions. The overall approach is
commendable but there are minor modifications that would improve these
documents. Changes that are specific to a particular questionnaire are
presented in the individual survey evaluations. This section presents
recommendations that are applicable across surveys.

The overall package---lead letter, form, instructions, and
definitions---could benefit from review by a questionnaire specialist.
The lead letters lack "punch." The forms could make better use of space
to improve readability and other professional touches. The forms,
instructions, and definitions for some surveys had grammatical errors
and complex sentences that need to be fixed. Certain survey
conventions---like including the mail back address on the survey
form---are not being followed. None of these problems are expected to
create serious errors but they do not demonstrate the level of survey
professionalism that EIA desires.

RECOMMENDATION: The entire package should be reviewed by a questionnaire
specialist with an eye to ensuring that EIA is "putting its best foot
forward" as well as to ensure readability, ease of use, and accuracy.
Specific examples of problems are given in each survey's evaluation.

The lead letters in the 2003 OMB package disagreed with the description
in the methodology section so it was unclear if they were current or
even whether lead letters are being mailed out. For the recurring
monthly surveys, it would be appropriate to send a letter once a year. A
decision could be made as to whether the package needs to include the
survey form, instructions, and definitions, when these documents are
unchanged. The lead letters included in the 2003 OMB package were
adequate but need to be revamped to be more pertinent to the individual
survey. Special attention should be given to the mailout packages for
sample surveys to encourage response from smaller companies.

RECOMMENDATION: Except for EIA-863 survey which occurs every four years,
EIA should mail lead letters to sampled/censused businesses at least
once a year. These letters should be tailored to the specific
populations replying to the individual surveys. The letters for the
continuing censuses might be more like a "Thank You for Responding"
letter with a "What's New" discussion. The overall goal is to promote
positive feelings about the survey and about the company's participation
in this survey. The letter for the sample surveys must welcome newcomers
to the survey while convincing them to respond.

Some surveys require that the company submit separate forms for every
state in which they sell petroleum products. Other than by comparisons
to past reports, this approach leaves no obvious way to determine if the
firm failed to report for a particular state---say because accounting
data were unavailable or unclear or the respondent forgot to complete
the form. When large tables need to be completed, the respondent is
instructed to leave blank any fields where the response is zero. Again,
this approach makes it difficult to discern missing data from zero
responses. Making the situation worse are the instructions to enter "0"
for any value between 0 and 1.

RECOMMENDATION: EIA should consider revising forms so that respondents
enter 0 for states or products where they have no activity so that
missing can be distinguished from zero. Conventional rounding should be
used so that values less than 0.5 are rounded to zero.

The 11 surveys also provide detailed instructions for answering the
questions for petroleum-related concepts. Again, though, the survey
methodology constructs are not as detailed and well constructed. The
instructions for several surveys allow the company to define what
constitutes a "firm" and are not precise as to the activities that
should be reported as it relates to the structure of the firm. For the
sample surveys, the petroleum marketing activities that make a firm
eligible for each survey are not explained in detail on the survey form
in the "Who Must Submit" section nor why companies without these
attributes need to respond.

RECOMMENDATION: The forms in use should be reviewed to determine ways to
standardize data collection definitions and procedures across surveys.
Once the definitions of the target populations are tightened up, the
instructions and definitions should be revised to make the reporting
unit precisely defined as well as the level of the company for which the
unit should report. For each unit comprising the target population, the
petroleum marketing activities that they should be reporting should also
be clarified in survey forms.

Across the 11 surveys in the Petroleum Marketing Program, definitions
are provided for petroleum-related concepts that are detailed, easy to
understand, and common across surveys. Other concepts are allowed to
vary across surveys, however, such as what constitutes a "firm" or the
"United States." The variation seems unnecessary, possibly confusing to
the respondent, and perhaps detrimental to integration of data across
surveys.

RECOMMENDATION: The Petroleum Marketing Program is to be commended for
its concise, yet very precise, definitions for petroleum marketing
activities. However, common definitions also need to be provided for
concepts related to the survey design and these definitions need to be
constant across surveys.

Data Analysis, Products, and Documentation:

The data collected for the Petroleum Marketing Program are extensively
used by EIA to produce descriptive reports, analytic reports, and
model-based projections.[^4] The descriptive data are available on a
weekly, monthly or annual basis depending on the underlying data source
within the Petroleum Marketing Program. Weekly data on prices are
published in *Weekly On-Highway Diesel Prices* and *U.S. Retail Gasoline
Prices*, and *This Week in Petroleum*. The latter report is both
analytic and descriptive in nature and does not explicitly list sources
of input data although some articles appear to draw on Petroleum
Marketing Program data on prices for winter heating fuels (EIA-877),
motor gasoline (EIA-878), and diesel fuel (EIA-888). Other descriptive
reports include the *Fuel Oil and Kerosene Sales* Report, the *Petroleum
Marketing Monthly*, and the *Petroleum Marketing Annual.* An examination
of the tables in these reports show all data items collected in the
surveys being used, indicating that the program is not collecting
superfluous data. These reports are easily assessable from EIA's main
website. The descriptive reports include a Notes section as an appendix
with a description of methods used by some but not necessarily all
surveys whose data appear in the report. The descriptions, though
helpful, do not present a thumbnail sketch of all survey methods for the
profiled surveys.

RECOMMENDATION: For each survey component, the Petroleum Marketing
Program should develop a summary description of survey methodology
including survey objectives, target population, sampling frame, sample
design, data capture, editing and imputation, weighting, and time series
changes and other analytic considerations. This description should be
included in the Notes section of reports incorporating the data.

The Petroleum Marketing Program data are also used as input for EIA
energy models. Descriptive and model derived data are used to produce
analytic publications. The petroleum marketing papers are exceptionally
well written, understandable by laymen as well as industry analysts, and
are so interesting it is hard to stop reading them. The reports are not
always specific about the sources of the data and how the estimates were
derived, however. A notable example of how this information might be
conveyed was in the Feb. 24, 2005 *This Week in Petroleum* article on
"Thoughts of Summer in February," which included a link on Chart
Methodology (but did not report data sources).

RECOMMENDATION: EIA is to be commended on its highly readable analytic
reports. Analysts might want to include links that present the methods
and the data sets that were used in the presentation.

With the exception of EIA-821, EIA only publishes data when the
statistic is derived from at least three companies and when one or two
companies do not dominate the statistic. The survey instructions in
Section 8 for these surveys is somewhat strong, however, advising
companies that, "...the risk of disclosure of identifiable information
is very small." The "very" should be struck. Statistics derived from the
Annual Fuel Oil and Kerosene Sales Report (EIA-821) can be published
when they are based upon one or two companies and when one or two
companies dominate the statistic. The companies are advised that the
survey is not subject to statistical disclosure guarantees and that
individually identifiable data may be disclosed.

The documentation of each survey's methodology is limited and
incomplete, with no one document that can serve as the "go to" report to
understand the survey methodology. Because methods are not being
documented and reviewed, there is a potential for errors or substandard
approaches to creep into the surveys, particularly when the survey
contractor or the project officer changes. The Petroleum Marketing
Surveys are used extensively for time series analyses so it is also
important that changes in survey forms and procedures over time be
summarized. Accurate documentation of existing procedures and changes
over time would be valuable for producing survey methods summaries for
use in the Notes sections of descriptive reports and in the OMB package
submission.

RECOMMENDATION: A detailed Survey Methodology Report should be prepared
for each survey in the Petroleum Marketing Program describing the survey
objectives, target population, sampling frame and coverage, sample
design and allocation procedures, data collection and mailout procedures
and forms, weighting and nonresponse adjustment procedures, data
capture, editing and imputation, and analysis and reporting procedures.
An executive summary should be prepared that can be used to produce the
methods description for the OMB package and the Notes appendix for
descriptive reports.

The EIA web site is quite easy to navigate to find information about
petroleum markets with general interest articles like "A Primer on
Gasoline Prices," analytic pieces like "An Oil Market Urban Legend", and
feature articles like "Gasoline Price Pass-through." Survey methodology
is difficult to find on EIA's web site, however. It would be useful to
have a site where the interested reader could go to find descriptions of
each survey's study protocol (like the executive summary referenced
above) and articles and professional presentations that focus on the
statistical design issues and survey methods.

RECOMMENDATION: EIA should set up a link for papers and presentations
that staff members have prepared on statistical design and survey
methodology issues.

Summary of Findings:

The Petroleum Marketing Program is an important and valuable source of
petroleum marketing data. The program is, in general, configured well
and the recommendations given above should make the program stronger,
more effective, and better appreciated by survey respondents and data
users.

[^1]: The Petroleum Marketing 2003 OMB Report, No. 1905-0174, p. 53.

[^2]: Updating the name and address information for firms should not
    introduce bias, for instance. Using sample-derived information to
    remove frame members from strata subject to sampling could introduce
    bias.

[^3]: These response rates were derived from the 2003 OMB package and
    the *Petroleum Marketing Annual 2003*, respectively.

[^4]: These data are also used extensively by state governments and by
    industry analysts, but this report will center on EIA's own
    products.