# Presentation: 439521

## Simulation Results of Probability Proportional Size Sampling for EIA's Monthly Natural Gas Production Survey

- Preston McDowney
- October 28, 2004

## Outline

- Recap the Inderjit Kundra’s Spring 2004 Presentation
- Review the Steps of the Simulation Project
- Review the Results of the PPS Simulations
- Introduce Plan “B”

## 2004 Spring Presentation Recap

- EIA was in the process of developing a new survey, the EIA-914, to collect monthly production of natural gas in the United States
- EIA was considering a Probability Proportion to Size (PPS) sample.
- The goals were to estimate total US production with a coefficient of variation (CV) of 1% and regional production with a CV of 5%.

## Regions

-  Texas (TX)
-  Federal Gulf (FG)
-  Louisiana (LA)
-  New Mexico (NM)
-  Oklahoma (OK)
-  Wyoming (WY)
-  Others (excluding Alaska)

## How 

- Sample size was determined by using Presumed Optimum Allocation formula

- Where CSS	denotes calculated Sample Size
- 	h	denotes stratum in a region
- 	_N(h)_	denotes number of operators in a stratum
- 	S(h)	denotes standard Deviation in stratum h
- 	CV	denotes coefficient of variation
- 	y(hi)	denotes the production for operator i in stratum h

##  Estimated Sample Sizes * 

**1**** An operator within a region having a measure of size greater than or equal to total production divided by 2n was selected with certainty.****  **
*** Additional information in Supplemental Slides**

| 1% CV |  |  |  |
| --- | --- | --- | --- |
| Region | Certainty Group1 | Certainty Coverage | Proposed Sample Size |
| US | 95 | 82% | 176 |

| 5% CV |  |  |  |
| --- | --- | --- | --- |
| Region | Certainty Group1 | Certainty Coverage | Proposed Sample Size |
| TX | 52 | 74% | 96 |
| FG | 12 | 72% | 20 |
| LA | 34 | 72% | 57 |
| NM | 15 | 80% | 28 |
| OK | 35 | 72% | 63 |
| WY | 25 | 93% | 31 |
| OTHERS | 38 | 73% | 63 |
| SUBTOTAL | 211 | - | 358 |

**Notes:**
Using the formula on the previous slide we calculated that 176 operators would give us the 1% cv at the us level, 95 certainty companies covering 82% of  total national production.

To get the 5% cv at in each of the regions, we need 358 operators, with 211 certainty operators.  However, almost half of the certainty operators were selected with certainty in multiple states, so the actual number of unique operators in that group was approximately 102 after accounting for multistate reporting.

The Committee didn’t have any objects with this approach and gave the green light. 

We were really happy with this sample size.  So happy that we actually increase the cv in FG to the 1%, see the supplement slide

In theory this project was looking golden and we were excited
I ran back to my office and began to test our design with simulations

## Steps of the Simulation Project

- Created a “pseudo frame”
- Selected the certainty groups *
- Selected PPS samples *
- Estimated the total production of the “pseudo frame” of each sample using variations of four procedures

*** Additional information in Supplemental Slides**

**Notes:**
I will talk about the pseudo frame in more detail shortly
Then we selected the certainty groups from the pseudo frame, These certainty operators were selected for each region independently and remained constant throughout the simulations.  more information in the supplemental slides.

Then we selected the pps samples, one region at time.

## The Pseudo Frame	

- We created a “pseudo frame” of operators common to both the 2000 EIA-23 sample and the 2002 EIA-23 Frame
- This was done to adjust for not having a complete frame for 2000.
- The two year difference reflects the approximate preparation period needed to complete and finalized data for a given year

**Notes:**
So when its time to select the sample for the 2005 survey year, the 2004 data probably will not be finalized, so we will end up using 2003 data to select our operators

## Four Estimation Procedures

- Probability Proportion to Size (PPS) Estimator *
- Weighted Least Squares (WLS) Estimator *
- Ordinary Least Squares (OLS) Estimator *
- Difference Estimator *

*** Additional information in Supplemental Slides**

**Notes:**
The pps estimator uses the inverse of the probability of selection to weigh the responses of the operators in the pps sample

The weighted least squares estimator is the classic ratio estimators

The ordinary least squares estimator

And difference estimator, where the ratio 1, and uses the 2000 values of the nonsampled operators as their estimate in 2002.

## Variations of the WLS and OLS

- Two Methods
- Using the non-certainty group only in the estimation procedure
- Using both the certainty and non-certainty groups in the estimation procedure
- Three Variations
- Using all of the operators in the method
- Removing outliers *
- Removing outliers and overly influential observations *

*** Additional information in Supplemental Slides**

**Notes:**
Outlier if the absolute value of studentized residual was greater than 3.5
 Influential if the DFFITs value was greater than 2 over the sqrt of the number of operators in that region

## Simulation 

- @ Risk was used generate random samples and to compute summary statistics of all the simulations
- 10,000 samples were simulated

**Notes:**
As we were developing the simulation we were running the simulation using 100 simulation and getting slightly different results each time.   So then we simulated 1000 sample at a time and got more consistent results, so then we up it to 10000 samples.

While it might be a overkill, we achieved steady state so speak.

## Simulation Results,Percent Error Estimates for Total Natural Gas Production * 

*** Additional information in Supplemental Slides**

**Notes:**
These are some of the summary statistics from the simulation.
This slide only shows the results without removing outliers and influential observations.  The sup. Slides have a similar slide with the outliers and influential observation removed.

The pps estimator has the smallest mean percent error, but the highest range of errors.

The regression estimator all have a much tighter range of errors, but they all have a negative bias.

None of the estimators give of the desired cvs

90 % of the errors falling between

## Conclusions

- The PPS sampling did not provide the desired accuracy
- None of the estimators produced the desired CVs
- All of the estimation procedures had a negative bias
- Removing the outliers and influential observations had little impact
- Decided to use a cut off sample with 90 percent coverage at the national level