# Presentation: 667402

## Statistical Significance

- Tony Tersine, U.S. Census Bureau

**Notes:**
Good morning

## Overview of the Session

- Basic Concepts
- Margin of Error
- Confidence Intervals
- Standard Error Formulas
- Statistical Testing

## Basic Concepts - 1

- _Sampling error_ is introduced due to sampling, selection of  a subset of the population to draw inferences about the entire population.
- _Standard error_ is an estimate of the precision of the estimates. It measures the variability of an estimate due to sampling.

## Basic Concepts - 2

- The sampling error is often reported as the estimate “plus or minus” the **margin of error, **a measure of how precise the estimate is.
- The _margin of error_ describes the precision of the estimate at a given confidence level.

## Basic Concepts - 3

- The _confidence level_ measures the likelihood that the true value is within the margin of error of the sample estimate.
- The Census Bureau statistical standard for published data is to use the 90 percent confidence level.

## Margin of Error

- The margin of error is important because relying on statistical inference can save you from drawing incorrect conclusions from data based on a sample.
- It can help prevent you from interpreting small or nonexistent differences as important.

## Margin of Error (MOE)

- MOE = 1.65 * Standard Error
- 	1.65 is used for the 90 percent confidence level
- Standard Error = MOE / 1.65

## Confidence Interval

- Confidence IntervalEstimate ± Margin of Error
- 90 percent confidence levelMargin of Error = 1.65 * Std Error
- 95 percent confidence levelMargin of Error = 1.96 * Std Error

## Confidence Interval

- The confidence interval tells you the upper and lower bounds of a range of values that may contain the true value.
- It provides important information about the true value or the population parameter.
- It tells you the limitations on using the estimates.

## MOE / Confidence Interval

- Median Family Income – $30,000
- Standard Error – $1,500
- 90% MOE = 1.65 * $1,500 = $2,475
- 90% CI = $30,000 ± $2,475 =
-   $27,525 to $32,475

## Standard Error – Sum/Difference

- Standard Error of X + Y or X – Y
- SE(X+Y)  =  SE(X-Y)

**Notes:**
Will underestimate the SE if two items in a sum are highly positively correlated or if the two items in a difference are highly negatively correlated.
Will overestimate the SE if two items in a sum are highly negatively correlated or if the two items in a difference are highly positively correlated.

V(X + Y) = V (X) + V(Y) + 2 COV(X,Y)
you get COV(X,Y) = 1/2 [V (X + Y) - V(X) - V(Y)]

From the collapsed and detailed tables get the appropriate values and
compute COV(X,Y).  You can get it directly or through an approximation
using characteristics thought to have the same degree of correlation.
Again, the user will need to use his judgment to make this determination.

## Standard Error – Sum

- SE(X1+X2+...+Xn)

## Standard Error – Proportions

- P= X / Y – X is a subset of Y
- SE(P)

## Standard Error – Ratios

- X / Y – X is not a subset of Y
- SE(X / Y)

## Statistical Testing

- Two estimates are "significantly different" at the 90 percent confidence level if the difference between them is large enough to infer that there was a less than 10 percent chance that the difference was purely random.
- Users may want to compare estimates across years or geographies.
- It is important to note that small differences, which may be statistically significant, may not have any practical significance.

**Notes:**
This information is included in the “Using Data from the 2005 American Community
Survey” under the “Using the Data” tab on the ACS website.

## Statistical Testing - Steps

- State that two estimates are statistically different if the difference between the two estimates is statistically different from zero.
- Calculate the standard error of the difference.

## Statistical Testing - Steps

- Calculate the margin of error of the difference.
- Compare the original difference between the estimates to the margin of error of the difference.

## Statistical Testing - Steps

- If the difference is greater than the margin of error, then you conclude that the two estimates are significantly different.
- If the difference is less than the margin of error, you conclude that the two estimates are not significantly different.

## Statistical Testing - Example

- Percent with Bachelor’s Degree or Higher
- Geography	Percent	MOE	 CI
- Area 1		   20.0	± 5.0	   15.0-25.0
- Area 2		   12.3	± 4.7     7.6-17.0
- Difference = 20.0 – 12.3 = 7.7

## Statistical Testing - Example

- MOE of the Difference - Standard Errors for Each Estimate
- 	SE = MOE / 1.65
- SE(Area 1) = 5.0 / 1.65 = 3.03
- SE(Area 2) = 4.7 / 1.65 = 2.85

## Statistical Testing - Example

- Standard Error of the Difference
- Margin of Error of the Difference
- 	MOE(X - Y) = 1.65 * 4.16 = 6.9

## Statistical Testing - Example

- Compare the Difference to MOE
- Difference = 7.7%
- MOE = 6.9%
- Difference > MOE
- Conclude that the two estimates are significantly different with 90 percent confidence

## Census 2000 – Example

-   Percent Bachelor’s Degree or Higher – Alexandria, VA
-   51,982 / 95,730 = 54.3%
-   DF = 1.2 – (13.4% in sample)
-   90% MOE =1.65 * 0.4 = 0.7
-   90% CI = 54.3 ± 0.7 = 53.6 to 55.0

## Rules to Remember

- Don’t make a big deal of small differences.
- If the confidence intervals overlap you **cannot **conclude the difference is not statistically significant.
- Always talk to subject matter experts before making any conclusions.

## Contact Information

**Contact Information**