What is NCHS doing to 

get “fit”? 

Amy M. Branum, PhD 

Acting Associate Director for Science 

National Center for Health Statistics 
Home of NHANES, NHIS, NSFG, health care surveys, and vital 

statistics

The US’ federal statistical agency for health

NCHS - who are we? 
ANNUALLY

What do we produce?

` 

 ~100 

scientific 

reports 

and 

analyses 

Dozens of 

data files 

Reams of 

technical 

documentation 

Hours of 

technical 

expertise and 

guidance
Production of high-quality, reliable, transparent statistics on the 

health of the US that educate, inform, and shape health policy 

What are we known for?
https://www.cdc.gov/nchs/covid19/index.htm

NCHS’ COVID-19 index page has received 14.2 million hits since March 

COVID-19 mortality pages have received 8 million hits
Needed a more consistent approach across data divisions 

Narrowly focused on standard errors as measures of variation

Needed better guidance for staff and data users

What is reliable? What is “fit”? Why did we need to revisit this?

Statements from ASA and others regarding p-values

Q

A

A

A

A
Internal workgroup 

formed 

Recognized a need to have clear and 

transparent presentation criteria that can be 

broadly and efficiently implemented 

Published this report as a culmination of that 

effort

WWW.CDC.GOV/NCHS/DATA/SERIES/SR_02/SR02_175.PDF
estimate from a complex survey, the effective sample size, 

n

e, is defined as

the sample size, n, divided by the design effect (7). One approach used to calculate ne for estimated proportions from a complex 

sample survey is: 

where, in this case, the design effect is: 

Documentation for specific surveys should be consulted when calculating design effects, as approaches can differ among surveys 

and for specific analytic purposes. 

If the number of numerator events

is 0 or equal to the denominator (the complement of 0 events), the estimated proportion will be 0 or 1, respectively.

As a result, the estimated variance of the proportion will be 0, and the effective sample size will be 

undefined. In these cases, the sample size should be used to determine whether the minimum sample

size criterion is met, and it should also

be used for CIs and other computations that include the effective sample size. Because observing no 

events or events for everyone in a category can provide important information (e.g., in the context of 

rare health outcomes or conditions), estimates based on 0 events (or the complement) that meet 

absolute CI and degrees of freedom criteria should be flagged and considered for presentation after 

statistical review by a clearance official to confirm the validity of the point and interval estimates. 

For complex sample surveys, due

to sampling design and variability,

there may be cases where the effective sample size is greater than the sample size. When the effective 

sample size is greater than the sample size, the sample size should be used to determine whether the 

minimum sample size criterion is met, and it should also be used for CIs and other computations that 

include the effective sample size. 

Standard ● Estimated proportions should be based on a minimum denominator sample size and effective 

denominator sample size (when applicable) of 30. Estimates with either a denominator sample size or 

an effective denominator sample size (when applicable) less than 30 should be suppressed. ● If the number of numerator events

is 0 (or its complement), then the denominator sample size should be used to obtain confidence 

intervals. If all other criteria are met for presentation, an estimate based on 0 events (or its 

complement) should be flagged for statistical review by the clearance official. The review could 

result in either the presentation or the suppression of the proportion. 

Confidence Intervals 

The NCHS Data Presentation Standards for Proportions are based on the evaluation of absolute and relative 95%

CI widths. CIs provide a way to assess

an estimate’s precision, and technical definitions are available in many standard statistical texts, including Bickel and Doksum (8) 

and Casela and Berger (9). More generally, under repeated sampling, if a proportion and its 95% CI are estimated from each 

sample, the true value of the proportion is expected to be contained in 95% of the calculated intervals. A handful of methods to 

calculate CIs for proportions are available and the expectation of 95% coverage may not be attained for some intervals or under 

some conditions. Methods used to calculate a CI lead to undercoverage if the true proportion is contained in fewer than the 

expected number of intervals (e.g., less than 95%). Conversely, methods are considered conservative if the true proportion is 

2

Degrees of Freedom 

For complex sample surveys, the precision of the estimated variance is approximately related to the square root

of the degrees of freedom. Using resulting SEs with low precision to assess estimated proportions may lead to poor measures

of effective sample size and CI widths. Under certain conditions, the variance estimate is approximately proportional to a chisquared distribution, and the RSE

of the variance obtained from a complex sample survey can be approximated as 

. From this expression, estimated proportions based 

on fewer than 8 degrees of freedom have !

freedom can be calculated as the number of PSUs minus the number of strata. This calculation is used 

in most NCHS surveys and implemented in survey software, although specific calculations can vary 

across packages. However, default calculations of degrees of freedom from survey software may not 

be appropriate for subgroups represented in only a subset of PSUs (e.g., some racial and ethnic 

groups and region-specific estimates) and when calculating annual or survey cycle estimates using a

multiyear or multicycle data file. In these instances, the relevant information should be extracted and 

the degrees of freedom directly calculated to assess estimate precision. The calculation of degrees of 

freedom as a measure of precision for the SE may not be applicable for all surveys (see surveyspecific documentation) and does not apply to vital statistics. For additional information on degrees 

of freedom, see Korn and Graubard (13) and Valliant and Rust (14). 

Standard ● When applicable for complex surveys, if the sample size and CI criteria are met for presentation 

and the degrees of freedom are fewer than 8, then the proportion should be flagged for statistical 

review by the clearance official. This review may result in either the presentation or the suppression 

of the proportion. 

Complementary Proportions 

The SE and width of the CI for the complement of a proportion (1 – p) are the same as those for the 

proportion, p. As described in previous sections, relative measures for the smaller proportion are much larger 

than for its larger complement. Consequently, there is a range of proportions where the CI criteria 

will yield conflicting assessments of reliability. For these proportions, the relative CI width may indicate that a small proportion 

is unreliable but that its complement is not. 

For a given health indicator or publication, the larger proportion may be the most salient measure, while for others, the smaller 

proportion may be the most important. Typically, both proportions are not shown (e.g., only

the proportion with health insurance would be shown, not both the proportion with and the proportion without health insurance). 

Given that the complement of the presented proportion can be determined by subtraction, consideration of the precision of the 

complement

is important. For some publications,

the practice has been to suppress both proportions if one of the proportions is identified as unreliable. However, this practice may 
NCHS highly recommends that data users familiarize themselves with 

the data standards for proportions and implement them when using 

NCHS data 

Standards can be implemented with a number of software packages

STANDARDS FOR RATES IN PROGRESS AND COMING SOON!
In our “big data” systems, does a 0.1 

percentage point change mean anything 

even if statistically significant at the 

p<0.05 level? 

NO CONCRETE GUIDANCE ON NAVIGATING 

SIGNIFICANCE

BIG CHANGES, LIMITED STATISTICAL POWER

NCHS has not issued guidance to staff 

on how to assess statistical significance 

Work in progress

Some surveys have had decreasing 

sample sizes 

Harder to retain sufficient power to detect 

seemingly large differences

3

4

1

2 SIGNIFICANT BUT NOT MEANINGFUL CHANGES

ASSESSING SIGNIFICANCE 

NCHS has a few challenges 

WHAT ABOUT THAT P-VALUE?