Using Classification Trees to Recommend Hot Deck Imputation 

Methods: A Case Study

Laura T. Bechtel1, Darcy Steeg Morris, Katherine Jenny Thompson 

Economic Statistical Methods Division, U.S. Census Bureau 

4600 Silver Hill Road, Washington, DC 20233 

Abstract 

The U.S. Census Bureau conducts an Economic Census every five years. Besides collecting a set of common items 

from all eligible establishments, the Economic Census collects detailed information on each establishment’s 

products. Beginning in 2017, the Economic Census will use the North American Product Classification System to 

produce economy-wide product tabulations from cross-sector collections. This marks a major departure from the 

current collection – which explicitly links products to industry – and makes the trade-area specific missing data 

adjustment practices impossible. An interdisciplinary research team was established to address this, and the outcome 

was the recommendation of a single (unified) methodology for the treatment of missing product data. 

 

The team conducted a comprehensive evaluation study using empirical and simulated data from the 2012 and 2007 

Economic Census. In all of the studied industries, a form of hot deck imputation appeared to be the best compromise 

of the considered methods. However, the recommended variation (nearest neighbor or random) was split between 

trade areas. Consequently, the team investigated whether certain properties of establishments or products might be 

predictive of one method having better statistical properties than another across different subdomains. 

Such exploratory analyses readily lend themselves to classification tree analysis. A classification tree starts with a 

categorical outcome (often binary) and grows branches that represent an explanatory variable. The nodes (splits) at 

the top of the tree represent the covariates that are most strongly related to predicting outcome. In our application, 

the outcome was the choice of hot deck method, and the predictors were characteristics of establishment data within 

the imputation cells. In this paper, we describe how we used classification trees to develop an understanding the 

underlying causes that led to one method preferable to another and provide some limited guidance for 

implementation in the 2017 Economic Census and future analyses. 

Introduction 

The U.S. Census Bureau conducts an Economic Census every five years. Besides collecting a set of common items 

from all eligible establishments, the Economic Census collects detailed information on each establishment’s 

products. Beginning in 2017, the Economic Census will use the North American Product Classification System 

(NAPCS) to produce economy-wide product tabulations from cross-sector collections. This marks a major departure 

from the current collection – which explicitly links products to industry – and makes sector-specific missing data 

adjustment practices impossible. Moreover, beginning with the 2017 Economic Census, data collection will be 

electronic, and the respondents will have greater flexibility in reporting products. NAPCS allows the collection of 

the same product in different industries. This will allow products to be classified in different industries. Prior to that, 

each product was only defined within a specified set of industries.

Consequently, an interdisciplinary research team was established to research alternative imputation methods and 

recommend a single (unified) methodology for the treatment of missing product data. The results of this research are 

fully reported in Thompson et al (2014), with the experimental design described in Research Background Section. 

Hot deck imputation produced completed datasets with solid statistical properties in all studied data. However, the 

 

1 Contact Laura.Bechtel@census.gov, Darcy.Steeg.Morris@census.gov, or Katherine.J.Thompson@census.gov.This report is released to inform 

interested parties of ongoing research and to encourage discussion. Any views expressed on statistical or methodological issues are those of the 

author(s) and not necessarily those of the U.S. Census Bureau. 
recommended hot deck variation (nearest neighbor or random) was split between sectors. For production 

implementation, subject matter experts require some guidance on selecting a method for a given industry. In 

practice, little historical data will be available to facilitate this decision-making process besides sampling frame 

properties (e.g., average weight per sampled unit) and historic imputation cell counts (e.g., donor/recipient ratios, 

number of sampled units). Taking this into account, the team investigated which of these imputation cells properties 

might be predictive of improved performance of one hot deck method over the other in a given industry. 

Such exploratory analyses readily lend themselves to classification tree analysis. A classification tree starts with a 

categorical outcome (often binary) and grows branches that represent an explanatory variable. The nodes (splits) at 

the top of the tree represent the covariates that are most strongly related to predicting outcome. In our application, 

the outcome is the choice of hot deck method, and the predictors are characteristics of establishment data within the 

imputation cells. In this paper, we describe how we use classification trees to develop an understanding of the 

underlying causes that lead to one method being preferable to another and provide some limited guidance for 

implementation in the 2017 Economic Census and future analyses. 

Research Background 

The Economic Census collects a core set of data items from each establishment called general statistics items: 

examples include annual payroll, total receipts or shipments, and number of employees in the first quarter. In 

addition, the Economic Census collects information on the revenue obtained from product sales (hereafter referred 

to as “product data”). Product data are collected towards the end of the questionnaire and are requested from all 

establishments included in the Census. The types of products that an establishment is expected to produce or to sell 

are strongly related to the primary industry in which the establishment operates. The number and frequency of 

products reported by an establishment in a given industry varies greatly, depending on the sector and the 

establishment size. However, in a given industry, the majority of the product sales revenue is generally derived from 

six or fewer products (Ellis and Thompson 2015), although the establishment can provide information on many 

more products, depending on the industry in which it operates. In most industries, the frequently reported products 

are highly correlated with total receipts (and generally make up the majority of the total value of receipts), whereas 

the remaining products are not. 

For many sectors, the term “Economic Census” is a bit of a misnomer. The majority of industries comprise a small 

probability subsample of small single-unit establishments, while surveying all multi-unit establishments and the 

largest single-unit establishments. There are exceptions: prior to the 2017 Economic Census, the manufacturing and 

mining sectors utilize cut-off sampling, the construction sector selects a probability proportional to size sample of all 

establishments, and the wholesale trade sector conducts a complete census. In the remaining sectors, the sampling 

rates tend to be quite high (on average, one-in-five) and never exceed one-in-20. With the exception of the 

construction sector, all industries construct a complete universe of general statistics values using administrative data. 

However, product information is collected from only the sampled establishments. 

Prior to the 2017 Economic Census, respondents reported their product data on one of more than 400 industryspecific versions of the questionnaires (paper and electronic), with over 8,000 different products reported. Note that 

many products were rarely reported; product data are characterized by poor item response rates for all but the most 

frequently reported products. There are additivity constraints, with the reported product dollar values expected to 

sum to the total receipts reported earlier in the questionnaire. Missing product data can occur when an establishment 

does not respond to the census (unit nonresponse), when a responding establishment provides no product

information, or when a responding establishment provides product information that does not sum to its total receipts 

(partial product information). 

Recall that the research team was charged with finding a single missing data treatment that worked well in all 

sectors. The best predictors of an establishment’s products are the industry assigned to the establishment from the 

sampling frame (which may change after collection), the total receipts value (RCPTOT), and the value or percentage 

distribution of the other reported products in the same questionnaire. Besides the additivity constraints, there is a 

reasonable expectation that the majority of establishments in an industry report common products; these products 

should be imputed more frequently than other, more rarely reported, products. 



studied data are limited to a small subset of industries focusing on at least two well-represented products. Before we 

examine the relationship of the predictors with the dependent variable, we study the predictors themselves. 

We started with the small pool of possible predictors available from auxiliary information at the imputation 

cell/product level provided in Table 1. While the data are at the imputation cell/product level, we require that the 

variables used for production rules be available at the imputation cell level from historical data. To elaborate, in 

order to implement the criteria specified from the tree, we need (1) adequate historic data available to estimate the 

variables so that they can be calculated at the time of implementation, (2) variables that are available at the 

imputation cell level (not imputation cell/product level) because hot deck imputation is designed to impute all 

products simultaneously for an establishment, not product-by-product, and (3) representative values for the variables 

included in the analysis (not just the “top” two products in an industry’s values). These restrictions eliminate both 

CORR2

 and RNGPROD from use. 

Table 1: List of Available Predictors for Classification Tree 

Predictors Description 

CORR Correlation of products to receipt total 

REC Number of recipients in an imputation cell 

DON Number of donors for a product in an imputation cell 

SIZE Number of donors and recipients in an imputation cell 

DRRATIO Donor (using DON above) to recipient ratio in an imputation cell 

CELLPRODS Number of distinct products reported in an imputation cell 

TRADEPRODS Number of distinct products reported in a trade area (a sector or a combination of sectors) 

RNGPROD Range of product values in an imputation cell 

RNGRCPTOT Range of receipts values at the imputation cell by product level 

WGT_F Percentage of establishments with WGT greater than one at the imputation cell by 

product level 

For the remaining variables, we noted the redundancy in including DON, REC, SIZE and DRRATIO. To make an 

informed choice about which subset of these variables to use, we look at kernel density plots comparing the 

distributions; see Figure 2. We find very subtle differences in the distributions for DON, while the differences in the 

distributions of REC mirror the differences captured by SIZE. Furthermore, we notice that the estimated distribution 

of DRRATIO is shifted slightly higher for the cases where HDN is the preferred method. Based on these visual and 

intuitive differences, we retain DRRATIO and SIZE as predictors, dropping DON and REC. Using a similar

process, we retain CELLPRODS and drop TRADEPRODS (kernel density plots are available upon request). 

 

2 Despite it not being practical for use in the classification tree analysis, exploratory data analysis revealed pleasing 

results about CORR. There is a noticeable difference in the distributions of CORR when separated by winning 

method, indicating that when there is a strong correlation between a product and the receipts value, HDN is the 

preferred method. 

discussion and conclusions on the trees pruned to have a maximum of 5 splits, an arbitrary number selected because 

an overfit tree may perform as well, practically, as a tree comprising fewer branches. Fewer splits also have the 

added benefit of safeguarding against data changes that tend to occur between collection periods. Furthermore, 

imposing a maximum tree size eases interpretation and comparisons across trees (i.e. a high correct classification 

rate can be obtained from an over-fit tree, but that does not necessarily make it better). 

Table 3 shows the correct classification rates for all combinations of tree, pseudo-population and pruning method. 

Note that, not surprisingly, we observe the highest correct classification rates for each pseudo-population when the 

tree is fit and evaluated on the same data (the diagonals of the table). Instead, we will focus on the cross-validated 

cells. Looking at the results from the maximum split pruning approach, we find that the tree fit from the combined 

pseudo-populations (ALL) performed the best across all pseudo-populations. Furthermore, the correct classification 

rates are higher from implementing the ALL trees than from assigning a hot deck method at random based on 

observed rates. For example, in the HDN pseudo-population, assigning hot deck method via the specifications from 

the ALL tree achieves a correct classification rate of 0.647. If all cases had been selected for imputation using HDN 

(the “winning” method in this population), then the correct classification rate would be 0.567. In this case, the CC 

rate obtained from the tree is clearly preferable. Note that the correct classification rates for the ALL tree using the 

one standard error rule are misleadingly high because the tree is overfit with minimal pruning, which can lead to 

predictions that are not robust. 

Table 3: Correct Classification (CC) Rates 

Tree Observed 

PseudoPopulation ALL HDN HDR SRMI CC Rate Winner 

Pruning: Maximum Number of Splits = 5 

ALL .625 .577 .604 .608 .512 HDN

HDN .647 .746 .532 .572 .567 HDN

HDR .659 .497 .746 .578 .566 HDR

SRMI .570 .488 .535 .674 .535 HDN

Pruning: One Standard Error Rule* 

ALL .732 .595 .581 .512 .512 HDN

HDN .751 .757 .503 .567 .567 HDN

HDR .728 .555 .699 .434 .566 HDR

SRMI .715 .471 .541 .535 .535 HDN

*HDR tree was pruned to the lowest cross-validated error because the one standard error rule resulted 

in a single node. SRMI tree is null – the lowest cross-validated error is observed with zero splits. 

Figure 3 graphically displays the rules determined by the classification tree grown on the combined pseudopopulations (ALL). In this figure, darker terminal nodes denote an HDN “win.” The predicted winner is determined 

to be the method with the empirical majority in the given node. Predicted HDR “win” probabilities are displayed 

below the HDN/HDR prediction followed by the number of observations that follow that path in the classification 

tree. 

The classification tree results3

 indicate that HDR should only be used when the establishments in the imputation cell 

are observed to report a variety of products, have many more recipients than donors (a donor-to-recipient ratio that is 

much smaller than one), have large sample sizes within imputation cell (at least 1,000 units in our study data), and 

contain a not insubstantial sample contribution from noncertainty units (more than ~10% of the weights are greater 

than 1). Otherwise, HDN yields improved performance in terms of imputation error. 

This intuitively makes sense. Nearest neighbor imputation is most effective when the auxiliary variables used in the 

distance function are highly correlated with the items being imputed. When the establishments report a handful of 

products (e.g., three of four), this assumption is valid. As the average number of reported products increases, this 

assumption is less valid. When there are more recipients than donors, then it is less likely that the recipient’s 

 

3

 Notice that RNGRCPTOT is not included in the tree indicating that it is not a strong predictor. 
“nearest neighbor” is actually close (similar) to the recipient. Large cell sizes combined with relatively low donor to 

recipient ratios just exacerbates this problem. When there are a lot more recipients than donors, HDR has a less 

restrictive donor pool available to each recipient than HDN. Finally, the sample composition and the donor selection 

method should be intertwined. With a high proportion of noncertainty units, it is not unreasonable to assume that 

donors are somewhat interchangeable, giving an advantage to random selection. With the sample design used in the 

census, the inverse is actually true, and the donors tend to be quite dissimilar. 

Figure 3: Classification Tree for Imputation Error 

One thing to note is that in the actual application of the hot-deck imputation methods, the covariate values indicated 

in the tree nodes may be very different as our pseudo-population data are not necessarily representative of the entire 

Economic Census. Therefore, some translation may be needed to match the values of splitting variables with the 

distribution of the observed data. For this translation, we propose looking at the percentile values associated with the 

splitting values and using those to find a more appropriate splitting value for the observed data. We find that the 

splits values in the classification tree correspond to about the 15th, 80th, 45th and 26th percentile for CELLPRODS, 

DRRATIO, SIZE, and WGT_F, respectively. That said, we recommend retaining the same explanatory variables 

and the same form of splitting rules for determining the default hot deck variation by industry. 

For example, a preliminary analysis of the donor-to-recipients ratios in the production imputation cells for on 

selected 2012 Economic Census product data obtained by a separate team showed median values above one and 

below two in the studied sectors; approximately 10-percent of these imputation cells did have less than six donor 

records and more than six recipient records (tables available upon demand from the authors). Consequently, the 

distribution of donor-to-recipient ratios in our study data appears to deviate greatly from the full census distribution. 

In this case, the imputation cell characteristics in the Economic Census are even more conducive to the HDN 

conditions presented in the tree above than shown in our datasets. 
Conclusion 

Often, the focus of exploratory research is to determine a viable solution (or viable solutions) to a specific problem. 

When a single solution presents itself, then the research recommendation easily translates to a production 

application. However, when two or more equally satisfactory solutions present themselves, then the production 

application may be determined on other factors, such as cost or timeliness or ease of implementation. If these other 

practical factors cannot serve as a tiebreaker, then the production application determination might be made on less 

tangible factors, such as personal preference. 

The product imputation research for the Economic Census that motivated this research is such a problem. In the 

studied datasets, both hot deck methods produced microdata with similar properties in terms of FMI and very few 

differences in imputation error. Both methods are equally easy to program, run in about the same amount of 

computing time, require about the same amount of computing space, and can be easily customized to meet survey 

needs. Nevertheless, there were instances where one method outperformed the other. The processing schedule for 

the Economic Census is tight, especially for product data, which are reviewed at the end of the collection and 

processing cycle. There is little time for rework, and it is therefore important to designate an imputation method 

from the beginning that avoids instances of poor performance and requires extensive rework. 

Applying classification trees to the outcome data provided a useful way of categorizing the imputation cell 

characteristics to develop a single rubric for imputation method choice by industry. Using the intuitive rules derived 

from the classification tree, we achieved higher correct classification rates than random assignment indicating that 

we have identified strong criteria to discriminate between HDR and HDN. In part, these rules work because we 

chose a small list of predictors after performing exploratory data analysis. Indeed, we found that we needed a 

hypothesis accompanying each predictor to obtain an interpretable tree; earlier (unreported) applications that used a 

larger list of available covariates yielded classification rules that would likely not apply to the larger census data. 

The need to specify and interpret covariates before classification is a departure from other practices such as those 

reported in Phipps and Toth (2012) who used regression trees to develop response propensity models. 

With the Economic Census, different reporting behaviors are often exhibited between large and small businesses in 

the same industry. Although there has been little research on the relationship between product reporting and 

establishment size, subject matter experts have expressed some reservations about randomly selecting donors 

without accounting for establishment size in the selection process. The presented results mirrored our subject matter 

experts’ intuition, employing a statistically defensible method to arrive at rules they might have arrived at on their 

own. 

Acknowledgements 

The authors thank Carma Hogue, Yves Thibaudeau, and John Ward for their careful review of earlier versions of 

this manuscript. 

References

Andridge, R. and Little, R. (2010). “A Review of Hot Deck Imputation for Survey Non-response.” International 

Statistical Review, 78 (1), pp 40-64.

Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Monterey, 

CA: Wadsworth & Brooks/Cole Advanced Books & Software. 

Chen, J. and Shao, J. (2001). “Jackknife Variance Estimation for Nearest-Neighbor Imputation.” Journal of the 

American Statistical Association, 96(453), pp. 260-269. 

Dong, Q., Elliott, M. R., and Raghunathan, T. E. (2014). “A Nonparametric Method To Generate Synthetic 

Populations To Adjust For Complex Sampling Design Features.” Survey Methodology, 40(1), pp. 29-46.

Ellis, Y. and Thompson, K.J. (2015). “Exploratory Data Analysis of Economic Census Products: Methods and 

Results.” In JSM Proceedings, Section on Survey Research Methods. Alexandria, VA, American Statistical 

Association. 

Garcia, M., Morris, D., and Diamond, L.K. (2015). “Implementation of Ratio Imputation and Sequential Regression 

Multiple Imputation on Economic Census Products.” In JSM Proceedings, Section on Survey Research 

Methods. Alexandria, VA, American Statistical Association. 
Knutson, J. and Martin, J. (2015). “Evaluation of Alternative Imputation Methods for U.S. Census Bureau Economic 

Census Products: the Cook-Off.” In JSM Proceedings, Section on Survey Research Methods. Alexandria, 

VA, American Statistical Association. 

Nordholt, E.S. 1998. “Imputation: Methods, Simulation Experiments and Practical Examples.” International

Statistical Review: 66(2), pp. 157-180.

Phipps, P. and Toth, D. (2012). Analyzing Establishment Nonresponse Using An Interpretable Regression Tree 

Model With Linked Administrative Data. The Annals of Applied Statistics, 6(2), pp. 772-794. 

Raghunathan, T. E., Lepkowski, J. M., Van Hoewyk, J., and Solenberger, P. (2001). “A Multivariate Technique for 

Multiply Imputing Missing Values Using a Sequence Of Regression Models.” Survey Methodology, 27(1), 

pp. 85–95. 

Therneau, T., Atkinson, B. and Ripley, B. (2013). rpart: Recursive Partitioning. R package version 4.1-3. 

http://CRAN.R-project.org/package=rpart 

Thompson, Katherine J., Liu, X., Bechtel, L., Diamond, Lisa K., Davie Jr., W., Dorsett, F., Ellis, Y., Garcia, M., 

Kern, J., Knutson, J., Martin, J., Morris, Darcy S., Schuyler, J., Struble, R., Tolliver, K., Ward, J., and 

Thibadeau, Y. (2014). “Recommendation for Product Line Imputation for 2017 Economic Census: Report 

from the Product Line Research Team.” The Census Bureau internal document, dated 12/9/2014. 

Tolliver, K. and Bechtel, L. (2015). “Implementation of Hot Deck Imputation on Economic Census Products.” In 

JSM Proceedings, Section on Survey Research Methods. Alexandria, VA, American Statistical 

Association.