Model Compression
Cristian Bucilă

Rich Caruana

Alexandru Niculescu-Mizil

Computer Science
Cornell University

Computer Science
Cornell University

Computer Science
Cornell University

cristi@cs.cornell.edu

caruana@cs.cornell.edu

alexn@cs.cornell.edu

ABSTRACT
Often the best performing supervised learning models are
ensembles of hundreds or thousands of base-level classifiers.
Unfortunately, the space required to store this many classifiers, and the time required to execute them at run-time,prohibits their use in applications where test sets are large(e.g. Google), where storage space is at a premium (e.g.
PDAs), and where computational power is limited (e.g. hearing aids). We present a method for “compressing” large,
complex ensembles into smaller, faster models, usually without significant loss in performance.
Categories and Subject Descriptors: I.5.1 [Pattern Recognition]: Models – Neural nets.
General Terms: Algorithms, Experimentation, Measurement, Performance, Reliability.
Keywords: Supervised Learning, Model Compression

1.

INTRODUCTION

An ensemble is a collection of models whose predictions
are combined by weighted averaging or voting. Ensemble
methods have been the focus of significant research in thepast decade, and a variety of ensemble methods have been
introduced. Well known ensemble methods include bagging[2], boosting [14], random forests[3], Bayesian averaging [9]and stacking [17]. Much of the interest in ensemble methods
has been fueled by their excellent empirical performance.
Ensembles, however, have one disadvantage that often
is overlooked: many ensembles are large and slow. This
makes ensemble methods unusable for applications with limited memory, storage space, or computational power such as
portable devices or sensor networks, and for applications inwhich real-time predictions are needed. Consider, for example, boosted decision trees, bagged decision trees or random
forests. These models often contain hundreds or thousandsof decision trees, each of which must be stored, and executedat run-time to make predictions. Executing a single tree isfast, but executing a thousand trees is not.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
KDD’06, August 20–23, 2006, Philadelphia, Pennsylvania, USA.
Copyright 2006 ACM 1-59593-339-5/06/0008 ...$5.00.

In this paper we show how to compress the function thatis learned by a complex model into a much smaller, fastermodel that has comparable performance. Specifically, weshow how to train compact artificial neural nets to mimic thefunction learned by ensemble selection, an ensemble learning
method introduced by Caruana et al. [5]. To achieve this,
we take advantage of the well known property of artificial
neural nets, namely that they are universal approximators:
given enough training data, and a large enough hidden layer,
a neural net can approximate any function to arbitrary precision. Instead of training the neural net on the original(often small) training set used to train the ensemble, we use
the ensemble to label a large unlabeled data set and thentrain the neural net on this much larger, ensemble labeled,data set. This yields a neural net that makes predictions
similar to the ensemble, and which performs much better
than a neural net trained on the original training set.
The key difficulty when compressing complex ensembles
into simpler models this way is the need for a large unlabeled data set. In some domains, unlabeled data is easy toobtain. In other domains, however, large data sets (labeled
or unlabeled) are not available. In these domains, we generate synthetic cases that as closely as possible match the
distribution of the original training set. We introduce a new
method for generating synthetic cases called MUNGE that
outperforms other methods to which we have compared it.
Using MUNGE, we are able to train neural nets that are athousand times smaller and faster than ensemble selection
ensembles, but which have nearly the same performance as
the far more complex ensembles.

2. MODEL COMPRESSION
In some situations, it is not enough for a classifier or regressor to be highly accurate, it also has to meet stringent
time and space requirements. In many cases, however, thebest performing model is too slow and too large to meetthese requirements, while fast and compact models are lessaccurate, because either they are not expressive enough, orthey overfit to the limited training data. For such situations,
we propose using model compression to obtain fast, compact
yet highly accurate models.
The main idea behind model compression is to use a fastand compact model to approximate the function learned bya slower, larger, but better performing model. Unlike thetrue function that is unknown, the function learned by ahigh performing model is available and can be used to label
large amounts of pseudo data. A fast, compact and expressive model trained on enough pseudo data will not overfit

and will approximate well the function learned by the highperforming model. This allows a slow, complex model suchas a massive ensemble to be compressed into a fast, compact
model such as a neural net with little loss in performance.
An important question is how do we get the pseudo data.
In some domains large amounts of unlabeled data is easy
to collect (e.g. in text, web and image domains) and canbe used as pseudo data. In other domains, however, unlabeled data is not readily available and synthetic cases need
to be generated. This is more difficult than it might seemat first. It is important that the synthetic data match well
the distribution of the real train and future test cases. Usually real data lay in a small submanifold of the complete
attribute space. If the synthetic data is drawn from a distribution that has little overlap to this manifold, the labeled
synthetic points will fail to capture the target function in
the region of interest. On the other hand, if the distribution from which the synthetic data is sampled is too broad,
only a fraction of the points will be drawn from the true
manifold and many more samples will be necessary to adequately sample the region of interest. The best case is when
the synthetic distribution is very similar to the true distribution. Then a minimum number of samples will be necessary
to adequately sample the target function.
We experiment with three methods of generating pseudo
data: RANDOM, generate data for each attribute independently from its marginal distribution; NBE, estimate thejoint density of attributes using the Naive Bayes Estimation
algorithm [12] and then generate samples from this joint distribution; and MUNGE, a new procedure we propose thatsamples from a non-parametric estimate of the joint density.

2.1 RANDOM
The simplest way to generate pseudo data is to independently sample the value of each attribute from the marginal
distribution of that attribute. This is the procedure predominantly used in the literature whenever there is a need for
artificial data (e.g. [13, 6]). Usually the nominal attributes
are generated from a multinomial distribution whose parameters are estimated from the training data. The continuous
attributes are usually modeled using a uniform distribution,a Gaussian distribution with mean and variance estimatedfrom the training set, or via kernel density estimation [15].
The RANDOM method for generating pseudo data uses
a nonparametric bootstrap approach. For each attribute,a value is selected uniformly at random from the multiset(bag) of all values for that attribute present in the train set.1
When the attribute values are generated independently,
all conditional structure is lost and the pseudo examples are
generated from a distribution that is usually much broader
than the true distribution of the data. As a consequence
many of the generated pseudo examples will cover uninteresting parts of the space, and this may prevent the mimicmodel from focusing on the important regions.

2.2 NBE
Another approach to generating pseudo data is to estimate the joint distribution of attributes using the training
set, then sample pseudo examples from this joint distribu1
For nominal attributes this is equivalent to generating thevalues from the multinomial distribution. For continuous
attributes this procedure is slightly different than previously
proposed ones, but generates similar values in practice.

tion. Assuming that the true joint distribution can be estimated well, the conditional structure of the domain would
be preserved and the new artificial examples would coverwell the interesting regions of the space.
One way to estimate the joint distribution of a set of variables is to use mixture model algorithms. These algorithms
model the data as coming from a mixture of components,
each component with a different distribution. The most wellknown algorithm in this category, used in domains with only
continuous attributes is the mixture of Gaussians [7], where
each component consists of a Gaussian distribution with adifferent mean and covariance matrix.
A mixture model algorithm that handles both discreteand continuous attributes, NBE (Naive Bayes Estimation),
was recently introduced by Lowd and Domingos [12]. We
used NBE to estimate the joint distribution of the attributes
because it handles mixed attributes, it is simple to use, it
performs as well as learning a Bayesian Network from thesame data [12], and it is readily available.

2.3 MUNGE
Estimating a full joint distribution is difficult when there
are many attributes and few training cases. Instead of trying
to reliably estimate a joint distribution, we have developed a
new algorithm that samples directly from a non-parametricestimate of the joint distribution.
Algorithm 1 MUNGE
Require: set of training examples T , size multiplier k,probability parameter p, local variance parameter s
Returns: unlabeled training set D of size k × size(T )1: D ← ∅2: loop k times
3:
T0 ← T4:
for all examples e in T 0 do
5:e0 ← the closest example of e from T 0
6:
for all attributes a of example e (excluding thelabel attribute) do
7:
if a is continuous then8:
with probability p: ea ← norm(e0a , sd), ande0a ← norm(ea , sd), where sd ← |ea − e0a |/s,
and norm(a, b) is a random value taken fromthe normal distribution with mean a and standard deviation b.
9:
else
10:
with probability p: swap the values of attributea for examples e and e0
11:
end if
12:
end for13:
end for14:
D ← D T0
15: end loop
16: Return D
Starting from the original training set, we visit each example once and determine its closest neighbor. To measure
distance between cases, we use euclidean distance for continuous attributes and hamming distance for nominal attributes. Continuous attributes are linearly scaled to [0,1].
Given example e and its closest other example e0 , the values for each noncontinuous attribute are swapped betweene and e0 with probability p and are left unchanged withprobability 1 − p. For each continuous attribute a, with

TRUE DIST

NBE

Table 1: Description of problems.

RANDOM

MUNGE

problem#attr

train size

test size

%poz

adult
covtypehs
letter.p1
letter.p2
medis
mg
slac

14/104
54
200
16
16
63
124
59

4000
4000
4000
4000
4000
4000
4000
4000

35222
25000
4366
14000
14000
8199
12807
25000

25%
36%
24%
3%
53%
11%
17%
50%

0.3

Figure 1: Synthetic data generated for a simple 2D
problem.

RAND
NBE
MUNGE
ensemble selection
best single model
best neural net

0.295
0.29

3.

EXPERIMENTAL EVALUATION

We evaluate the effectiveness of model compression oneight binary classification problems. ADULT, COVTYPE
and LETTER are from the UCI Repository [1]. COVTYPE
has been converted to a binary problem by treating thelargest class as positive and the rest as negative. We converted LETTER to a binary problem in two ways. LETTER.p1 treats the letter ”O” as positive and the remaining
25 letters as negative, yielding a very unbalanced binary
problem. LETTER.p2 uses letters A-M as positives andthe rest as negatives, yielding a difficult, but well balanced,
problem. HS is the IndianPine92 data set [10] where the difficult class Soybean-mintill is the positive class. SLAC is aproblem from the Stanford Linear Accelerator. MEDIS and
MG are medical data sets. See Table 1 for characteristics ofthese problems.

3.1 Experimental Setup
We experiment with using neural networks to compress
the models built using the ensemble selection algorithm proposed by Caruana et al. in [5]. The ensemble models generated by ensemble selection are very large, complex models
that have very good generalization performance, thus they
are a perfect candidate for model compression.
2

The dictionary defines munge as “To imperfectly transform
information” or “To modify data in a way that cannot be
described succinctly”.

0.285
RMSE

probability p, ea is assigned a random value drawn from anormal distribution with mean e0a and standard deviation
sd = |ea − e0a |/s, and e0a is assigned a random value drawn
from the normal distribution with mean ea and the same
standard deviation sd. We call this approach to generating
artificial data by swapping values between neighboring cases
MUNGE.2 The method is presented in Algorithm 1.
Figure 1 shows samples generated from a simple 2D distribution (TRUE DIST), and the distributions learned by
RANDOM, NBE and MUNGE from a train set of 4000
points drawn from TRUE DIST. As expected, the samples
generated by RANDOM cover an area much larger than thetrue distribution, so only relatively few of the samples overlap with the region of interest. NBE does a better job atapproximating the true distribution, but still has problems,
especially in the “corners”. Of the three methods, MUNGE
clearly approximates the true distribution the best.

0.28
0.275
0.27
0.265
0.26
4k

10k

25k

50k
training size

100k

200k

400k

Figure 2: Average perf. over the eight problems.
Ensemble selection first builds a library of diverse baselevel modes using many different learning algorithms andparameter settings. After the library is built, the basic ensemble selection procedure builds the ensemble model bygreedily selecting at each iteration the model from the library that when added to the ensemble improves the performance of the ensemble the most. Caruana et al. also
propose a number of enhancements to the basic ensemble
selection algorithm that improve its performance, but as aside effect increase the size of the ensemble by increasing thenumber of base-level models it contains.
For all problems, a training set of 4000 points is used
to train the base-level models, and a validation set of 1000
points is used as hill climb set for ensemble selection. For
compression, the 4000 training points are used as a training
set for the three algorithms for producing artificial data:
RANDOM, NBE and MUNGE. The artificial data generated
with each algorithm is then labeled by the ensemble modeland used to train a neural net model that will mimic theensemble. When necessary, the 1000 points validation set isused for early stopping.
We compare the performance of the compressed models
with the performance of the target ensemble selection models on the eight test problems. We also show the performance of the best single base-level model from the ensemble
selection library, selected using the same 1000 points validation sets, and the best neural network that could be trainedon the original 4000 points training sets, using the 1000
points validation sets for early stopping and for selecting
the number of hidden units. All the reported results reflect
the root-mean-squared-error (RMSE) of models predictions
to the binary 0/1 targets on large independent final test sets.

ADULT

COVTYPE

0.34
MUNGE
ensemble selection
best single model
best neural net

0.335
RMSE

HS

0.4

0.33

0.32
MUNGE
ensemble selection
best single model
best neural net

0.39
0.38
0.37

0.24

0.35
0.32

0.22

0.34

0.315

0.33
1

2

4

8

16

32

64 128 256

0.2
1

2

number of hidden units

4

8

RMSE

32

64 128 256

MUNGE
ensemble selection
best single model
best neural net

MUNGE
ensemble selection
best single model
best neural net

0.4
0.35

0.25
0.2
0.15
4

8

16

32

64 128 256

1

2

number of hidden units

4

8

16

32

MUNGE
ensemble selection
best single model
best neural net

0.434
0.432

8

16

32

64 128 256

number of hidden units

64 128 256

1

2

4

8

16

32

64 128 256

number of hidden units

MUNGE
ensemble selection
best single model
best neural net

0.33
0.32
0.31
0.3
0.29

0.426

0.28

0.424

0.27

0.422
4

32

AVERAGE

0.428

0.29

16

0.34

0.43
0.295

2

8

MUNGE
ensemble selection
best single model
best neural net

SLAC

MUNGE
ensemble selection
best single model
best neural net

1

0.296
0.294
0.292
0.29
0.288
0.286
0.284
0.282
0.28
0.278
0.276

64 128 256

0.436

0.3

4

number of hidden units

number of hidden units

MG
0.305

2

MEDIS

0.3

2

1

LETTER.P2
0.45

1

RMSE

16

number of hidden units

LETTER.P1
0.18
0.17
0.16
0.15
0.14
0.13
0.12
0.11
0.1
0.09
0.08
0.07

0.28
0.26

0.36

0.325

MUNGE
ensemble selection
best single model
best neural net

0.3

0.26
1

2

4

8

16

32

64 128 256

number of hidden units

1

2

4

8

16

32

64 128 256

number of hidden units

Figure 3: Performance of compressed models vs compressed models complexity.

3.2 Results
Figure 2 shows the average RMSE performance on theeight test problems. Lower RMSE represents better performance. The top horizontal line in the figure shows theperformance of the best neural nets we could train on theoriginal 4k train set. The bottom horizontal line shows theperformance of ensemble selection trained on the same data.
Note that the models trained with ensemble selection perform considerably better than the neural net models. The
horizontal line in the middle is the average performance ofthe best single base-level models from the ensemble selection libraries, before an ensemble has been selected. Thisline represents the best performance we could achieve withany of the following learning methods: SVMs, bagged trees,
boosted trees, boosted stumps, simple decision trees, random forests, neural nets, logistic regression, k-nearest neighbor, and naive Bayes.
The other lines in Figure 2 show the performance of mimic
neural nets trained on different amounts of pseudo data labeled by the ensemble models. The lines for RANDOM,
NBE, and MUNGE correspond to neural nets with 128 hidden units trained on pseudo data generated using the threemethods described in Section 2. The graph starts at 4k
where there is no pseudo data added to the train sets. Because of this, the performance of all three methods is similar
to the performance of the best neural nets we could train on

the original train set. Performance of the mimic nets isslightly worse because they are restricted to using only 128
hidden units, which is not always optimal.
As the size of the train data increases beyond 4k, morepseudo data is being added to the train sets. At 400k, thetrain set contains 396k of pseudo data and the original 4k
train data. For mimic nets trained with pseudo data generated by RANDOM, performance improves slightly at 10k,
and at 100k and beyond performs worse than a neural net
trained on just the original 4k train set. The mimic neural nets trained on pseudo data generated by NBE performbetter, though the overall pattern is similar to the graph for
RANDOM. The peak performance of the NBE trained nets
occurs when the train set contains about 20k of pseudo data
and 4k of the original data, then degrades as more artificial
data is added to the train set.
The mimic neural nets trained on pseudo data generated
with MUNGE dominate the nets trained with RANDOM
and NBE data. Moreover, the performance with MUNGE
does not degrade as more data is added to the pseudo train
set. On average, once the pseudo training set contains 100k
or more data, the mimic neural nets perform considerably
better than the best individual models in the ensemble libraries, and nearly as well as the target ensemble itself.
This is remarkable given that the mimic neural nets are 100100,000 times smaller than the ensembles, and 100 to 10,000

Table 3: Time in seconds to classify 10k cases.

Table 2: RMSE results.
munge

ensembleannsingle

ratio

adult
covtypehs
letter.p1
letter.p2
medis
mg
slac

0.325
0.340
0.204
0.075
0.179
0.277
0.288
0.422

0.317
0.334
0.213
0.075
0.178
0.278
0.287
0.424

0.328
0.378
0.231
0.092
0.228
0.279
0.295
0.428

0.319
0.349
0.231
0.092
0.203
0.279
0.290
0.427

0.29
0.84
1.47
1.01
0.98
2.29
0.88
1.69

average

0.264

0.263

0.282

0.274

0.97

times faster to execute. It suggests that much smaller highperforming models are possible if we only knew how to train
them on the original training data. In summary, it appears
that MUNGE is the preferred method for generating pseudodata. In the experiments in the remainder of this section wewill examine results for MUNGE only.
Figure 3 presents the performance of the mimic neuralnets trained with MUNGE data as a function of the number of hidden units in the trained network. Performanceis shown for all eight problems. The graph in the bottom
right shows the average performance across all eight problems. All MUNGE neural nets are trained on pseudo datacontaining 100k samples (4k original data + 96k MUNGE
data). The graphs also show the performance of the bestneural nets trained on the original data, the performance ofthe best single models from the ensemble libraries and theperformance of the target ensemble selection models.3
Looking at the graph that averages the eight problems, theoverall trend is that performance improves until around 128
hidden units when it levels off. Looking at graphs for eachindividual problem, we see that for some of the problems asmall number of hidden units is enough for obtaining good
performance. For MEDIS and MG, performance does not
improve if we use networks with more than 2 hidden units,
and for SLAC 16 hidden units are enough. Since the training
set is large enough to prevent overfitting, performance does
not degrade as more hidden units are added.
The ADULT and COVTYPE problems are the only ones
where there is a significant difference between the performance of the MUNGE neural nets and the performance ofthe ensemble selection models. A more detailed discussion of
the results on these datasets will follow later in this section,and in Section 4.
Figures 2 and 3 show that, on average, performance ofthe mimic networks trained on the MUNGE data improves
with more hidden units and more pseudo data. As a final
experiment, we compress the ensemble selection models using 256 hidden unit nets and 400k MUNGE data for every
problem. Table 2 shows, for each of the eight problems, theperformance of mimic neural nets trained on the MUNGE
data, the target ensemble selection model, the best neuralnet trained on the original data, and the best single modelfrom the ensemble library. The performance of the mimic
neural nets is as good as or better than the performance ofthe ensemble models they are trained to mimic on six of the3

On HS, LETTER.p1 and MEDIS problems the best single
model from the ensemble library is actually a neural net so
the lines for best neural net and best single model overlap.

munge

ensembleannsingleadult
covtypehs
letter.p1
letter.p2
medis
mg
slac

7.88
4.46
12.09
2.59
2.59
4.78
6.98
3.60

8560.613440.99
1817.171630.212651.95
190.181220.04
23659.03

3.94
1.05
3.85
0.25
0.74
2.85
1.80
2.85

48.31
37.31
3.85
0.25
526.34
2.85
53.58
74.48

average

5.62

5396.27

2.17

93.37

eight problems, and always better than the performance ofneural nets trained on the original 4k data.
The values in the last column of the table indicate how
effective compression is at retaining the performance of thetarget ensemble selection models. These values are the ratio between the improvement in performance the mimic nets
provide over the best neural nets and the improvement in
performance the target ensemble selection models provide
over the best neural nets. For example, if the mimic neuralnet has performance half way between the original neuralnet and the ensemble, the ratio is 0.5. If the mimic neuralnet has performance equal to the target ensemble, the ratiois 1.0. The only problem on which the ratio is less than
0.8 is ADULT. (The results on this problem are discussed
in the next paragraph.) For a few problems the ratio isbetter than 1.0, indicating that the mimic neural net outperforms the ensemble. Note, however, that in two of thecases where the ratio is much larger than 1 (SLAC at 1.69
and MEDIS at 2.29), the range in performance is very small
so this large ratio does not actually indicate a very large
increase in performance. The ratio in the bottom row is theratio calculated for the average RMSE performances in thetable (not the average of the ratios, which would be inflatedby the two problems with artificially high ratios). On average, model compression with MUNGE is able to achieve 97%of the performance increase that could at best be expected.
The only problem for which compression is ineffective is
ADULT. On this problem the mimic net performs only alittle better than a neural net trained on the original 4k
data, and the mimic net does not perform as well as the best
single model in the ensemble selection library. Interestingly,
ADULT is the only data set that has high-arity nominal
attributes. The three attributes with the highest arity have14, 16, 41 unique values. To train a neural net on ADULT,
these attributes must first be converted to 14, 16, and 41
distinct binary attributes. The ADULT problem has only
14 attributes to begin with, yet these three attributes alone
expand to 71 sparsely coded binary inputs. It is possiblethat neural nets are not well suited to this kind of problem,
and this may prevent the mimic neural net from learning
the ensemble target function. An alternate possibility isthat the MUNGE procedure is not effective at generating
pseudo data for this kind of problem.
Table 3 shows the time in seconds required to classify
10,000 test cases for the mimic neural nets with 256 hidden
units, the target ensemble models trained by ensemble selection, the best neural nets trained on the original 4k train
set and the single best model in the ensemble library. There0.4

Table 4: Size of the models in MB.
ensembleannsingleadult
covtypehs
letter.p1
letter.p2
medis
mg
slac

0.45
0.23
0.79
0.08
0.08
0.27
0.50
0.25

1234.721108.1674.37
1.23
325.805.24
25.75
1627.08

0.22
0.03
0.12
0.01
0.04
0.14
0.03
0.13

3.95
3.41
0.12
0.01
0.07
0.14
3.25
0.30

0.38

average

0.33

550.29

0.09

1.41

RMSE

munge

0.39

RAND
NBE
MUNGE
TRUE DIST
ensemble selection
best single model
best neural net

0.37
0.36
0.35
0.34
0.33
4k

is significant variability in the speed of the best single model
because different kinds of models are best for different problems and some of the models (e.g. boosted trees) are much
more expensive than others (e.g. logistic regression). As
expected, the ensemble is extremely expensive. On average,
the ensemble takes about 0.5 seconds to classify a single
training case (on a single workstation) and, on the SLAC
problem, it takes 2.4 seconds per test case! The mimic neural nets, however, are very fast and take on average only
about 0.5 milliseconds per test case.4
Table 4 shows the size in megabytes for the different models in Table 2. A similar picture emerges as with execution
times: on average ensembles are about 500 megabytes, about
500 times larger than the best single models, and the largest
ensembles are more than a gigabyte. The mimic neural nets,
however, are four times smaller than the best single models,and more than 1000 times smaller than the ensembles.

4.

DISCUSSION

The number of hidden units needed in the compression
neural net to mimic the ensemble function provides an indication of how complex the function learned by the ensemble
actually is. For some problems, only a few hidden units
were required to learn the ensemble function with high fidelity. For other problems (e.g. COVTYPE) the target
ensemble function is more complex, and requires using 128
hidden units or more. Note that one cannot always use thesize of a neural net trained on the original training data
as a measure of function complexity because of the interaction between overfitting and network size and train set size.
With compression, however, we can make the train set size
arbitrarily large, so that overfitting is not an issue, and thus
reliably measure the effect of network size on performance.
Generating unlabeled pseudo data is not computationallyexpensive. Labeling large amounts of pseudo data, however,can be expensive if the target model is a large, complex
ensemble. Training the neural net can also be expensive,
particularly if the problem has many input features, requires
a net with many hidden units, and/or requires a large pseudotraining set. In the worst case, it can be more expensive
4
The speed of different models depends significantly on how
they are implemented. The times reported here are for typical implementations. For example, we use the SNNS neural net package [18], the IND decision tree package [4], theSVMlight SVM package [11], WEKA [16] random forests,
etc. With care, some of these numbers probably could be
improved by a factor of 10 or more, though we suspect theoverall picture would not change substantially.

10k

25k

50k

100k

200k

400k

training size

Figure 4: Perf. vs train set size for COVTYPE.
to label the pseudo data and train the mimic neural net
than it was to train the original ensemble library and build
the ensemble model. The expense of model compression isjustified only when high performing models must be used inapplications with limited storage or computational power,
in applications where predictions are needed in real time, orwhere there will be very many test cases.
The amount of pseudo data needed to train a high-fidelity
mimic model depends on the effectiveness of the method forgenerating artificial cases. It is clear from Figure 2 that
MUNGE is more effective than RANDOM and NBE, but
it would be interesting to assess the sample efficiency of
MUNGE compared to unlabeled data drawn from the true
distribution. Fortunately, more than 500,000 cases are available for the COVTYPE problem. Figure 4 shows the performance of the mimic neural nets when pseudo data is generated using RANDOM, NBE and MUNGE, as well as theperformance of the mimic nets when real unlabeled test cases
are used instead of pseudo data (TRUE DIST). Data drawn
from the true distribution appears to be about 2-8 times
as efficient as MUNGE data: MUNGE needs about 25k tomatch TRUE DIST at 10k, about 50k to match TRUE DISTat 15k, and about 200k to match TRUE DIST at 25k. Thissuggests that it may be possible to make further improvements to MUNGE, and that in domains where unlabeleddata is available model compression will work even better.
Interestingly, on the HS problem, the performance of themimic neural net trained on MUNGE pseudo data is better
than the performance of the ensemble model it is trying tomimic. See HS in Figure 3. We suspect that the ensemble
has overfit the data, and that the neural net may provide abeneficial form of smoothing/regularization.

5. RELATED WORK
Zeng and Martinez also used neural nets to approximate
ensembles of classifiers [19]. In the experiments they presented, Zeng and Martinez tryed to approximate only ensembles of ten neural nets. The mimic nets were trainedon synthetic examples generated using an algorithm similar to RANDOM. We found that, although it yields an improvement on a couple of problems, on average, generating
synthetic data using RANDOM does not provide a significant improvement over simply training the neural nets on
the original data. Generating synthetic data using NBE

or MUNGE works much better. Also, Zeng and Martinez
showed that only a small number of synthetic examples,comparable to the size of the initial train set, is sufficient
to obtain good results. This might be an indication that thefunctions that were approximated, ensembles of ten neuralnets, were not hard enough to learn by a single neural net.
In contrast, we tried to approximate much harder functions
generated by ensemble selection, and as a consequence weneeded a lot more synthetic data to obtain the best results.
TREPAN was used to extract tree-structured representations of trained neural nets [6]. The TREPAN approachalso made use of additional artificial data generated from thesame train set as the one used for training the nets. New
examples were generated as needed by randomly selecting
new values for each attribute, while ensuring that some constraints are satisfied. For discrete attributes the values weregenerated the same way as RANDOM. For continuous attributes, a kernel density estimator [15] was used. Although
TREPAN models sometimes were more compact than theneural nets they explained, and occasionally outperformed
decision trees trained directly on the original training data,
the goal in TREPAN was not to train compact models orhigh performing models.
CMM is a meta-learner that learns one single model froma bagged ensemble of the same type of models [8]. Again, theapproach makes use of additional, artificially created data
and labeled according to the bagged model. The method
used to generate synthetic data was specific to the base models used in the ensemble. The base models were decision rulesets. For each decision rule in the rule set, some examples
were generated randomly from the hyperspace classified by
the decision rule. Although CMM generates smaller models, they are not optimized for size or performance, but forcomprehensibility instead.
DECORATE [13] uses artificial data to increase diversity
so that better ensembles can be trained. Before DECORATE trains a new base learner, it generates and labels
new data the opposite way to the predictions of the current
ensemble. The new trained model is added to the current
ensemble only if the addition is beneficial to the ensemble.
This approach generates the synthetic data randomly fromthe distribution of the train set. For continuous attributes,
the new values are generated from a Gaussian distributionthat has the same mean and standard deviation as the set ofvalues for that attribute. For discrete attributes, frequency
counts are used, which is similar to RANDOM.

the true data. We show that MUNGE compares favorablywith other methods for generating artificial cases. MUNGE,
however, is not as effective as drawing unlabeled data fromthe true distribution, so using true unlabeled data is preferred when it is available.
We present experiments where complex ensemble models
are compressed using neural networks. Results on eight test
problems show that, on average, the loss in performance due
to compression is usually negligible, yet the mimic neuralnets are 1000 times smaller and 1000 times faster.

6.

[13] P. Melville and R. Mooney. Constructing diverse classifier
ensembles using artificial training examples. In Proceedings ofthe IJCAI-2003, pages 505–510, Acapulco, Mexico, 2003.

CONCLUSIONS

Some of the highest performing models currently available
are complex ensembles containing hundreds or thousand ofbase-level classifiers. These models have excellent performance, but can be so large and slow that it may be infeasible to use them when memory or computational power islimited, when predictions are needed in real time, or whentest sets are extremely large.
We present a method for model compression that is able totrain fast, compact models to mimic better performing, but
slow and complex models with little loss in performance.
Compression works by labeling a large unlabeled data set
with the target model, and then training a neural net using the newly labeled data. Where unlabeled data is notavailable, we present a new method called MUNGE for generating pseudo data from a distribution similar to that of

Acknowledgments
This work was supported by NSF Award 0412930.

7. REFERENCES[1] C. Blake and C. Merz. UCI repository of machine learning
databases, 1998.
[2] L. Breiman. Bagging predictors. Machine Learning,24(2):123–140, 1996.
[3] L. Breiman. Random forests. Machine Learning, 45(1):5–32,
2001.
[4] W. Buntine and R. Caruana. Introduction to IND andrecursive partitioning. Technical Report FIA-91-28, NASA
Ames Research Center, 10 1991.
[5] R. Caruana, A. Niculescu-Mizil, G. Crew, and A. Ksikes.
Ensemble selection from libraries of models. In Proc. 21stInternational Conference on Machine Learning, 2004.
[6] M. W. Craven and J. W. Shavlik. Extracting tree-structured
representations of trained networks. In D. S. Touretzky, M. C.
Mozer, and M. E. Hasselmo, editors, Advances in Neural
Information Processing Systems, volume 8, pages 24–30. The
MIT Press, 1996.
[7] A. P. Dempster, N. M. Laird, and D. Rubin. Maximum
likelihood from incomplete data via the em algorithm. Journalof the Royal Statistical Society, 1(39):1–38, 1977.
[8] P. Domingos. Knowledge acquisition from examples via
multiple models. In Proc. 14th International Conference on
Machine Learning, pages 98–106. Morgan Kaufmann, 1997.
[9] P. Domingos. Bayesian averaging of classifiers and theoverfitting problem. In Proc. 17th International Conf. on
Machine Learning, pages 223–230. Morgan Kaufmann, San
Francisco, CA, 2000.
[10] A. Gualtieri, S. R. Chettri, R. Cromp, and L. Johnson.
Support vector machine classifiers as applied to aviris data. In
Proc. Eighth JPL Airborne Geoscience Workshop, 1999.
[11] T. Joachims. Making large-scale SVM learning practical. In
Advances in Kernel Methods, 1999.
[12] D. Loyd and P. Domingos. Naive Bayes models for probability
estimation. In Proceedings of the 22nd International
Conference on Machine Learning (ICML’05), Bonn,
Germany, 2005.

[14] R. Schapire. The boosting approach to machine learning: An
overview. In MSRI Workshop on Nonlinear Estimation andClassification, 2001.
[15] B. W. Silverman. Density Estimation for Statistics and Data
Analysis. Chapman and Hall, 1986.
[16] I. Witten and E. Frank. Data Mining: Practical Machine
Learning Tools and Techniques with Java Implementations.
Morgan Kaufmann, 1999.
[17] D. H. Wolpert. Stacked generalization. Neural Networks,
5:241–259, 1992.
[18] A. Zell, N. Mache, R. Huebner, M. Schmalzl, T. Sommer, andT. Korb. SNNS: Stuttgart neural network simulator. Technicalreport, University of Stuttgart, Stuttgart, 1992.
[19] X. Zeng and T. R. Martinez. Using a neural network toapproximate an ensemble of classifiers. Neural Processing
Letters, 12(3):225–237, 2000.