Classification of Ford Motor Data
Joerg D. Wichard
Abstract— In this work we apply mixed ensemble models inorder to build a classifier for the Ford Classification Challenge.
We build feature vectors from the data sequences in terms offirst order statistics, spectral density and autocorrelation. Ourmodel selection scheme is a mixture of cross-validation andbagging. The outcome is an ensemble model, that consits ofseveral different models trained on random subsamples of theentire data set.

I. I NTRODUCTION
The Ford Classification Challenge [1] is part of the WCCI2008 Competition Program [2] and was motivated by anautomotive application. The given task is the classification offinite data sequences, which includes also data preprocessingand generation and selection of feature vectors. Our method
consists of three steps:• Data preprocessing• Generating and selecting feature vectors• Building ensemble based classification models
II. DATA P REPROCESSING
The Ford Classification Challenge [1] consists of two data
sets that include data samples from an automotive subsystem.
The data was collected in batches of N = 500 samples per
diagnostic session, splitted in a training, a validation and atest set. The size of training, validation and test set is listed
in Table I. The training set provided also the classification
NameFord A
Ford B

Training
3271
3306

Validation
330
330

Test1320
810

TABLE IT HE SIZE OF TRAINING , VALIDATION AND TEST SET FOR THE F ORD
C LASSIFICATION C HALLENGE .

labels, wherein +1 indicates that a specific symptom exists
and -1 indicates that the symptom does not exist. Later during
the competition, the validation labels were also published on
the competition website [1].
In order to generate proper feature vectors for the classification task we had to scale the data. Let ~x = {xi }i=1,...,Ndenote a sampled time series from a diagnostic session. We
assume, that the ~x sequence represents a set of uniformlyspaced time-samples of some measured signal x(t), where t
represents time.
We removed the mean from the each series and scaled it:~x − x̂~xs =,(1)sx
Joerg D. Wichard is with the FMP Berlin, Molecular Modelling Group,
Robert-Roessle-Str. 10, 13125 Berlin, Germany, (email: wichard@fmpberlin.de)

wherein sx is given by the sum of the absolute values
sx =NXkxi k(2)

xi .

(3)

i=1

and x̂ is the mean over timex̂ =NX
i=1

We further define the σ(~x) as the variance of ~x.
III. F EATURE S ELECTION
Let spec(~xs ) denote the the spectral density and acf (~xs )denote the autocorrelation of the scaled time series (see [3]for definitions). We build preliminary feature vectors ~y fromthe time series in the following way~y = (spec(~xs ){1,...,40} , acf (~xs ){1,...,60} , x̂, sx , σ(~x)), (4)wherein we included the first 60 values of the autocorrelationand the first 40 values of the spectral density.
We used a feature selection approach to decide which parts ofthe spectral density and the autocorrelation should be taken
into the final feature vectors.
Our feature selection approach follows in principle themethod of variable importance as proposed by Breiman[4]. The underlying idea is to select descriptors on thebasis of the decrease of classification accuracy after thepermutation of these descriptors. Briefly, an ensemble ofclassification models is built, which uses all descriptors as
input variables and the classification accuracy on an holdout data set is calculated. In a second step, the same isdone after the successive permutation of each descriptor. The
relative decrease of classification accuracy of the permutateddescriptor compared to the unchanged case is the variable
importance following the idea that the most discriminativedescriptors are the most important ones ((see Breiman [5]).
This is done several times and the mean variable importance
together with variance is calculated.
The spectral density for higher frequencies is almost zero, so
only the low frequency part is useful for classification and for
calculating the variable importance so we decided to use only
the first 40 values for the preliminary feature vectors. The
autocorrelation is not vanishing for higher values of the timelag, so there is no natural cutoff. We had to make a choice
and we decided to use the first 60 values into the preliminaryfeature vectors. In Figure 1 the variable importance is shownfor the preliminary feature vectors on both data sets (Ford Aand Ford B). It turned out, that in both cases the first valuesof the autocorrelation are the most discriminative descriptors.
We decided to use only the most discriminative descriptorsFig. 1. The variable importance for the preliminary feature vectors on on
both data sets (Ford A at the top, Ford B at the bottom). The first 40 values
are the low frequency part of the spectral density, the next 60 values are thefirst values of the autocorrelation. The last three values are x̂, sx , σ(~x) as
defined in Section II.

that had a mean variable importance above the variance. Only
31 descriptors were left for Ford A and 30 Ford B. In Figure
2 we plotted the mean autocorrelation function of the twotraining data sets separately for the two classes ( +1 and -1) with the variance as error bars. It is obvious, that the firsttwo zero-crossings and the values around the first minimum
are the most discriminative descriptors, as pointed out by thefeature selection method descried above.
IV. C LASSIFIER E NSEMBLES
The average output of several different models fi (x) iscalled an ensemble modelfˆ(x) =KX

ωi fi (x),(5)i=1wherein we define that the model weights ωi sum to one
PKi=1 ωi = 1. There are several ways to define the model
weights (see Perrone et al. [6] or Hashem et al. [7]), but we
decided to use uniform weights with ωi = 1/K for the sakeof simplicity and not to run into over-fitting problems.
The central feature of the ensemble approach is the generalization ability of the resulting model. In the case of regression

Fig. 2. The mean autocorrelation for both data sets (Ford A at the top, Ford
B at the bottom). The positive labeled data is plotted in blue, the negative
labeled data in red, together with the variance of the data shown as error
bars.

models (with continuous output values) it was shown, thatthe generalization error of the ensemble is in the averagecase lower than the mean of the generalization error of thesingle ensemble members (see Krogh 1995 [8]).
A. Model Selection
Our model selection scheme is a mixture of bagging[9] and cross-validation. Bagging or Bootstrap aggregating
was proposed by Breiman [9] in order to improve theclassification performance by combining classifiers trainedon randomly generated subsets of the entire training sets. We
extend this approach by applying a cross-validation scheme
for model selection on each subset and after that we combine
the selected models to an ensemble. In contrast to classicalcross-validation, we use random subsets as cross-validation
folds.
In K-fold cross-validation, the data set is partitioned into K
subsets. Of these K subsets, a single subset is retained as thevalidation data for testing the model, and the remaining K- 1 subsets are used for model training. The cross-validation
process is then repeated K times with each of the K subsets
used only once as the validation data. The K results from thefolds then can be averaged to produce a single estimation.

real class +1
real class -1

predicted class +1
true positive (tp)false positive (fp)predicted class -1
false negative (fn)true negative (tn)

TABLE II
T HE CONFUSION MATRIX FOR A BINARY CLASSIFICATION PROBLEM .

added to the final ensemble, receiving the weight ωi = k1 .
All other models in this CV-fold are deleted.
B. Error Measures
We used the accuracy as defined in Eq. 6 in order totrain and to compare the different classification models.
Therefore we have to define the four possible outcomes ofa classification that can be formulated in a 2 × 2 confusion
matrix, as shown in Table II. The accuracy is defined as theratio of the correct classified samples:
Accuracy =Fig. 3. For every partition of the cross-validation, the data is divided ina training and a test set. The performance of each ensemble model wasassessed on validation set which was initially removed and never included
in model training.

tp + tn.
tp + tn + f p + f n

Another error measure is the false positive rate (FP-Rate),
which is defined as the proportion of negative instances thatwere erroneously reported as being positive:
FP-Rate =If we lack relevant problem-specific knowledge, crossvalidation methods could be used to select a classification
method empirically [10]. This is a common approach because
it seems to be obvious that no classification method is uniformly superior, see for example Quinlan [11] for a detailed
study. It is also a common approach to select the model
parameters with cross-validation [12]. The idea to combine
the models from the K cross-validation folds (stacking) was
described by Wolpert [13].
We suggest to train several models on each CV-fold, toselect the best performing model on the validation set and tocombine the selected models from the K-folds. If we trainmodels of one type but with different initial conditions (forexample Neural Networks with different numbers of hidden
neurons) then we find proper values for the free parameters ofthe model. We could extend that be combining models fromdifferent classes in order to increase the model diversity. We
call this a heterogeneous ensemble or mixed ensemble andapplied this method effectively to several problems [14]–[17].
Our model selection scheme works as follows: For the Kfold CV the data is divided K-times into a training set anda test set, both sets containing randomly drawn subsets ofthe data without replications. The size of each test set was
50% of the entire data set. In every CV-fold we train several
different models with a variety of model parameters (seeSection IV-C for an overview of the models). In each foldwe select only one model to become a member of the finalensemble (namely the best model with respect to the test set).
This means, that all models have to compete with each otherin a fair tournament because they are trained and validated on
the same data set. The models with the lowest classification
error (the highest accuracy) in each CV-fold are taken out and(6)fp.
tn + f pC. Classification Models
In this section we give a short list of the model classes
that we used for ensemble building. All models belong to
the standard collection of machine learning algorithms forclassification and regression tasks so details can be foundin the textbooks like for instance Hastie et al. [18]. The
implementation of these models in an open source MATLAB
toolbox is available on-line [19] and allows the integration ofuser defined classification algorithms. A detailed descriptionof our model classes was given recently [20]. The models
that we use in our model selection scheme were:• Linear Discriminant Analysis (LDA)• Penalized Discriminant Analysis (PDA)• Multi Layer Perceptron (MLP) with one or two hiddenlayers and randomly drawn number of neurons• Support Vector Machines with RBF-kernels• Classification and regression trees (CART)
V. R ESULTS
We applied our ensemble building approach to build aclassification model for the two data sets from the Ford
Classification Challenge [1]. The initial training set wasused to train the model as described above. During thedevelopment phase (before the validation set labels wererevealed) we tried different feature vectors and made ourchoice as described in Eq. 4. We further decided to use 101
cross-validation folds1 in order to train the ensemble for thefinal submission, where the ratio of training/test samples
1 A smaller number of cross-validation folds is possible, but a larger
number cannot degrade the result.

Name
Ford A (CV)Ford B (CV)Ford B (Orig.)

Accuracy
0.953 ± 0.002
0.946 ± 0.004
0.794

FP-Rate0.041 ± 0.001
0.047 ± 0.001
0.172

TABLE IIIT HE VALIDATION R ESULTS FROM A 10- FOLD CROSS - VALIDATION RUN
FOR THE TWO DATA SETS . W E FURTHER SHOWED THE RESULT OF THE
ORIGINAL VALIDATION SET FOR THE F ORD B DATA .

was 50/50 (see Section IV-A). After the disclosure of thevalidation labels, the validation set was combined with thetraining set in order to enlarge the set of labeled training
samples for the final training.
From this enlarged training set, we generated small labeledvalidation sets of 20% size, that were use for a 10-foldcross-validation to estimate the expected classification
error of our final model. It is worth to mention, that thecross-validation for the Ford B data set leads to relative
good results while the accuracy for the original validation
set is below average. The results of the cross-validation runs
are reported in Table III.
Following the description on the challenge website [1]the data samples of hidden classification were collectedunder noisy conditions, while the data samples of knownclassification were collected in typical operating conditions.
So training and operation conditions are different, whichleads to a significant reduction of classification accuracy.
In a final step, the validation set was combined with thetraining set and the model was trained for the submission.
The ensembles for the final submission were build with
101 cross-validation folds, so each ensemble consits of101 different models. For the Ford A data set the resulting
ensemble consists of 41 SVMs with RBF-Kernels and 60
MLPs and for the Ford B data set we have 61 SVMs withRBF-Kernels and 40 MLPs.
VI. C ONCLUSIONS
We successfully generated descriptors to describe the times
series and applied a feature selection scheme in order toshrink the feature vectors. Our model selection approachselected and combined mostly MLPs and SVMs and theresulting ensemble models performed quite well on a 10-foldcross validation.

ACKNOWLEDGMENT
The authors would like to thank the members of the
Molecular Modelling Group at FMP Berlin.
R EFERENCES[1] M. Abou-Nasr and L. Feldkamp, “Ford Classification Challenge,”
http://home.comcast.net/˜nn-classification.
[2] “IEEE World Congress on Computational Intelligence (WCCI),”
http://www.wcci2008.org/.
[3] H. Kantz and T. Schreiber, Nonlinear time series analysis. Cambridge
UK: Cambridge University Press, 1997.
[4] L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp.
5–32, 2001.
[5] ——, “Arcing classifiers,” The Annals of Statistics, vol. 26, no. 3, pp.
801–849, 1998.
[6] M. P. Perrone and L. N. Cooper, “When Networks Disagree: Ensemble
Methods for Hybrid Neural Networks,” in Neural Networks for Speechand Image Processing, R. J. Mammone, Ed. Chapman-Hall, 1993,pp. 126–142.
[7] T. Hashem and B. Schmeiser, “Improving model accuracy using
optimal linear combinations of trained neural networks,” IEEE Trans.
Neural Networks, vol. 6, no. 3, pp. 792–794, 1995.
[8] A. Krogh and J. Vedelsby, “Neural network ensembles, cross validation, and active learning,” in Advances in Neural Information
Processing Systems, G. Tesauro, D. Touretzky, and T. Leen, Eds.,
vol. 7. The MIT Press, 1995, pp. 231–238.
[9] L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, no. 2,
pp. 123–140, 1996.
[10] C. Schaffer, “Selecting a classification method by cross-validation,” inFourth Intl. Workshop on Artificial Intelligence & Statistics, January
1993, pp. 15–25.
[11] J. Quinlan, Comparing connectionist and symbolic learning methods.
MIT Press, 1994, vol. I, pp. 445–456.
[12] I. Guyon and A. Elisseeff, “An introduction to variable and featureselection,” Journal of Machine Learning Research, vol. 3, pp. 1157–1182, 2003.
[13] D. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, pp.
241–259, 1992.
[14] J. Wichard, M. Ogorzałek, and C. Merkwirth, “Detecting correlationin stock markets,” Physica A, vol. 344, pp. 308–311, 2004.
[15] A. Rothfuss, T. Steger-Hartmann, N. Heinrich, and J. Wichard,“Computational prediction of the chromosome-damaging potential ofchemicals,” Chemical Research in Toxicology, vol. 19, no. 10, pp.
1313–1319, 2006.
[16] J. Wichard, “Model selection in an ensemble framework,” in Proceedings of the IEEE World Congress on Computational Intelligence,
Vancouver, Canada, 2006, pp. 2187 – 2192.
[17] J. Wichard and M. Ogorzałek, “Time series prediction with ensemble
models applied to the CATS benchmark,” Neurocomputing, vol. 70,
no. 13-15, pp. 2371–2378, 2006.
[18] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of StatisticalLearning, ser. Springer Series in Statistics. Springer-Verlag, 2001.
[19] J. Wichard and C. Merkwirth, “ENTOOL - A Matlab toolbox forensemble modeling,” http://www.j-wichard.de/entool/, 2007. [Online].
Available: http://www.j-wichard.de/entool/[20] J. Wichard, H. Cammann, C. Stephan, and T. Tolxdorff, “Classificationmodels for early detection of prostate cancer,” Biomedicine andBiotechnology, Special Issue of the FBIT 2007, January 2008.