Compression and Visualization of High-Dimensionality Data
Using Auto-Associative Neural NetworksZalhan Mohd Zin1 , Marzuki Khalid2, Ehsan Mesbahi3 and Rubiyah Yusof2
Section of Industrial Automation – UniKL-Malaysia France Institute (UniKL-MFI),
2
Center for Artificial Intelligence and Robotics (CAIRO) – Universiti Teknologi Malaysia (UTM)
3
Newcastle University, Newcastle Upon Tyne, UK
1

Abstract
Interpreting the information hidden in multidimensional data can be considered as a challenging and also acomplicated task. The compression, dimension reduction and visualization of these multidimensional data provide ways
to better understanding and interpretation of the problem. Usually, dimension reduction or compression is considered as
the first step to data analysis and exploration. Here, the focus is given on multidimensional data reduction using a
supervised artificial neural networks technique namely the Auto-Associative Neural Networks (AANN). The AANN canbe considered as very powerful tool in exploratory data analysis. It has the ability to deal with linear and nonlinear
correlation among variables. This technique is often referred to as nonlinear principal component analysis (NLPCA) or
sometimes is also known as the bottleneck neural network due to its specific structure that consists of a combination of
two networks – compression and decompression. By using this structure, AANN can reduce high dimensional data onto
lower dimensional data by compressing them on its bottleneck layer that later can be used for data visualization andinterpretation. In this paper, the technique of AANN is described, developed and applied on two different
multidimensional datasets. The results have shown that the AANN is able to compress multidimensional data into only
two nonlinear principal components at its bottleneck layer and these compressed data can provide visualization ofdifferent clusters of data.
Keywords—Auto-Associative Neural Networks, Dimension Reduction, Data Clustering, Iris flowers, Olive oils
1.

Introduction

The ability of artificial intelligence techniques toextract information hidden in high-dimensional data israther interesting to be investigated and explored. Thecompression, dimension reduction and visualization ofthese multidimensional data could provide ways tobetter understanding and interpretation of the data itself.
There are many techniques of dimension reduction thathave been used such as the famous principal component
analysis (PCA) (1) (2) (3) (4) (5). However, linear PCA
technique is limited to only linear correlated variables.
To deal with nonlinear variables problems, nonlinear
dimension reduction or projection methods should beused. Some examples of nonlinear dimension reduction
techniques are Multidimensional Scaling (MDS),Locally Linear Embedding (LLE), Isomap, Kernel PCA(6), Self-Organizing Maps (SOM) (7) (8) (9) and also
Auto-Associative Neural Networks (AANN) (10) (5)(3). AANN, is also known as Bottleneck NeuralNetworks (BNN), has been usually used for datacompression in chemical applications, missing data
estimations and image compressions (3) (11) (12). In
this research, the focus was given to AANN for
multidimensional data compression, clustering andvisualization. The technique has been described,
developed using high level computer language, appliedand analyzed by computing it on high-dimensional dataof Iris flowers and Italian olive oils datasets.

2.

Methods

The AANN has often been regarded as analternative to PCA for unsupervised learning,clustering, and outlier detection (13). As described in(5), it is also known as Non Linear Principal
Component Analysis (NLPCA) that can improve theperformance of data compression. This is because it candeal with nonlinear data problems more efficiently thanPCA. AANN is a feed forward network, which istrained to map approximations of input vectors to theircorresponding outputs. It can be viewed as circuits ofhighlyinterconnected
units
with
adjustableinterconnection weights and can be considered as aparticular class of artificial neural networks in which
the target output patterns is identical to its input patterns(14). The AANN actually works by trying to imitate
human brain abilities such as learning and memorizing.
Similarly to the biological neural system, the neuron is
the smallest functional unit in the AANN. The structureof AANN usually consists of several layers including
hidden layers such as input layer, map or compression
layer, bottleneck layer, de-map or de-compression layer
and finally output layer. This number of nodes atbottleneck layer must have smaller dimension than thenumber of nodes at input or output layers. During
training of AANN, the dataset is compressed to few
potential variables, number of which corresponds to thenumber of nodes in the bottleneck layer. Then theoutput of the bottleneck is decompressed at the demapping layer. Once the training is completed, thebottleneck outputs of AANN represent a type of
nonlinear principal components, which are frequently
more relevant than PCA for analyzing nonlinear, realworld datasets. They can be projected as visualizationof data clusters.

could also be undertaken to apply and analyze theAANN with other clustering technique such as the Self
Organizing Maps (SOM) (7) that might be useful forclustering other complicated type of data such as Gene
expression profiles.

3.

References

Results

The experiments in this paper used the AANN tocluster high-dimensional data which has been
conducted using the structure that consisted of fivelayers: two layers of input and output and three hidden
layers of map, bottleneck and de-map layers. The resultof iris flowers dataset’s (15) projection by the twonodes at the bottleneck layer is shown in Fig. 1. The
high dimensionality iris flowers data has been reducedby the AANN to only two dimensions represented bythese two bottleneck neurons. The Setosa and Virginica
flowers have been appropriately clustered far from eachother, while the class of Versicolor flower has beenclustered between these two classes, and closer toVirginica class. This distribution of iris flowers species
was consistent with the nature of the iris flowers dataset
itself (15). Meanwhile, the projection of Italian olive
oils dataset has also resulted similar and consistentbehaviour with the results obtained in (3). The data wastaken from nine different growing regions in Italy
which are North Apulia, Calabria, South Apulia, Sicily,
Inland Sardinia, Coastal Sardinia, East Liguria, WestLiguria, Umbria.

1. Smith, Lindsay I. A Tutorial on Principle Component Analysis.
[Online]2002.
[Cited:05
February2012.]
http://neurobot.bio.auth.gr/2005/a-tutorial-on-principal-componentsanalysis/.
2. A Comparison of Self Organizing Map Algorithm and Some
Conventional Statistical Methods for Ecological Community
Ordination. J. L. Giraudel, S. Lek. s.l. : Elsevier, 2001, Journal ofEcological Modelling, Vol. 146, pp. 329-339.
3. A Journey Into Low-dimensional Spaces With Autoassociative
Neural Networks. M. Daszykowski, B. Walczak, D. L. Massart.
s.l. : Elsevier Inc., 2003, International Journal of Pure and Applied
Analytical Chemistry Talanta, Vol. 59, pp. 1095-1105.
4. Missing Data Estimation Using Principle Component Analysis andAutoassociative Neural Networks. Jaisheel Mistry, Fulufhelo V.
Nelwamondo, Tshilidzi Marwala. 3, s.l. : iiisci.org, 2009, Journal ofSystemics, Cybernatics and Informatics, Vol. 7, pp. 72-79.
5. Nonlinear Principle Component Analysis Using Autoassociative
Neural Networks. Kramer, Mark A. 1991, AIChE Journal, Vol. 37,
pp. 233-243.
6. A Framework for High Dimensional Data Reduction in theMicroarray Domain. Ali Anaissi, Paul J. Kenned, Madhu Goyal.
s.l. : IEEE Explore, 2010. IEEE Fifth International Conference on
Bio-Inspired Computing: Theories and Applications (BIC-TA). pp.
903-907.
7. The Self-Organizing Maps. Kohonen, Teuvo. s.l. : IEEE Xplore,
1990. Proceedings of the IEEE. Vol. 78, pp. 1464-1480.
8. On the Use of Self Organizing Maps for Clustering andVisualization. Flexer, Arthur. Prague, Czech Republic :Springerlink, 1999. International Conference on Principle on Data
Mining and Knowledge Discovery. Vol. 5, pp. 80-88.
9. Clustering of the Self Organizing Maps. Juha Vesanto, Esa
Alhoniemi. 3, 2000, IEEE Transaction on Neural Networks, Vol. 11,
pp. 586-600.
10. The Autoassociative Neural Network-A Network WorthConsidering. Stone, Victor M. Hawaii : s.n., 2008. World
Automation Congress (WAC).

Fig. 1: The projection of iris flowers dataset defined by the two
activation nodes at bottleneck layer in AANNs.

4.

Conclusions

In this research, the compression ability of AANN
has been used to compress the multidimensional data
into few principal components which later wereprojected for visualization of the clusters of data. The
research mainly involved in the studying, developingand applying AANN algorithm using high levelcomputer language. From the experimental results, theAANN has been able to cluster iris flowers and Italianolive oils datasets according to their three and ninedifferent classes respectively. This could give us theway to explore the potential of AANN to create a modelof multidimensional data clustering. Additional research

11. Non-linear PCA: A Missing Data Approach. Matthias Scholz, mFatma Kaplan, Charles L. Guy, Joachim Kopka. 20, 2005, Journalof Bioinformatics, Vol. 21, pp. 3887-3895.
12. Image Compression Using Hybrid Neural Networks Combining
The Auto-Associative Multilayer Perceptron and The Self Organizing
Feature Map. M. A. Abidi, S. Yasuki,P. B. Crilly. 4, 1994, IEEE
Transaction on Consumer Electronics, Vol. 40, pp. 796-811.
13. Augmented Efficient BackProp for Backpropagation Learning inDeep Autoassociative Neural Networks. Mark J. Embrechts, Blake
J. Hargis, Jonathan D. Linton. Barcelona : s.n., 2010. International
Joint Conference on Neural Networks (IJCNN). pp. 1-6.
14. Feature Extraction Using Auto-Associative Neural Networks.
Gaetan Kerschen, Jean-Claude Golinval. 1, 2004, Smart Materials
and Structures, Vol. 13.
15. Fisher, R. A. UCI Machine Learning Repository. Center for
Machine Learning and Intelligent System University of California
Irvine. [Online] 2006. [Cited: 31 January 2011.] Iris Database.
http://archive.ics.uci.edu/ml/datasets/Iris.