# Immersive Interactive Information Mining with Application to Earth Observation Data Retrieval

Mohammadreza Babaee, Gerhard Rigoll, Mihai Datcu

► To cite this version:

Mohammadreza Babaee, Gerhard Rigoll, Mihai Datcu. Immersive Interactive Information Mining with Application to Earth Observation Data Retrieval. 1st Cross-Domain Conference and Workshop on Availability, Reliability, and Security in Information Systems (CD-ARES), Sep 2013, Regensburg, Germany. pp.376-386. [hal-01506787](https://hal-01506787)

HAL Id: hal-01506787

https://inria.hal.science/hal-01506787

Submitted on 12 Apr 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Immersive Interactive Information Mining with
Application to Earth Observation Data Retrieval

Mohammadreza Babaee¹, Gerhard Rigoll¹, and Mihai Datcu²

¹ Institute for Human-Machine Communication, Technische Universität München, Munich Aerospace Faculty, Munich, Germany
{reza.babaee,rigoll}@tum.de

² Munich Aerospace Faculty, German Aerospace Center, Wessling, Germany
mihai.datcu@dlr.de

**Abstract.** The exponentially increasing amount of Earth Observation (EO) data requires novel approaches for data mining and exploration. Visual analytic systems have made valuable contribution in understanding the structure of data by providing humans with visual perception of data. However, these systems have limitations in dealing with large-scale high-dimensional data. For instance, the limitation in dimension of the display screen prevents visualizing high-dimensional data points. In this paper, we propose a virtual reality based visual analytic system, so called *Immersive Information Mining*, to enable knowledge discovery from the EO archive. In this system, Dimension Reduction (DR) techniques are applied to high-dimensional data to map into a lower-dimensional space to be visualized in an immersive 3D virtual environment. In such a system, users are able to navigate within the data volume to get visual perception. Moreover, they can manipulate the data and provide feedback for other processing steps to improve the performance of data mining system.

**Keywords:** Immersive visualization, Information mining, and Dimension reduction

# 1 Introduction

The volume of multimedia data in different applications is growing intensively since the last decade. For instance, the amount of collected EO images is increasing dramatically in the order of hundreds of terabytes a day. Simultaneously, new approaches for information retrieval and knowledge discovery are in high demand. Traditional methods for exploring EO data are based on Image Information Mining whose main steps are feature extraction, data reduction, and labeling. Since all these methods are developed to perform automatically, the gap of human interaction is very large. Therefore, developing a new process chain, mainly based on involving humans’ perception of data can be a promising solution to fill in the gap. In order to process image data, first the features of the images are represented by discrete feature vectors (e.g., SIFT [1], Weber [2], Color Histogram [3], etc). In order to avoid loss of information during
discretization, the dimension of the feature vectors is set rather high. However, high-dimensional features make data understanding and knowledge discovery more challenging.

Visual analytic systems have shown a great contribution in data analysis by providing human with a visual representation of data. However, they all have limitations in the number of data points and dimensionality of feature space [4]. For example, dealing with high dimensionality is the main challenge in the visualization part of these systems due to the limitation in dimensionality of display screens.

In this paper, we propose an Immersive Information Mining framework as a novel visual analytic system to handle large-scale high-dimensional EO images. The main features of this system are: 1) reducing the dimension of high-dimensional data to three dimensions, utilizing a library of state-of-the-art dimensionality reduction (DR) techniques; 2) visualizing the data in an immersive 3D virtual environment. The proposed system allows users to play around with various DR techniques along with different parameters. They can visually compare the visualizations and choose the one that shows the structure of the given feature space better. Further, this system allows users to interact with data. More precisely, the user can change the structure of feature space by changing the position of feature points which can be used not only as a hint for feature descriptors to distinguish various classes better but also to correct available image annotations.

The rest of the paper is organized as follows. In Section 2, we discuss several current visual analytic systems. We illustrate the concept of Immersive Information Mining in Section 3. Section 4 presents samples of immersive visualization of EO images. The evaluation part of system is presented in Section 5, and finally Section 6 presents the conclusion and future works.

## 2 Related Work

In this section, we briefly review several currently available visual analytic systems. IN-SPIRE [5] is a well-known visual analytic system for document processing comprising, mainly, dimension reduction and clustering. It first extracts high-dimensional features from documents utilizing a bag-of-words model and then applies k-means clustering (with pre-defined number of clusters) on the features for data reduction. In order to visualize features, PCA reduces the dimension of features to two dimensions and then the results are plotted on screen. Another visual analytic system for document processing is Jigsaw [6] using named entities for visualization. In this system, clustering is carried out by the k-means algorithm and the results are plotted on screen. iPCA [7] also applies PCA on high-dimensional data for dimension reduction. Additionally, it visualizes both low-dimensional data along with the principal axes in high-dimensional space via parallel coordinates. Finally, Testbed [4] claims to offer an interactive visual system for dimension reduction and clustering. This system has a built-in library of dimension reduction and clustering techniques. This system aims to help the
user to understand data by visualizing the results of different dimension reduc-
tion and clustering methods. It claims to reveal valuable knowledge from data
and assist the user to choose the most appropriate processing path along with
proper parameter(s).

Since the last decade, numerous projects have utilized virtual reality for
information visualization. For instance, VRMiner is a 3D interactive tool for
visualizing multimedia data utilizing virtual reality [8]. In this system, a set of
numeric and symbolic attributes along with multimedia data (e.g., music, video,
and websites) are presented in a 3D virtual environment. But images and videos
are displayed on a second PC in order to have a real-time system. Another sam-
ple illustrating the usage of virtual reality for information visualization is an
application named 3D MARS [9]. This application is mainly for content-base
image retrieval, in which the user browses and queries images in an immersive
3D virtual environment. The main aim of this system is visualizing the results
in 3D space.

Beside aforementioned technologies for the visualization and exploration of
data, Human-Computer Interaction (HCI) has shown valuable contribution in
the domain of Data Mining and Knowledge Discovery. The main aim is providing
the user with a way to learn how to analyze the data in order to get knowledge
to make proper decisions. For example, Holzinger [10] has investigated HCI for
interactive visualization of biomedical data. As another example, Wong et al [11]
have shown first the similarity between intelligent information analysis and med-
ical diagnosis and then proposed what kind of issues should be considered during
the design of an interactive information visualization supporting intelligent in-
formation analysis.

As is clear from above, the main processing step in every visual analytic
system is dimension reduction. Since the last decade, numerous linear and non-
linear DR techniques have been proposed in different research areas. While lin-
ear approaches assume data lies in a linear d-dimensional subspace of a high-
dimensional feature space, nonlinear approaches consider data as a d-dimensional
manifold embedded in high-dimensional space. The most useful techniques in our
project are explained briefly here. Perhaps, the most famous linear algorithm is
Principal Component Analysis (PCA) which projects data into d eigenvectors
corresponding to d largest eigenvalues of the covariance matrix of the data.
Among nonlinear methods, Locally Linear Embedding (LLE) [12] aims to pre-
serve the structure of data during dimension reduction. It assumes that the data
belongs to a low-dimensional smooth and nonlinear manifold which is embed-
ded in a high-dimensional space. Then, the data points are mapped to lower-
dimensional space in such a way that neighborhood is preserved.
Laplacian Eigenmaps (LE) [13] is a nonlinear technique in the domain of spectral
decomposition methods and locally transforms data into low-dimensional space.
It performs this transformation by building a neighborhood graph from the given
data whose nodes represent data points and edges depict the proximity of neigh-
boring points. This graph approximates the low-dimensional manifold embedded
in a high-dimensional space. The eigen-functions of the Laplace Beltrami oper-
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** The workflow of immersive information mining. This comprises two parallel process pipeline. The blue one is the main processing line existing in traditional data mining techniques. The yellow one is our proposed immersive pipeline composed of dimension reduction and visualization.</figcaption></figure>

ator on the manifold serve as the embedding dimensions. Stochastic Neighbor Embedding (SNE) [14] is a probabilistic based approach attempting to preserve the neighborhoods of points based on converting the distances into probabilities. Therefor, the neighborhood relation between two data points is represented by a probability such that closer points to a specific point have larger probability than further points. Then, data points are mapped to low-dimensional space such that the computed probabilities are preserved. This is done by minimizing the sum of the Kullback-Leibler divergences of the probabilities.

# 3 Immersive Information Mining

Our proposed approach for data mining (i.e. Immersive Information Mining) suggests a new processing pipeline comprising feature extraction, dimension reduction and immersive visualization and interaction. This line runs in parallel to the main process pipeline (feature extraction and machine learning) in order to provide users with a mechanism to give feedback to other processing steps. Fig. 1 depicts the diagram of our Immersive Information Mining system.

As Fig. 1 shows, the immersive pipeline consists of two main steps, dimension reduction and visualization. The idea behind these steps is to reduce the dimensionality of the given high-dimensional feature vectors to 3D and then represent the resulting feature vectors in an immersive 3D virtual environment. In our proposed system, the dimension reduction step is a collection of linear and non-linear methods providing a meaningful and compact 3D representation of the feature vectors. Moreover, for visualization and user interaction, we use *Virtual Reality (VR)* technology (we discuss each step in more details in the following sections. See Fig. 2).

## 3.1 Dimension Reduction for Visualization

Dealing with visual data, representing images by their important features is a vital pre-processing step. Varieties of feature descriptors have been introduced
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** An immersive visualization system provides the user with a visual representation of data. Here, high-dimensional features are extracted from a database of Earth Observation images and are fed into a dimension reduction technique to be visualized in an immersive 3D virtual environment.</figcaption></figure>

during recent years to describe images from different aspects (e.g., shape, texture, color, etc). In this paper we deal with three different feature descriptors: Scale Invariant Feature Transform (SIFT) [1], Webers Local Descriptors (WLD) [15], and Color Histogram [3]. These descriptors are explored because they represent three different visual contents of images (shape, texture, and color, respectively). In order to cover all the important features of the images, the extracted feature vectors are usually high-dimensional. However, to represent the extracted feature vectors in our virtual environment, we have to reduce the dimension of features to 3D. The DR step of our system currently consists of a number of both linear and nonlinear dimension reduction techniques such as PCA [16], LDA [17], LLE [12], LE [13], SNE [14], and Non-Negative Matrix Factorization (NMF) [18, 19]. However, our system allows employing other techniques in addition to the available methods. Providing various DR techniques by the system allows users to switch between different techniques to see how they affect the structure of the feature space during the dimension reduction process.

## 3.2 Visualization and Interaction

In our visual analytic system, we utilize an immersive 3D virtual environment for visualization of feature vectors and user interaction. To provide a virtual reality environment, the so called Cave Automatic Virtual Environment (CAVE) is used. CAVE consists of four room-sized walls which are intended to play the role of four display screens. They are aligned in such a way as to form a cube-shape space. This configuration allows users to have a 180 degree horizontal view. The computer generated scene is projected onto the walls, using two projectors per wall, in order to have stereoscopic scenarios. Furthermore, a real-time tracking system comprising six infrared cameras, mounted on top of the walls, computes the position and orientation of objects (e.g., Wii controller and glasses) inside the cube. This object tracing is based on the position and orientation of a number of markers attached to every object. The computation behind this projection and tracking is done by a set of PCs in three layers. The layers collaborate in such a way as to provide users a 3D perception of the processed scene. As can be seen
in Fig. 3, the first layer is responsible for processing the signals coming from the user's motion and navigation. In our work, mainly the position and rotation of the head of the user is tracked by tracking the markers mounted on the shutter glasses worn by the user. These glasses help users to have a 3D perception of the scene generated by multiple projections in the cube. Furthermore, as the navigation tool (wand), a Wii controller in this layer, provides navigation as well as scene management signals. The middle layer, which consists of a master PC, is responsible for receiving signals and commands from the first layer. The master PC in this layer modifies the scene interpretation based on the incoming signals and then sends rendering signals to the third layer. The third layer comprises four synchronized PCs for rendering and displaying the scene on the walls. It is possible to have a strong PC for rendering instead of a cluster of 4 PCs, but such a system should be specially powerful which is much more expensive than a cluster of normal PCs. Finally, all PCs are connected to a wired LAN and are synchronized in order to have a real-time visualization. We did not use wireless LAN due to two main reasons. First, all PCs are close to each other, and second, wireless LAN has higher latency effecting the real-time property of system. The software library used for rendering the 3D environment is named 3DVIA Virtools which is used to create 3D real-time applications.

# 4 Experiments

In order to show how our visual analytic system performs for different kinds of data, we visualize three different data sets containing multi-spectral and optical EO images (UCMerced-LandUse dataset [20], optical multimedia images (Corel dataset [21]), and Synthetic Aperture Radar (SAR) images. The UCMerced-Land-Use dataset is an annotated dataset consisting of 21 classes where each class contains 100 image patches from aerial orthography. The Corel dataset comprises 1500 multimedia images categorized in 15 different groups where each group contains 100 images. Finally, the data set of SAR data is an annotated collection of 10000 SAR images. These images are categorized in 100 classes with 100 images in each. Samples of these data sets are shown in Fig. 4.

In order to process the aforementioned data sets, the images are represented by three different feature descriptors; namely, SIFT [1], WLD [15], and color-histogram [3]. These descriptors are high-dimensional feature vectors whose dimensionality is reduced in the DR step of our system to be visualized in the virtual reality space. In our experiments, we apply three different dimensionality reduction techniques to the given feature vectors to reduce their dimensionality to 3D (e.g., LE [13], SNE [14], and LLE [12]). Fig. 5–7 depict the visualization of our data sets in the CAVE.

In our visualization system, the user is allowed to navigate within the data and select some features by a provided selection tool. Zooming is completely provided and the user has the ability to view features in different views. Additionally, when it is necessary, the system can visualize the images corresponding to feature points.
<figure><img src="image_4.png" /><figcaption>**Fig. 3.** The physical diagram of immersive visualization. The visualization system is composed of three layers with different responsibilities. The first layer comprises two PCs for motion capturing (tracking) and control. A master PC in the middle layer for the synchronization, and finally four PCs for rendering for each wall of the CAVE. All PCs are connected together via an Ethernet network.</figcaption></figure>

# 5 Evaluation

Our proposed system comprises, mainly, dimension reduction and interactive visualization. Therefore, the type of evaluation for dimension reduction is different from visualization.

The used techniques for the quality assessment of the dimension reduction step include a local continuity meta-criterion (Qnx) [22, 23], trustworthiness and continuity measures (Qtc) [24], and mean relative rank error (Qnv) [25, 26]. We extracted Color-Histogram, SIFT, and Weber features from Merced and Corel data sets and then utilized Laplacian Eigenmaps (LE) [13], Stochastic Neighbor Embedding (SNE) [14], and Locally Linear Embedding [12] as dimension reduction techniques to reduce the dimensionality of extracted features. The 9 different combinations of features and dimension reductions, which are called methods here, are; 1) Color-LE, 2) Color-SNE, 3) Color-LE, 4) SIFT-LE, 5) SIFT-SNE, 6) SIFT-LLE, 7) Weber-LE, 8) Weber-SNE, and 9) Weber-LLE. The aforementioned quality measures were applied on these methods whose results are depicted in Fig. 8.

It can be concluded that the performance of features and dimension reduction depends on the type of data set. For instance, the SNE applied on extracted
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Sample images from used datasets; a) Corel; b) UCMerced-LandUse; c) Radar Images.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Three sample images of immersive visualization of UCMerced dataset</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Three sample images of immersive visualization of Corel dataset</figcaption></figure>

Weber features from Corel data set excels the SNE applied on Weber features from Corel data set.

In order to evaluate the visualization part of proposed approach, we plan to accomplish useability studies and define some objective and subjective measurement. For instance, how much time is needed to accomplish a special task? Or
<figure><img src="image_8.png" /><figcaption>Fig. 7. Three sample images of immersive visualization of SAR images</figcaption></figure>

how robust is the data manipulation? or how much time is needed for training.
As subjective measurements, data understanding and manipulation are consid-
ered. However, it is clear that visualization in the CAVE is superior to the
head-mounted-display based and monitor-based visualization since in the CAVE
the user is able to walk freely and look at the data from different angles.

<figure><img src="image_9.png" /><figcaption>Fig. 8. The quality assessment of dimension reduction techniques applied to extracted features from Merced and Corel data sets. A combination of three different features (color-histogram, SIFT, and Weber) and three different DR techniques (LE, SNE, LLE) gives 9 feature-DR methods which are: 1) color-LE, 2) color-SNE, 3) color-LE, 4) sift-LE, 5) sift-SNE, 6) sift-LLE, 7) weber-LE, 8) weber-SNE, 9) weber-LLE. Those methods whose quality measurements are closer to 1 have better performance. a) Results from Corel data set; b) Results from Merced data set.</figcaption></figure>

6 Conclusion and Future Work

In this paper, we present a virtual reality based visual analytic system, a so-called
immersive information mining system with application to earth observation im-
ages. The main features of this system are dimension reduction and immersive
visualization. Technically, we reduce the dimension of high-dimensional feature
vectors to 3D and then visualize them in an immersive 3D virtual environment.
This environment allows the user to navigate inside the data and get a visual un-
derstanding of structure of data. The feedback coming from visual analytic helps
the user to choose a proper processing path with suitable parameter(s). Another
advantage of this approach is interactivity with data. Potentially, the user could
be able to manually change the structure of data and impose constraints on the
learning process which is considered as future work.

**Acknowledgments.** This work is supported by PhD scholarship Award by
Munich Aerospace Faculty of German Aerospace Center (DLR). The authors
would like to thank reviewers for their valuable comments.

References

1. Lowe, D.G.: Object recognition from local scale-invariant features. In: Computer vision, 1999. The proceedings of the seventh IEEE international conference on. Volume 2., Ieee (1999) 1150–1157

2. Bahmanyar, R., Datcu, M.: Measuring the semantic gap based on a communication channel model. (2013)

3. van de Sande, K.E., Gevers, T., Snoek, C.G.: Evaluating color descriptors for object and scene recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on 32(9) (2010) 1582–1596

4. Choo, J., Lee, H., Liu, Z., Stasko, J., Park, H.: An interactive visual testbed system for dimension reduction and clustering of large-scale high-dimensional data. In: IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics (2013) 865402–865402

5. Wise, J.A.: The ecological approach to text visualization. Journal of the American Society for Information Science 50(13) (1999) 1224–1233

6. Stasko, J., Görg, C., Liu, Z.: Jigsaw: supporting investigative analysis through interactive visualization. Information visualization 7(2) (2008) 118–132

7. Jeong, D.H., Ziemkiewicz, C., Fisher, B., Ribarsky, W., Chang, R.: ipca: An in-
teractive system for pca-based visual analytics. Volume 28., Wiley Online Library
(2009) 767–774

8. Azzag, H., Picarougne, F., Guinot, C., Venturini, G., et al.: Vrminer: A tool for multimedia database mining with virtual reality. Processing and Managing Complex Data for Decision Support (2005) 318–339

9. Nakazato, M., Huang, T.S.: 3d mars: Immersive virtual reality for content-based image retrieval. In: IEEE International Conference on Multimedia and Expo. Volume 46. (2001)

10. Holzinger, A.: On knowledge discovery and interactive intelligent visualization of biomedical data-challenges in human-computer interaction & biomedical informatics. In: 9th International Joint Conference on e-Business and Telecommunications (ICETE 2012), pp. IS9-IS20. (2012)

11. Wong, B.W., Xu, K., Holzinger, A.: Interactive visualization for information analysis in medical diagnosis. In: Information Quality in e-Health. Springer (2011) 109–120
12. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science **290**(5500) (2000) 2323-2326

13. Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation **15**(6) (2003) 1373-1396

14. Hinton, G., Roweis, S.: Stochastic neighbor embedding. Advances in neural information processing systems **15** (2002) 833-840

15. Chen, J., Shan, S., Zhao, G., Chen, X., Gao, W., Pietikainen, M.: A robust descriptor based on weber's law. In: Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, IEEE (2008) 1-7

16. Jolliffe, I.T.: Principal component analysis. Volume 487. Springer-Verlag New York (1986)

17. Mika, S., Ratsch, G., Weston, J., Scholkopf, B., Mullers, K.: Fisher discriminant analysis with kernels. In: Neural Networks for Signal Processing IX, 1999. Proceedings of the 1999 IEEE Signal Processing Society Workshop., IEEE (1999) 41-48

18. Seung, D., Lee, L.: Algorithms for non-negative matrix factorization. Advances in neural information processing systems **13** (2001) 556-562

19. Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature **401**(6755) (1999) 788-791

20. http://vision.ucmerced.edu/datasets/landuse.html

21. http://vision.stanford.edu/resources\_links.html\#datasets

22. Chen, L., Buja, A.: Local multidimensional scaling for nonlinear dimension reduction, graph layout and proximity analysis. PhD thesis, Citeseer (2006)

23. Chen, L., Buja, A.: Local multidimensional scaling for nonlinear dimension reduction, graph drawing, and proximity analysis. Journal of the American Statistical Association **104**(485) (2009) 209-219

24. Venna, J., Kaski, S.: Local multidimensional scaling. Neural Networks **19**(6) (2006) 889-899

25. Lee, J.A., Verleysen, M.: Nonlinear dimensionality reduction. Springer (2007)

26. Lee, J.A., Verleysen, M.: Quality assessment of dimensionality reduction: Rank-based criteria. Neurocomputing **72**(7) (2009) 1431-1443