<figure><img src="image_1.png" /></figure>

HAL
open science

# APoD eXplorer: Recommendation System and Interactive Exploration of a Dynamic Image Collection

Frédéric Rayar, Sabine Barrat, Fatma Bouali, Gilles Venturini

► To cite this version:

Frédéric Rayar, Sabine Barrat, Fatma Bouali, Gilles Venturini. APoD eXplorer: Recommendation System and Interactive Exploration of a Dynamic Image Collection. 20th International Conference on Information Visualisation, Jul 2016, Lisbonne, Portugal. hal-01315653

HAL Id: hal-01315653

https://hal.science/hal-01315653

Submitted on 7 Mar 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# APoD eXplorer:
Recommendation System and Interactive Exploration
of a Dynamic Image Collection

Frédéric Rayar*, Sabine Barrat*, Fatma Bouali*†, Gilles Venturini*

*Université François Rabelais, LI EA-6300, Tours, France
rayar@univ-tours.fr, barrat@univ-tours.fr, venturini@univ-tours.fr

†Université de Lille, IUT, Dpt STID, Lille, France
fatma.bouali@univ-lille2.fr

***Abstract***—The amount of captured images has increased exponentially these last years. Online available image collections are becoming common thanks to social networks or institutes digitization programs. The context of our work falls into the need to explore such image collections. The literature paradigms are leveraged to meet three constraints: (i) handling medium to large image collections, (ii) handling dynamic image collections and (iii) providing interactive visualisations.

In this paper, we describe how our work has been used to build a recommendation system and an interactive exploration platform for dynamic image collection. To illustrate the relevance of such tools, we present APoD eXplorer, an online available platform that enhances the exploration of the NASA Astronomy Picture of the Day image collection. The platform is available at http://frederic.rayar.free.fr/apod/.

## I. INTRODUCTION

In recent years, the cost of capturing devices such as cameras, scanners and storage media have tremendously been reduced. This has resulted in a sharp increase of the amount of captured images. These images can be generated in a private setting, a commercial framework or in the context of digitisation projects. However, the advent of the Internet has emphasised the fact that a large number of these images are uploaded online. For instance, thousands of photos are added each minute on social networks such as Flickr, Instagram or Facebook [1]. Another example is the OpenGLAM initiative [2]: many institutions such as galleries, libraries, archives and museums promote "free and open access to digital cultural heritage" by making digital scans or photos of their collections available to the public. Notable institutions are the British Library <fnref n="1" /> of London, or the Rijkmuseum of Amsterdam <fnref n="2" />.

The context of this work falls into the need to explore such image collections. Most of the available services are image retrieval systems that display images in a grid layout. However, they lack support for exploratory data mining - in which the user can wander around the images - and do not provide means for serendipity in the exploration process. A few studies have been proposed to visualise image collection [3]. Among the proposed paradigms, three stand out (see [4] and [3]).

* *Projection-based:* the idea is to project all the images

that lie in the original descriptors space in a two-dimensional (or three-dimensional) space.

* *Graph-based:* an image graph is built, in which each image is assigned to a node, and an edge between two nodes corresponds to the similarity of the related images.

* *Clustering-based:* the amount of images to display is reduced using clustering techniques; similar images are brought together to form homogeneous groups that are described by a representative.

Our endeavour work leverages the three paradigms and jointly study the indexing and the visualisation for browsing large dynamic image collections. The constraints of the work are:

* *Handling medium to large image collections:* the challenge is to handle real world data sets that can contain thousands to millions of images.

* *Handling dynamic image collections:* as images keep flooding in a continuous way, it becomes mandatory to build structure in a dynamic way: images shall be either added or edited if needed.

* *Providing interactive visualisations:* relevant visualisations - that are easy to interpret by the user - must be combined to interactions to allow the user to wander around the collection and make discoveries by following intuition.

In this paper, we describe how our work has been used to build a recommendation system (Section II) and an interactive exploration platform (Section III) for dynamic image collection. To illustrate the relevance of such tools, we describe in Section IV an online service that enhance the exploration of the NASA Astronomy Picture of the Day (APoD) [5] image collection.

## II. RECOMMENDATION SYSTEM

In order to extract information from the image collection, one needs to have a relevant representation of the collection. To this end, *proximity graphs* [6] have been used. Proximity graphs are weighted graphs with no loops. They aim at extracting the structure of a data point set, in which each data

<footnote n="1">¹http://www.bl.uk/</footnote>

<footnote n="2">²https://www.rijksmuseum.nl/en</footnote>
point is represented by a node. They associate an edge between
two data points if they are close enough to be considered
as neighbours. Notable proximity graphs include k-nearest
neighbour graph (k-NNG), relative neighbourhood graph [7]
(RNG), Gabriel graph [8] (GG) and Delaunay graph [9] (DG).

We have decided to use the RNG because it allowed us
to handle high dimensional data while assuring to have a
connected graph. In our previous work [10], we have managed
to propose an incremental algorithm to build an approximate
RNG. The algorithm allows to incrementally insert new data in
an existing RNG. Its scalability has been tested up to 1,000,000
images. One may refer to the paper [10] for further details on
the algorithm.

Extending this work, we propose to leverage the proposed
image graph to build a simple proximity graph-based recom-
mendation system. The key idea is to provide to the user
recommendations that correspond to similar images to the
current one. Several schemes could be considered:

* Build a graph, for a given image descriptor (e.g. color), and consider the direct neighbours of the current image as recommendations.

* Build a graph, for a given image descriptor, and consider all nodes that are within a given *graph geodesic distance* *d* as recommendations.

* Build a graph, for a given image descriptor, and consider a fixed number *k* nodes that are within a given *graph geodesic* distance as recommendations.

* Build several graph, for a set of image descriptors (e.g. color, texture, shape), and consider one of the three strategies given above to create the recommendations.

In the use case presented in Section IV, we have imple-
mented the first scheme described above.

III. INTERACTIVE EXPLORATION

A. *Hierarchical and graph-based hybrid structure*

To allow one to interactively explore an image graph,
displaying thousands of images is not a relevant solution.
Therefore, the clustering-based approach - that reduce the
number of images to be displayed - appears mandatory. Sev-
eral works in the *Graph Drawing* research community, have
proposed to build a graph and then aggregate nodes to be able
to display it - sometimes, in a multilevel way. However, the
coarsening phase relies mainly on the graph topology - which
fits for social networks - at the expense of the data distribution
in the descriptors space.

We have proposed a strategy that leverages a partitioning
clustering - namely the BIRCH [11] algorithm - that yields a
hierarchical tree structure. The tree structure is enhanced by,
among others:

* Assigning a set of relevant image representatives to each internal nodes, in order to allow the user to easily interpret them and to be assisted in the exploration.

* Structuring the tree elements at different levels, to give the user an idea of the distribution of the tree elements; to this end, the RNG has again been used. Figure 1

illustrates the proposed hierarchical and graph-based
hybrid structure.

The proposed algorithm build the structure incrementally,
and its scalability has been tested up to 1,000,000 images<fnref n="3" />. For
more details on this structure, one can refer to the paper [12].
Note that the proposed structure could also be leveraged to
extract recommendations of a given image of interest.

<figure><img src="image_3.png" /><figcaption>Fig. 1: Proposed hybrid hierarchical and graph-based structure. For each internal node, its children are organised using a RNG. Images are assigned to each level to guide the user in the tree exploration.</figcaption></figure>

B. *Graph drawing algorithm*

To draw the graphs that are embedded in the structure,
several algorithms of the state-of-the-art could be used [13].
Most common used are *force-directed* algorithms. They assign
forces to nodes and edges in order to position the nodes in an
aesthetically pleasing way, while minimising the edge crossing.
However, we have selected a different family of algorithms that
exploit *projection* concepts. Indeed, these last allow a better
taking into account of the distance between node (or the edges
weights). In the system described in Section IV, a slightly
modified version of the *stress majorization* [14] algorithm has
been used to layout the graphs.

C. *Visualisation*

The hybrid structure is then visualised using a multilevel
strategy in a custom platform that has been developed. It has
been realised using web technologies - namely HTML5, CSS3
and Javascript.

This choice is explained as follows: first, nowadays, a
majority of users that are not experts in computer science can
still manage well web navigation. Thus, such users are familiar
with web browsers. We think that presenting the system as a
light web platform, would make users more disposed to exploit
it. Second, the choice of discarding a server can be justified
by two arguments: (i) the upload of the pictures - operation

<footnote n="3">³It will soon be tested with a collection of more than 14,000,000 images</footnote>
that may cost time - to a server is not mandatory, and (ii) as the images are not sent nor stored in an external server, we respect the potential confidentiality or license issues that are related to the images.

Common interactions have been implemented such as zoom, pan, node selection. More details on available interactions will be given in Section IV, throughout the description of APoD eXplorer.

## IV. APOD EXPLORER

### A. APoD image collection

In this experiment, the NASA Astronomy Picture of the Day data set has been used. It is a nice example of growing image collection. Every day, a new astronomical picture - sometimes a video - is put online, along with a brief description written by a professional astronomer. The project started in 1995 - almost 7600 entries up-to-date - and is still active - more than 14,000 likes and followers on Facebook and Google+ fan pages, respectively. Note that this use case deals with copyrighted images:

> "All the images on the APOD page are credited to the owner or institution where they originated. Neither NASA nor APOD can grant permission to use copyrighted images."

In the main page of the project website [5], one can see the new image and its information. A link to the archive and a text-based search engine are also available. We think that a richer experience could be brought to users, by using our indexing and visualisation interface. Indeed, as soon as a new image is posted, it could be retrieved, analysed and then inserted in the proposed structures. The system can then propose similar images as recommendations to users and allow an interactive exploration of the whole collection. Thus, one could wander around the collection instead of just seeing the image of the day. This may lead to nostalgia and even serendipity.

### B. Offline processing

To yield the presented results, images and their description have been retrieved from the APOD website using NASA Open API <fnref n="4" />. 6,871 images have been crawled, from January 1996 until a cut-off date (October 2015). Sometimes, a video is proposed instead of an image. We did not retrieve nor process the videos in this work. A few images in GIF format have also not been considered.

Color descriptor, namely Color Layout Descriptor (CLD) [15], have been extracted for each image, and used to yield our structures. The information extracted have also been stored by year to be leveraged in timelines.

### C. Toward an online scenario

Thanks to the incremental property of our algorithm, our system can be deployed on a server, and daily take into account the new image posted on the APOD website. Figure 2 illustrates a feasible workflow.

<figure><img src="image_4.png" /><figcaption>Fig. 2: APOD eXplorer: Online scenario workflow.</figcaption></figure>

A deamon could run on the APoD eXplorer server. It will get the day information using the NASA API. If it is a video, we could directly update the main page of our system, and propose recent videos as recommendations. If it is an image, the deamon will download it and eventually convert it to be handle by our descriptor extraction application. As soon as, the descriptors is computed, we can discard the image from the server to avoid any copyright related issues.

The new image is now described by the CLD and some meta data given by the API (e.g. title, url, etc.). It is then inserted in the structures and related files are updated. Finally the main page of our platform is updated to present today astronomy picture; its information and its recommendations.

<footnote n="4">⁴https://api.nasa.gov/api.html</footnote>
One can also imagine a scenario where users will be notified
of this update with a mail or a RSS stream.

The online scenario presented above is not currently de-
ployed, but rather the offline processing result is.

D. Platform description

Figures 3, 4 and 5 present the view of the three main views
of APoD eXplorer. APOD eXplorer is available to test online
at http://frederic.rayar.free.fr/apod/. Any feedbacks from the
reviewers are welcome. Note that only the offline scenario
described in the previous section is available for the time being.

In Figure 3, the current day image is displayed, along with
its information. In the right panel, recommendations are pro-
posed, and the user can navigate and wander in the collection
by selecting successively proposed recommendations.

In Figure 4, the user can explore the archive using a well known visualisation: timelines. Indeed, as the images have time related information, it is relevant to propose a visualisation which most users are familiar with. In order to avoid having more than 6,000 elements to display in the timeline, we have segmented the images by year. Timeline JS [16] - an open source JavaScript library - has been used to implement the timelines.

In Figure 5, the interactive explore visualisation described
in Section III is available. It consists of the graph visualisation
canvas (left) and controls/information panel (right). Linkurious
JS [17] - another open source JavaScript library - has been
used to manage the graph visualisation. The first level of the
proposed hybrid structure is displayed in the main frame. Some
of the available interactions are given below.

a) *Graph manipulation:* Common interactions such as zoom, pan and graph dragging are available to allow the user to manipulate the graph and select a region of interest in the graph. Furthermore, if the user do not want to have a visual notification of the adjacencies and their information, he can hide the edges, that will not be rendered. A graph drawing algorithm is also runnable to layout the graphs.

b) *Image visualisation:* One important feature of the platform is the possibility to display the images that are represented by a node. Thus, the user can either (i) continue to see the node as circles and visualise the related images as thumbnails by hovering the cursor over nodes or (ii) display each node as images and select the size of the thumbnails thanks to a slider.

c) *Focusing on an image:* The user can focus his attention on a single image by clicking on it. A zoom is done by placing the selected image on the center of the visualisation canvas. Its adjacencies are highlighted and related information are given in the right panel. A larger thumbnail is displayed and can be clicked to display the image in its original size in a separate window. Displayed information can be either meta data, description or annotations. Furthermore, the graph neighbours of the image are displayed to allow the user to quickly visualise the neighbourhood of the selected image.

d) *Exploring a graph:* In order to explore a graph,
several means are proposed to the user. First, one can navigate
in the graph canvas by clicking successively on the nodes

he is interested in. Second, one can navigate using the graph
adjacencies. Once an image is selected, its graph neighbours
are displayed in the information panel and are clickable.
By clicking one of the thumbnail, the graph focus on the
canvas changes to the selected thumbnail node. Thus, one can
perform a step by step exploration of the image collections by
leveraging the graph adjacencies. Finally, the navigation in the
image graph is stored and a log is available to allow one to
rapidly go back to any of its previously visited images.

e) *Exploring the hierarchical structure:* If the user decide to display nodes images, internal nodes are represented by their first representative. Regarding the navigation, when the user fly over an internal node, its first representative is shown as a thumbnail in the bottom-right of the graph canvas and seven of its representatives are displayed in the information panel. Seven of the farthest representatives - that can be considered as outliers - are also displayed in the information panel. Thus, the user is assisted in his navigation of the hierarchical structure. Clicking on an internal node will display its subtree. The number of images contained in the subtree is displayed in the information panel. Roll-up is also possible to explore different branch of the tree.

V. CONCLUSION AND FUTURE WORK

In this paper, we have described how our joint study of
indexing and visualisation has been used to build a recom-
mendation system and an interactive exploration platform for
dynamic image collection. We present APoD eXplorer, an on-
line accessible platform that allows to enhance the exploration
of the NASA Astronomy Picture of the Day image collection.

Two directions are the subject of future work. First, we want to finish implementing a few functionalities and interactions under development. Also, the whole image collection will be accessible and the online scenario will be deployed. Second, we aim at performing a complete user evaluation on the proposed interactive exploration using the hybrid structure. The first feedbacks are very promising but the effectiveness of such an interaction visualisation has to be showed using either a task-based evaluation or the study of other real world use cases.

REFERENCES

[1] Domo. (2015) Data never sleeps 3.0. https://www.domo.com/learn/data-never-sleeps-3-0.

[2] DM2E. (2015) Openglam. http://openglam.org/.

[3] W. Plant and G. Schaefer, "Visualisation and browsing of image databases," in *Multimedia Analysis, Processing and Communications*, ser. Studies in Computational Intelligence, 2011, vol. 346, pp. 3–57.

[4] J. Camargo and F. González, "Visualization, summarization and exploration of large collections of images: State of the art," *Latin-American Conference On Networked and Electronic Media*, 2009.

[5] R. Nemiroff and J. Bonnell. (1995) Astronomy picture of the day. http://apod.nasa.gov/.

[6] G. T. Toussaint, "Some unsolved problems on proximity graphs," 1991.

[7] —, "The relative neighbourhood graph of a finite planar set," *Pattern Recognition*, vol. 12, pp. 261–268, 1980.

[8] R. K. Gabriel and R. R. Sokal, "A New Statistical Approach to Geographic Variation Analysis," *Systematic Zoology*, vol. 18, no. 3, pp. 259–278, Sep. 1969.

[9] B. N. Delaunay, "Sur la sphère vide," *Bulletin of Academy of Sciences of the USSR*, vol. 7, pp. 793–800, 1934.
<figure><img src="image_5.png" /><figcaption>Fig. 3: APOD eXplorer: Recommendations visualisation. On the left, information about the selected day are displayed. On the right, recommendations related to the selected day are proposed to the user.</figcaption></figure>

[10] F. Rayar, S. Barrat, F. Bouali, and G. Venturini, "An approximate proximity graph incremental construction for large image collections indexing," in *Proceedings of Foundations of Intelligent System 22nd International Symposium, ISMIS 2015*, 2015.

[11] T. Zhang, R. Ramakrishnan, and M. Livny, "Birch: An efficient data clustering method for very large databases," in *Proceedings of the 1996 ACM SIGMOD International Conference on Management of Data*, ser. SIGMOD '96. New York, NY, USA: ACM, 1996, pp. 103–114.

[12] F. Rayar, S. Barrat, F. Bouali, and G. Venturini, "Incremental hierarchical indexing and visualisation of large image collections," in *24th European Symposium on Artificial Neural Networks, ESANN 2016*, Bruges, Belgium, April 27-29, 2016, 2016.

[13] G. D. Battista, P. Eades, R. Tamassia, and I. G. Tollis, *Graph Drawing: Algorithms for the Visualization of Graphs*. Prentice Hall PTR, 1998.

[14] E. R. Gansner, Y. Koren, and S. North, *Graph Drawing: 12th International Symposium, GD 2004, New York, NY, USA, September 29-October 2, 2004, Revised Selected Papers*. Springer Berlin Heidelberg, 2005, ch. Graph Drawing by Stress Majorization, pp. 239–250.

[15] B. S. Manjunath, J. R. Ohm, V. V. Vasudevan, and A. Yamada, "Color and texture descriptors," *IEEE Transactions on Circuits and Systems for Video Technology*, vol. 11, no. 6, pp. 703–715, Jun 2001.

[16] N. U. K. Lab, "Timeline js," https:// timeline.knightlab.com/, 2015.

[17] L. SAS, "Linkurious js," http://linkurio.us/, 2013.
<figure><img src="image_6.png" /><figcaption>Fig. 4: APOD eXplorer: Timeline visualisation with Timeline JS. The user can visualise one timeline per year.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 5: APOD eXplorer: Interactive exploration visualisation. On the left, the visualisation of the proposed hybrid hierarchical and graph-based structure. On the right, selected node information are displayed, along with general controls.</figcaption></figure>