Dilpreet Singh Faculty of Information Technology Monash University Melbourne, VIC, 3145 dilpreet.singh@monash.edu Patrick Hutchings Faculty of Information Technology Monash University Melbourne, VIC, 3145 patrick.hutchings@monash.edu
Abstract
We present a mobile application that uses visual similarity search in-combination with input from the device's camera to enable real time hands-on exploration of large gallery collections. The system uses a Deep Convolutional Neural Network for compressed image representation and approximate nearest neighbour search to find and display visually similar images. We apply the system to a large historic photographic archive and discuss our findings and experience. Figure 1: The system being used to explore the Archives of the Planet collection. Mobile device's camera input is shown in the background and visually similar images from the collection are presented in the foreground.
1
Introduction
Galleries typically have more works in their collection than they can physically display, thus a large portion of their collection is rarely seen by the public in a single visit. To expose more of the collection to the public, a growing number of art institutions are making their collections available online. Currently, search and discovery of these collections often relies on searching by attributes, which is sufficient for individuals that arrive with a purpose, but has limited scope for hands-on exploration. These collections are a rich resource, and present opportunities to go beyond digital catalogues and act as a meaningful extension of the gallery experience. Recent advances in machine learning for image analysis provide interesting and engaging ways for exploring online collections–to help bring them into the environment of people all around the world. Deep Convolutional Neural Networks (DCNNs) have led to improved machine semantic analysis of images [1], and enabled creation of systems that allow people to engage with images at a deeper level 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada. [2]. The Museum of Modern Art (MoMA), Uffizi Gallery, the British Museum and other institutions have partnered with Google Arts & Culture to utilize their rich archives to tell engaging stories. The collaborations have resulted in projects such as "X Degrees of Separation" by Mario Klingemann [3] and Google's "Art Selfie" [4] which have been successful at capturing the public attention [5]. We have developed a content based search system by coupling compressed image representation and approximate nearest neighbour (ANN) search, and use it as a building block for creating an engaging method for visual discovery. We use the system to create a mobile application that captures the user's immediate environment–through the device's camera–to enable real time serendipitous exploration of a large photographic dataset by way of visual similarity.
2
System Design
Content Based Search
2.1
The search system comprises of two parts: image feature representation and ANN search. For feature representation we use the second-last layer of a pre-trained ResNet50 DCNN [6], which captures details of both content and style. We were able to achieve a better balance of performance and quality of results with ResNet50 compared to tests with feature extraction models used in other visual search systems [7] [8] . The search system uses a forest of randomized-partition binary trees [9] to perform ANN search. Once the initial search index is built, subsequent search queries are computed in under 50ms on mobile devices (iPhone X). Feature vectors from the ResNet 50 model are used to build the ANN search index with a cosine distance measure. Any image with a computed feature vector can then be used as a search input to find visually similar results from the corpus.
2.2  Mobile Application
Using our search system we developed a mobile application that captures the users immediate environment to show similar looking artworks from a collection (Figure 1). Any of the displayed artworks can be selected to see more detail, and function as a way of discovering the collection. The system runs at 15 frames/second (on a iPhone X), which makes for an inviting method of serendipitous exploration. Engaging with the environment–seeing things from different perspectives, distances, and details–engages with the collection in the same way.
3
Archives of the Planet
Archives of the Planet [10] is a large photographic archive created by Albert Kahn as a way to capture. human life on earth. The project, alongside hours of film, consists of 72,000 colour photographs, captured across more than 50 countries between the years of 1909 and 1931.  This dataset is particularly well suited for our exploration as it's a historically significant, the corpus is available for cultural and scientific use [11], and at present, the only way to engage with the archive is through simple attribute based search. Exploring the collection through our mobile application changes the nature of interaction–instead of passive browsing, it becomes real world discovery.
4
Reflections
The mobile application works as a creative and playful way of discovering content in a rich archive. Embracing the external environment adds an experiential dimension, driving the desire to try the application in different environments and point the camera outwards. The system can be adapted to work with other visual art collections and environments. Used in galleries, it would allow visitors to explore thematically similar artworks that exist at partner galleries around the world. The simplicity of the experience naturally encourages exploration among all the layers of abstraction that affect visual similarity–without directly exposing these aspects to the user.