Running head: CROSS-LANGUAGE IMAGE RETRIEVAL User Experiments with the Eurovision Cross-Language Image Retrieval System

In this paper we present Eurovision, a text-based system for cross-language (CL) image retrieval. The system is evaluated by multilingual users for two search tasks with the system configured in English and five other languages. To our knowledge this is the first published set of user experiments for CL image retrieval. We show that: (1) it is possible to create a usable multilingual search engine using little knowledge of any language other than English, (2) categorizing images assists the user's search, and (3) there are differences in the way users search between the proposed search tasks. Based on the two search tasks and user feedback, we describe important aspects of any CL image retrieval system. 1. User Experiments with the Eurovision Cross-Language Image Retrieval System A great deal of research is currently being conducted in the field of Cross Language Information Retrieval (CLIR), where documents written in one language (referred to as the target language), are retrieved by a query written in another language (referred to as the source language). Most work up to now has concentrated on locating or creating translation resources and methods that automatically transform a user’s query into the language of the documents. Additional methods to match queries and documents include translating the document collection into the source language and converting both the queries and documents into a common language (Oard, 1997). Query translation is the dominating approach because can be made to work successfully with simple translation methods, e.g. dictionary-lookup (Ballesteros & Croft, 1998), and does not require the overhead of translating collection documents which is often computationally expensive (e.g. Oard (1998) spent ten machine-months translating the TREC SDA/NZZ German collection – 251,840 newswire articles). With the right approach, CLIR systems are able to achieve retrieval effectiveness that is only marginally degraded from the effectiveness achieved had the query been manually translated (referred to as monolingual retrieval). For example, Ballesteros & Croft (1998) achieved CLIR effectiveness at 90% of monolingual retrieval by using query expansion before and after query translation. Other CLIR work has concentrated on means of presenting the retrieved documents in some surrogate form such that users can judge relevance without having access to a full translation (Resnik, 1997; Gonzalo & Oard, 2003). One area of CLIR research that has received less attention is retrieval from collections where text is only used to describe the collection objects, and the object’s relevance to a query are hopefully clear to anyone regardless of their foreign language skills. One such collection is a picture archive where each image is accompanied by some kind of text, e.g. metadata or captions, semantically related to the image. Images can then be retrieved using standard IR methods based on textual queries. Many image collections exist where textual captions accompany individual or groups of images such as historic or stock-photographic archives, medical case notes and art and history collections. Here CLIR offers the opportunity of broadly expanding the range of potential searchers to an image archive through multilingual access. As Oard (1997) comments: “an image search engine based on cross-language free text retrieval would be an excellent early application for cross-language retrieval. ” Retrieval from an image collection offers distinct characteristics from one in which the document to be retrieved is natural language text (Armitage & Enser, 1997; Goodrum, 2000). For example, the way in which a query is formulated, methods used for retrieval (e.g. based on low-level features derived from an image, or based on associated texts), the types of query, how relevance is assessed, the involvement of the user during the search process, and fundamental cognitive differences between the interpretation of visual versus textual media. Methods of image retrieval are typically based on visual content1 (e.g. colour, shape, spatial layout and texture), or by text/metadata associated with the image (see, e.g. (Smeulders et al., 2000) and (Goodrum, 2000)). Retrieval from such an archive presents a number of challenges and opportunities. The challenges come from matching queries to the relatively short descriptions associated with each image. Opportunities come from the unusual situation for CLIR systems of users being able to easily judge images for relevance. For those organisations managing image repositories in which text is associated with images (e.g. on-line art galleries), one way to increase user numbers is by enabling multilingual access to them. This paper is divided as follows: in section 2 we present previous work, section 3 the Eurovision system, section 4 our experimental methodology, section 5 our results and discussion, and section 6 our conclusions and avenues for future work in this area. 2. Previous Work To date there appears to be little work in cross language image retrieval, so in this section we first review the two main component research areas separately: image retrieval by associated text and cross language IR; followed by a review of the proposals and occasional attempts at performing image CLIR. 2. 1 Image retrieval via associated text Retrieval of images by text queries matched against associated text has been long researched, and approaches tend to reflect the nature of the application being addressed and the structure of the image collection. For example, collections such as the Web are networks of hypertext links which can be exploited to improve retrieval from hypermedia collections. As part of his work on multimedia retrieval, Dunlop (1993) examined the use of hyperlinked networks for image retrieval, using links to calculate representations for non-textual nodes. The ideas in this work were later extended to a study of image retrieval from art gallery Web sites by Harmandas et al. (1997), who showed that associated text was well-suited for retrieval over a range of query types. More recently, Chen et al. (1999) presented a method of multi-modal caption retrieval involving both content-based and textual-based modalities, enabling images in Web collections to be browsed. Images are clustered according to similarity based on textual features extracted from the URL, from the image ALT tag, and the image link, together with a simple colour histogram content-based approach. Other approaches have focused on extracting grammatical relations from the captions in order to describe image content. Although in the past it has been shown that Natural Language Processing (NLP) is detrimental to retrieval from documents (Smeaton, 1997), there are some 1 These are called Content-Based Information Retrieval (CBIR) systems. applications which involve short texts, e.g. image captions, where the situation becomes different and NLP can help improve retrieval. For example, Flank (1998) shows how searching on heads and head-modifier combinations obtains high precision and recall for image retrieval using captions. Elworthy et al. (2001) discuss an information retrieval system for searching a photographic database called ANVIL. The system deals with queries such as “camera with a lens” by creating a dependency structure where words and their relationships are captured. The grammatical relations of both query and caption are compared and a similarity score assigned based on matching dependency structures. The use of grammatical relations enable the query “yellow car” to match against variations of the request such as “car which is yellow” with high similarity score, but result in a low score for the caption “car which is not yellow” which would have otherwise been given a high score in a simple IR approach. Given the typically small caption lengths, attention has also been given to expanding the query using lexical resources to reduce the effects of mismatch between query and caption words. Smeaton and Quigley (1996) show that by pre-computing word-word similarities based on a WordNet-derived knowledge base, the effectiveness of retrieval can be improved for image retrieval where the captions are typically of length one sentence. Other systems involving natural language processing for image retrieval include the MARIE system (Guglielmo,1996) in which captions and queries are translated into a logical form based on the meaning of nominal compounds (sequences of consecutive nouns). MARIE has been used on a naval warfare image collection where photographs include pictures of weapons, aircraft, aerial views and test equipment. The captions describe events in the images, or unique characteristics and features of weapon systems. Queries to this collection reflect the captions by either describing an action, e.g. “aircraft hitting a test drone”, or specific object(s), e.g. “Sidewinder missile”. Flank et al. (1993) describe the SEYMOUR system which provides WWW access to more than 300,000 images via textual captions using NLP analysis on both the query and caption. Flank (2000) extends this work to deal with cross-language requests using both a dictionary lookup and online machine translation approach to translation. Finally, Srihari (1995) and Houghton (1999) describe the interaction of textual and photographic information in the Piction and NamedFaces systems respectively. Both aim to extract information from image captions, in particular extract proper names and grammatical relations such that people identified using image analysis can be labeled in the accompanying photographs, and photographs can be retrieved based on proper names assigned to the images. The approach taken in the NamedFaces system is simpler in that photographs of people are identified from Web pages, and hypertext links and image captions used to extract proper names for labeling. 2.2 Cross-Language Information Retrieval (CLIR) A considerable body