Integrating Collaborative Tagging and Emergent Semantics for Image Retrieval

In this paper, we investigate the combination of collaborative tagging and emergent semantics for improved data navigation and search. We propose to use visual features in addition to tags provided by users in order to discover new relationships between data. We show that our method is able to overcome some of the problems involved in navigating databases using tags only, such as synonymy or different languages, spelling mistakes, homonymy, or missing tags. On the other hand, image search based on visual features can be simplified substantially by the use of tags. We present technical details of our prototype system and show some preliminary results. 1. FROM IMAGE RETRIEVAL TO EMERGENT SEMANTICS Searching large databases of images is a well known problem commonly referred to as image retrieval. The main challenge in this field has been quoted as automatically generating high-level descriptions of images. This is often called “bridging the semantic gap” between high-level descriptions a user will naturally look for, and low-level visual features that can be extracted from the data. This problem is known to be hard, if not impossible to solve. In order to provide a high-level description of an image, two requirements need to be fulfilled: first, the objects in an image need to be recognised. This is, however, not yet possible in a general way. Second, starting from the image components, the system would need to be able to infer the meaning of the image. It has been argued though that it is not always possible to break down the meaning of an image into its components. Images in general do not have an intrinsic meaning but the meaning emerges from the interaction with a user and by placing an image into the context of other images. This concept is called emergent semantics [7] (see also [3, 8]). According to this argument, solving the problem of automatically annotating an image in a general way is not fea∗Luc Steels is also with the Sony Computer Science Laboratory Paris, 6 rue Amyot, 75005 Paris, France Copyright is held by the author/owner(s). WWW2006, May 22–26, 2006, Edinburgh, UK. sible, even if it was possible to detect all its objects. It was long assumed that manual annotations for images would not be provided by users because it is a tedious and time consuming process. However, recent developments like Flickr [1] have shown that this is no longer true. Users do tag images, mainly their own, but also those of other users. Reasons for this development might be the motivation that comes along with the exposure of personal photos to others. If a photo is tagged in a reasonable way, others can find it and leave comments that are mostly encouraging. User tagging does, however, not only work for personal data. Sites like Last.fm [2] where user tag music tracks are very successful as well. These recent developments indicate a shift of the traditional image retrieval paradigm. Provided with the right motivation, users do give high-level descriptions for images which seem to make the need for automatic annotation less relevant. We believe that the new challenge is to combine both user-provided image descriptions and low-level visual features extracted from the data. 2. PERSONAL AND SOCIAL TAGS Personal tags, introduced by a specific person for himself, are very different from expert tags. Personal tags are mostly associative, high-level, subjective, and inconsistent. Expert tags, on the other hand, are an attempt to be objective and consistent. The underlying assumption of expert tags is that all users have the same interpretation of the data, and the same categorisation. This is, however, not consistent with the concept of emergent semantics. Personal tags become social tags when exposed to others, as seen on Flickr, for example. Although still associative, subjective, and inconsistent, the mass effect of social tags leads to some form of relevance and coherence. The exposure to the tags and photos of other users creates an implicit feedback between users and the tagging system. It can be observed that over time, the relative frequency of tags used to label a resource tends to approach a constant value [5]. This indicates that collaborative tagging is able to coordinate the actions of Web users to create coherent annotations of the shared photos. This confirms the findings of earlier studies that used computational models to investigate the emergence of a lexicon shared by a population of autonomous agents [9, 10]. 3. INTEGRATING TAGS AND IMAGE FEATURES Although a powerful tool to navigate image archives, searching by tags alone has a number of drawbacks. First, people make mistakes while tagging, such as spelling mistakes, or accidential tagging with the wrong tag. Second, synonymy or different languages can only be handled by explicit tagging. Third, there is no solution to cope with homonymy, i.e. to distinguish different meanings of a word. We believe that by combining tags and visual features, we can overcome the disadvantages of both approaches taken on their own. In order to approach this problem, we first need to consider the relation between tags and visual features. Our first observation is that tags provide, in general, high-level descriptions that cannot be derived from image features. Although some tags can be related to visual features (the tag can be grounded) e.g. “red”, or “blackandwhite”, this is rather an exception than the rule. Second, a personal tag can refer to images that do not possess a common visual content. For example, the tag “paris” might be assigned to a photo of the Eiffeltower, a photo of a restaurant, and a photo of a hotel room. For social tags, this problem is even more severe. Even if there was visual consistency within a set tagged by one user, consistency cannot be generally assumed to be found across several users. A set of images sharing the same social tag represents the union of categories from different users. Since we do not believe that all users share excactly the same categories, visual consistency among images tagged in the same way is not very likely to occur. Thus, we cannot assume that a tag simply corresponds to a category. Therefore, we do not seek to capture the semantics for tags directly, but for a set of images, or a collection, explicitly selected by the user. A collection is aquired through the actions of a user, such as browsing, searching and selecting images. 4. TECHNICAL DETAILS We tested our ideas by implementing an interface that combines intuitive navigation by tags and a user-friendly way to search according to visual features. The images and tags used by our system come from real users, downloaded from Flickr. The current version includes about 3000 photographs from 12 randomly chosen users. In the following, we describe the visual features and the implementation of our retrieval process. Most of the techniques we used reflect either the state-of-the-art in image retrieval or are well-established standards in image analysis and pattern recognition. The idea of our system is to advance neither of these fields but to use the available tools in a new, intuitive, and creative way.