On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

Understanding and interpreting classification decisions of automated image classification systems is of high value in many applications, as it allows to verify the reasoning of the system and provides additional information to the human expert. Although machine learning methods are solving very successfully a plethora of tasks, they have in most cases the disadvantage of acting as a black box, not providing any information about what made them arrive at a particular decision. This work proposes a general solution to the problem of understanding classification decisions by pixel-wise decomposition of nonlinear classifiers. We introduce a methodology that allows to visualize the contributions of single pixels to predictions for kernel-based classifiers over Bag of Words features and for multilayered neural networks. These pixel contributions can be visualized as heatmaps and are provided to a human expert who can intuitively not only verify the validity of the classification decision, but also focus further analysis on regions of potential interest. We evaluate our method for classifiers trained on PASCAL VOC 2009 images, synthetic image data containing geometric shapes, the MNIST handwritten digits data set and for the pre-trained ImageNet model available as part of the Caffe open source package.

section_content: With the increasing amount of digital information on the Web and on personal computers, the need for systems that are capable of automated indexing, searching, and organising multimedia documents incessantly grows. Automated systems have to retrieve information with high precision in order to be accepted by industry and end-users. Often, multimedia retrieval systems are evaluated on different test collections with different performance measures, which makes the comparison of retrieval performance impossible and limits the benefit of the approaches. Benchmarking campaigns counteract these tendencies and establish an objective comparison among the performance of different approaches by posing challenging tasks and by distributing test collections, topics, and measures. This paper presents an overview of the ImageCLEF 2011 Photo Annotation and Concept-based Retrieval Tasks. The two tasks aim at the automated detection of visual concepts in consumer photos and the retrieval of photos based on a certain topic. Section 2 introduces the tasks and the evaluation methodology. Following, Section 3 discusses the visual concepts and query topics which simulate the user's information need in image search. Then, Section 4 describes the test collection and the relevance assessment process. Section 5 summarizes the approaches of the participants. Following, the results for the annotation task and the concept-based retrieval task are presented and discussed in Section 6 and Section 7, respectively. Finally, Section 8 summarizes and concludes this paper.
section_name: Introduction

section_content: The Photo Annotation and Concept-based Retrieval Tasks pose an image analysis challenge which consists of two sub tasks. The annotation task aims at the automated annotation of consumer photos with multiple concepts. It is similar to the visual concept detection and annotation task (VCDT) as it was posed in the last years [1, 2]. This year, the participants are asked to annotate a test set of 10,000 Flickr images with 99 visual concepts. To solve this task, an annotated training set of 8,000 images is provided. The evaluation considers a fully assessed test collection to compare the approaches of the participants. The second challenge poses a concept-based retrieval task. The participants are asked to retrieve (up to) the 1,000 most relevant images in ranked order for a given topic out of a test collection of 200,000 images. In total 40 topics, each consisting of a logical connection of concepts from the annotation task, are provided. Concept detectors may be trained on the training set of the annotation task (8,000 images annotated with 99 visual concepts). The assessment incorporates a pooling strategy with crowdsourced relevance assessments. Both tasks can be solved by following three different approaches:

1. Automatic annotation with visual information only ("V") 2. Automatic annotation based on Flickr user tags and image metadata ("T") 3. Multi-modal approaches that consider visual information and/or Flickr user tags and/or EXIF information ("M")

The participants can choose one task or participate in both. Both tasks make use of a subset of the MIR Flickr 1 Million image dataset [3]. The MIR Flickr collection supplies all original tag data provided by the Flickr users (further denoted as Flickr user tags). These Flickr user tags are made available for the textual and multi-modal approaches of both subtasks. For most of the photos, the EXIF data is included and may be used.
section_name: Task Description

section_content: The main evaluation objectives of the two tasks in 2011 lie in the exploitation of different knowledge sources, the benefit of annotation approaches as part of the retrieval process, and the automated prediction of subjective concepts such as sentiments. Moreover, participants need to deal with an unbalanced amount of data per concept, a varying number of labels per image, the diversity of image content per concept, and the different qualities of image metadata.
section_name: Evaluation Objectives

section_content: The novel sentiment concepts are included in the Photo Tagging ontology [4] of the last years. The hierarchy allows making assumptions about the assignment of concepts to documents. Additionally, other relationships between concepts determine possible label assignments. The ontology restricts, for instance, the simultaneous assignment of some concepts (disjoint items) or defines that one concept postulates the presence of other concepts. The ontology allows the participants to incorporate semantic knowledge in their annotation algorithms, and to make assumptions about probable concept combinations.
section_name: Ontology

section_content: In the annotation task, the evaluation sticks to the concept-based and examplebased evaluation paradigm. For the concept-based evaluation, the Mean interpolated Average Precision (MiAP) is utilized, while the example-based evaluation applies the example-based F-Measure (F-Ex). Additionally, we introduce a novel performance measure called Semantic R-Precision (SR-Precision) which is based upon the example-based R-Precision, but incorporates the Flickr Tag Similarity (FTS) [5] to determine the semantic relatedness of visual concepts in the case of misclassifications. R-Precision calculates Precision at perfect Recall in an example-based evaluation scenario. The SR-Precision variant assigns misclassification costs based on the semantic relatedness among misclassified concepts. The semantic relatedness is derived from the FTS measure. In contrast to the Ontology Score with Flickr Context Similarity (OS-FCS) which was used in 2010 [2], the SR-Precision is able to incorporate ranked predictions instead of forcing the systems to provide binary decisions. However, this measure requires a normalization of classifier scores over different classifier outputs to deliver meaningful results. This requirement was not explicitly posed to the participants and therefore algorithms might not be optimally parameterized for this measure. 

The concept-based retrieval task evaluates performance on a test collection with incomplete relevance judgments. All submissions of the participants are pooled by using a pool depth of 100 documents per topic and run. Finally, the runs are evaluated with the Mean uninterpolated Average Precision (MAP), Precision@10 (P@10), Precision@20 (P@20), Precision@100 (P@100), and conceptbased R-Precision (R-Prec) with the trec eval 8. 1 program.
section_name: Evaluation Measures

section_content: Topics and visual concepts are strongly related to user needs and define the use cases of a system. While concepts are modality-independent (i. e., the event "birthday" might be detectable in the visual modality (birthday cake, people celebrating) as well as in the auditory modality (people singing a birthday song)), visual concepts are solely described by the visual content of a photo and are therefore language independent. This section introduces the visual concepts that are applied in the annotation task and the derivation of query topics for the retrieval task based on these concepts, query logs, and related work. Please note that the process of collecting images and defining visual concepts is different from related work. While usually the concept lexicon exists before images are collected, in the case of the ImageCLEF VCDT test collection, this process is decoupled and the images have been collected first. This approach is much closer to reality and poses new challenges, as objects are not necessarily centred in the image and the distribution of images per concept varies considerably.
section_name: Incorporation of user needs in the evaluation

section_content: The test collection for the annotation task contains manual annotations for 99 visual concepts. These concepts describe the scene (indoor, outdoor, landscape. . . ), depicted objects (car, animal, person. . . ), the representation of image content (portrait, graffiti, art. . . ), events (travel, work. . . ), or quality issues (overexposed, underexposed, blurry. . . ). This year, a special focus is laid on the detection of sentiment concepts. All in all, 49 concepts of the 53 concepts used in 2009 [1] were utilized again. The concept Canvas as well as the concepts No Visual Season, No Visual Place, and No Visual Time were discarded in this year's challenge. The 41 concepts which were added in 2010 are all reused. In 2011, nine novel sentiment concepts were added to the test collection. For the definition of sentiments, we follow the approach of Russell [6], who defines an emotional space with two dimensions (arousal and valence) on which emotional adjectives/sentiments can be placed. Valence spans from the negative pole "misery" to the positive pole "pleasure" on the x-dimension, while arousal spans from "passive" to "active" in the y-dimension. In this model, adjectives are grouped into eight affect concepts in circular order. The model was slightly adapted and an additional concept funny was included. The eight sentiments are structured according to their degrees in the circle as proposed by Russel. Partly, the wording is changed as to better fit the sentiments to describe images (e. g., an image cannot be excited or astonished, but it may look exciting to a human being). Starting with happy at 0 •, the circle is further composed of funny (about 30 • ), euphoric (70 • ), active (90 • ), scary (150 • ), unpleasant (180 • ), melancholic (210 • ), inactive, (270 • ), and calm/comforting (330 • ).
section_name: Definition of visual concepts

section_content: Based on the visual concepts, 40 topics for the concept-based retrieval task were constructed. We conceived that each topic contains a different number of relevant images, and that the topics comprise a range of difficulty levels. For the definition of relevant topics, we followed two approaches: First, we adapted topics from the ImageCLEF Wikipedia retrieval task [7], [8], as these topics Table 1 : Topics of the concept-based retrieval task. Topics, labelled with WR and an abbreviated year, are taken or adapted from the ImageCLEF Wikipedia retrieval tasks. were designed based on web-query logs and because they range from simple to semantic (hence highly difficult) topics as described in [9]. A total of 17 topics were directly applicable to our test data. Second, we examined interesting queries for the test collection. Based on the output for each query, it was decided if the chosen topic comprises an adequate occurrence in the test collection. The 40 resulting topics and their source are shown in Table 1. Sample images of the dataset were taken for clarification and provided as examples for the topics.
section_name: Definition of topics for the concept-based retrieval task

section_content: The relevance assessments for the annotation task and the concept-based retrieval task were acquired with a crowdsourcing approach using Amazon Mechanical Turk 1 (MTurk). MTurk is an online marketplace which distributes mini-jobs to an undefined crowd of people. At MTurk these mini-jobs are called HITs (Human Intelligence Tasks). The workers at MTurk, called turkers, can choose the HITs they would like to perform and submit the results to MTurk. The requester of the work collects the results from MTurk and approves or rejects the work of the turkers. Experiences with MTurk from ImageCLEF 2010 show the applicability of crowdsourcing for ground truth acquisition of image labels. This year, additional quality assurance mechanisms were incorporated to reduce the impact of spammers on the annotations.
section_name: Ground Truth Acquisition

section_content: The assessment of the sentiment concepts was performed by asking the turkers what sentiments an image conveys. The HIT template includes a definition of sentiments, synonymous sentiments, and example images (see Figure 1 ). The definitions are derived from WordNet 3. 0 2 and the Free Dictionary 3. Each survey comprises ten images. The image is depicted on the left, while on the right the adapted circumplex model of Russel (see Section 3. 1) is visualised, as illustrated on the example of one image in Figure 2. The option no sentiment should be chosen if no sentiment fits to the image. After selecting this checkbox, the turkers were asked to give a mandatory reason why no sentiment fits. We included this question in the survey to prevent turkers from clicking at this checkbox without thinking about the task. For all other sentiments, several choices could be selected at the same time. Additionally, the turkers were asked which reason let them decide for a sentiment: the motif or the overall impression of the image. They could choose on a five-point scale with the scales "motif" -"mostly motif" -"both equally" -"mostly overall impression" -"overall impression". 

The HIT template includes an automated verification procedure. For all ten images that belong to one HIT, it is verified that the survey is completely filled out before the submission of the task works. In the case of missing answers, the turkers see the corresponding questions marked in red. This procedure ensures that it is not too easy to answer randomly and submit spam, and it helps reducing our work to filter out incomplete answers that need to be republished. While it does not assure that all random annotators are excluded (as turkers still can randomly answer each question), this at least assures that it also costs some amount of work to cheat compared to the time that is needed to answer honestly.
section_name: Design of the annotation HIT template

section_content: The ground truth was acquired in different annotation batches. The pretest included 400 images of the training set arranged in surveys of ten images per HIT. Each HIT was annotated three times by a total of 22 turkers in an average an-Fig. 2 : Example HIT for a complete annotation. notation time of 3 minutes and 12 seconds and paid with 0. 05$. The purpose of the pretest was to understand if the template design and the task were understandable and if the turkers were able to solve the task. Results show that in about 50% of the images the turkers are agreeing on the sentiment (or choosing neighbouring sentiments) while in the other 50%, they chose opposite sentiments (like happy and melancholic). The rest of the training set of the Photo Annotation Task was annotated in altogether 4,225 HITs. Each HIT contained nine photos of the training set and one photo of the pretest as gold standard. The gold standard was built by a majority vote of the pretest images excluding the no sentiment concept and randomly placed into each HIT survey. Each HIT was annotated five times and rewarded with 0. 07$. On average, they were completed in 2 minutes and 36 seconds by a total of 258 distinct turkers. The test set was assessed in 5,560 HITs which each included nine images and one gold standard image of the training set. Each image of the test set was annotated five times. The HITs of the test set were divided into two batches (in order not to pose too many HITs at the same time) and annotated by 156 distinct turkers. Each HIT was rewarded with 0. 07$, again. For the first batch of 2,745 HITs, each HIT was annotated on average in 2 minutes and 8 seconds, while the 2,815 HITs of the second batch were annotated in average in 1 minute and 44 seconds. 

The verification of the work of the turkers is difficult, as the task of sentiment annotation is very subjective. Therefore, we followed several strategies on how to compare the annotations. The verification of the HITs of the training and test set with the gold standard images lead to a direct acceptance of 3,204 and Fig. 3 : Instructions in the HIT template for topic 4. 

4,358 HITs, respectively, allowing a deviation of at most 90 • on the affect circle. For HITs that did not pass the gold standard test, we compared the results of the HIT to the four answers of other turkers for the same HIT. For all images of the HIT, the deviation to the annotations of the other HITs was computed and the HITs were accepted when the deviation was equal or less to 90 • on the affect circle per image. A total of seven out of the 10 images had to fit. With this procedure all remaining HITs could be accepted. 

The final construction of the ground truth considers the majority vote for each image. In the case that no clear answer was given, we decided to discard any sentiment information for that image. In total, about 15% of the training set images and 14% of the test set images have no sentiment information. Interestingly, the no sentiment option was rarely chosen by the turkers. For none of the images of the training set and only for one image of the test set a majority of people decided for this concept.
section_name: Assessment statistics of the annotation task

section_content: In the relevance assessment of the concept-based retrieval task, the turkers were asked to mark all relevant images on the HIT template for a given topic. Each HIT template includes a definition of the topic and example images (see Figure 3 ). A HIT contains 22 images plus two gold standards images, which were used as a means of reliability control for the assessments. For each topic, we selected one image that fits the definition and one image that is not relevant for the given topic. Special attention was taken in the design of the irrelevant Fig. 4 : Sample images that are not in the scope of the topic fish in the water (top) and images that are covered by the topic (bottom). images per topic. Instead of using images that are clearly out of the scope of the topic, images that match the meaning of the topic quite close, but not exactly, were chosen; see Figure 4 for sample images of the topic fish in the water. The gold images were placed randomly in the HIT templates.
section_name: Design of the topic HIT templates

section_content: The number of HITs per topic is dependent on the total number of distinct images that were retrieved by the runs of the participants. Each HIT contained 24 images and was assessed by three turkers; so in total 7,868 HITs were processed. Accepted HITs were paid with 0. 03$. Each topic was processed by at least five (topic 29) and at most 41 (topic 12) distinct turkers. The average grading time varies per topic between 31 seconds (topic 25) and 1 minute and 19 seconds (topic 15). To increase the reliability of the relevance judgments, the results were subjected to a post screening procedure. The assessments of a turker were rejected if the gold standard images were not marked correctly. These HITs were published again until for all HITs three reliable results were available. In the next step, the assessments of the three turkers per HIT were compared with each other. As the task of selecting relevant images for a topic is a subjective task, its verification is difficult. In an additional step, we visualized the images that were assessed as relevant and estimated the number of false assignments and missing assignments. Depending on these results, the number of votes that were necessary to define an image as relevant were chosen for each topic. For most topics, a majority vote from at least two of the three assessors was necessary. A minority vote was used for only five topics, while all assessors had to agree on relevance for three topics. 

Altogether, 48 groups registered for the challenge. 42 groups signed the license agreement and were provided with the test collections. For the annotation task, 18 groups submitted results in altogether 79 runs. The number of runs was restricted to a maximum of 5 runs per group. In total, there were 46 submissions using only visual information, 8 submissions using only textual information, and 25 submissions utilising multi-modal approaches. For the retrieval task, 4 groups submitted results in a total of 31 runs. The maximum number of runs per group was set to 10. The submissions include 14 visual runs, 7 textual runs, and 10 multi-modal runs. The runs can be subdivided into 16 runs that retrieved all images in a completely automated fashion and 15 runs that included a manual intervention in the query generation step or relevance feedback. All participants that submitted to the retrieval task also took the challenge in the annotation task. The teams and their approaches are briefly introduced in the following:

BPACAD [10] : The team of the Computer and Automation Research Institute of the Hungarian Academy of Science submitted one textual, two visual and three multi-modal runs to the annotation task. Their approach is based on a kernel weighting procedure using visual Fisher kernels and a Flickr-tag based Jensen-Shannon divergence based kernel. Classification uses a linear SVM trained for each concept separately. 

BUFFALO: The team of the University at Buffalo, New York, USA submitted five visual runs for the annotation task. They follow two approaches: the first considers a local linear coordinate method to learn concepts with a regression method. The second uses a combination of GIST and colour features and classifies the images by a neural network. 

CAEN [11] : The group of University of Caen, France participated with four visual runs in the annotation task. The proposed approach uses visual image features, such as SIFT, HOG, Texton, LAB, SSIM, and Canny, and aggregated them by a Bag-of-Words (BoW) model into a global histogram. Fisher Vectors and contextual information were used as enhancement of the BoW-models. The classification considers SVM models trained for each concept separately. 

CEALIST [12] : The team from the Laboratory of Vision and Content Engineering, France submitted one textual, one visual, and three multi-modal runs to the annotation task. The textual descriptor is based on semantic similarity between tags and visual concepts. Two distances were used: one based on the Wordnet ontology and one based on social networks. The visual component considers various local and global features, such as Fisher vectors as well as colour and edge features. Late fusion was used to combine visual and textual modalities. 

DBIS: The team of the Technical University of Cottbus, Germany submitted five runs in the visual configuration to the annotation task. They use various features and investigate the influence of several parameters in clustering on the annotation performance. 

HHI [13] : The team of Fraunhofer HHI, Berlin, Germany submitted five visual runs to the annotation task. Their approach is based on the BoW model. A feature fusion of the opponent SIFT descriptor and the GIST descriptor was done in order to improve the classification performance of scene-based concepts. HHI investigates a sampling of informative images in the training procedure, which resulted in qualitative as well as runtime performance gains. A post-classification processing step is incorporated, which refines classification results based on rules of inference and exclusion between concepts. 

IDMT [14] : The group of Fraunhofer IDMT, Ilmenau, Germany submitted one textual and four multi-modal runs to the annotation task. Their approach focuses on the fusion of multi-modal information and the exploitation of Flickr user tags. As visual features, they employ RGB-SIFT features in a codebook approach and classify the images with a one-against-all strategy using a SVM with RBF kernel. 

ISIS [15] : The Intelligent Systems Lab of the University of Amsterdam, The Netherlands participated with five runs in the annotation task (3V, 2M) and ten runs (10V) in the retrieval task. All runs of the annotation task use several colour SIFT features with Harris-Laplace and dense sampling, and apply the SVM classifier. The multi-modal runs further include binary vectors for the most frequent Flickr user tags. In the retrieval task, three runs are computed completely automated and seven include a manual intervention by following two approaches. In the fully automated runs, a combination of the provided positive example images and random irrelevant images were used to train the concept detector. For the human topic mapping, a human reads the topic and then selects the relevant concept(s). The probability scores of these concepts are then combined using either summation or multiplication. In the human topic inspection approach, relevance feedback was used to improve results. 

LAPI [16] : The group of Laboratorul de Analiza si Prelucrarea Imaginilor, Universitatea Politehnica Bucuresti, Bucharest, Romania submitted two runs using a visual-only approach. They combine colour and structural features and adopt a Linear Discriminant Analysis for classification. Post-processing considers joint probabilities of concept occurrences in the training set for label elimination. 

LIRIS [17] : The group of Université de Lyon, CNRS, France participated in the annotation task with two textual, one visual, and two multi-modal runs. They consider two textual descriptors: one is based on a semantic distance between the text and an emotional dictionary, the other one contains the valence and arousal meanings by making use of the Affective Norms for English Words dataset. In the visual approaches, different visual features including colour, texture, shape, and high level aesthetic features are applied. Performance is compared using different fusion strategies as well as Adaboost and SVM classifiers. 

MEIJI [18] : The group of Meiji University, Kanagawa, Japan submitted five runs (2V, 1T, 2M) to the annotation task and ten completely automated runs (2V, 2T, 6M) to the retrieval task. Their approach is based on visual word co-occurrence using the BoW model and global colour features as well as textual features derived by tf-idf weigths of Flickr user tags. Classification is performed by an adaptation of the so-called confabulation model. 

MLKD [19] : The Machine Learning and Knowledge Discovery group of the Aristotle University of Thessaloniki, Greece participated in the annotation task with five runs (1V, 1T, 3M) and in the retrieval task with two automated and eight semi-automated runs (2V, 4T, 4M). They approach the photo annotation challenge with multi-label learning algorithms based on Random Forests as base classifier. The visual features consider seven local descriptors with two sampling strategies. The textual models are based on a Boolean BoW representation including word stemming, stop words removal, and feature selection. The multi-modal approach considers a hierarchical late-fusion of the modalities. For the concept-based retrieval task two approaches were used: one based on the concept relevance scores in a manual configuration and one automated approach which is based solely on the sample images using textual information. 

MRIM [20] : The team of Grenoble University, France submitted four runs (3V, 1M) to the annotation task. Classification considers multiple SVM classifiers with RBF kernel. In the visual runs, several global and local colour and texture descriptors are applied and dimension reduction techniques are investigated. The multi-modal run additionally considers Flickr user tags as simple textual features in a late fusion of SVM classifier scores. 

MUFIN [21] : The Faculty of Informatics, Masaryk University, Brno, Czech Republic participated with four multi-modal runs in the annotation task. Their approach is based on a free-text annotation system that assigns arbitrary words to web images by visual and textual neighbour searching. For the textual search, the EXIF data and image descriptions were used, while the visual search considers different MPEG-7 descriptors. The search considers the Profimedia dataset to find the nearest neighbours and transfers its annotations to the ImageCLEF test collection including a removal of stopwords and names. The resulting words were transformed into the fixed set of 99 visual concepts of the annotation task with the help of WordNet and the provided ontology. 

NII [22] : The team of the National Institute of Informatics, Tokyo, Japan participated with five visual runs in the annotation task. Their models are using global and local features. As for global features, colour moments, colour histogram, edge orientation histogram, and local binary patterns are applied. As for local features, keypoint detectors such as Harris Laplace, Hessian Laplace, Harris Affine, and Dense Sampling are used to extract SIFT-descriptors. Classification is performed with a SVM classifier. 

REGIMVID [23] : The research group on Intelligent Machines, University of Sfax, Tunisia submitted one textual run to the annotation task and one textual, automated run to the retrieval task. Their approach focuses on the exploration of Flickr tags to extract contextual relationships of tag relations. Therefore, two types of contextual graphs are modeled: an inter-concepts graph and a concept-tags graph. 

TUBFI [24] : The joint submission of the Machine Learning Group, Berlin Insitute of Technology and Fraunhofer FIRST Berlin, Germany consists of four visual and one multi-modal run to the annotation task. Classification considers non-sparse multiple kernel learning and multi-task learning. Different extensions of the BoW models with respect to sampling strategies and BoW mappings were proposed. The multi-modal run further considers frequent Flickr user tags based on a soft mapping for textual BoWs and Markov random walks over tags. UNIKLU: The team of the Institute of Information Technology, Alpen-Adria University, Klagenfurt, Austria participated with four visual runs in the annotation task. They made use of the LIRE framework and applied several features such as SIFT, SURF, MSER, CEDD, FCTH, and colour histograms and classified the images with a linear SVM. Two of the runs incorporate an automated post-processing of the classification results.
section_name: Assessment statistics of the retrieval task

section_content: This section illustrates the results for the annotation subtask. First, the overall results of all teams are presented, independent of the configuration. In the following subsections the results per configuration are highlighted. The results for all runs can be found at the Photo Annotation Task website 4. 

The task was solved best with a MiAP of 0. 443 (TUBFI), followed by a MiAP of 0. 437 (LIRIS) as illustrated in Table 2. Both runs make use of multi-modal Table 3 : Summary of the results for the evaluation per example. The table shows the F-Ex, SR-Precision, and run configuration for the best run per group. 

Team Rank F-Ex Config. 

Team Rank SR-Prec. Config. information. Table 3 depicts the overall rankings for the results of the evaluation per example. The best results in terms of F-Ex are achieved in a multi-modal configuration with 0. 622 (ISIS), followed by 0. 600 F-Ex (CAEN) which makes use of a visual configuration. In terms of SR-Precision, the best run scores with 0. 742 SR-Precision (ISIS) in a multi-modal run, followed by 0. 729 SR-Precision (BPACAD) in a visual run.
section_name: Annotation Task: Results

section_content: Table 4 shows the results of the best run of each group that participated in the visual configuration evaluated with all three evaluation measures. The best results in the visual configuration are achieved by the TUBFI team in terms of MiAP, closely followed by the team CAEN (0. 388 vs 0. 382 MiAP). In the example-based evaluation, ISIS scored best for both measures followed by CAEN (F-Ex) and BPACAD (SR-Precision).
section_name: Results for the visual configuration

section_content: The results for the textual runs are presented in Table 5. The best run scores with 0. 346 MiAP (BPACAD), followed by 0. 326 MiAP (IDMT, MLKD). In the example-based evaluation the best run scores with 0. 525 F-Ex (IDMT) and 0. 677 SR-Precision (IDMT) followed by 0. 506 F-Ex (MLKD) and 0. 676 SR-Precision (CEALIST, LIRIS). 

Table 4 : Summary of the results for the evaluation per concept in the visual configuration. The table shows the best run per group. 

Team Rank MiAP Team Rank F-Ex Team Rank SR-Prec.
section_name: Results for the textual configuration

section_content: Table 6 depicts the results for the best multi-modal configuration of each group. As already stated, the run of TUBFI achieves the best overall results in terms of MiAP. In the example-based evaluation, ISIS scores best overall with 0. 622 F-Ex and 0. 742 SR-Precision.
section_name: Results for the multi-modal configuration

section_content: Last year only two runs considered the textual configuration. In contrast, this year eight textual runs were submitted by seven teams. This allows for a more reliable analysis of the performance of textual runs in image annotation. The Table 5 : Summary of the results for the evaluation per concept in the textual configuration. The table shows the best run per group. 

Team Rank MiAP Team Rank F-Ex Team Rank SR-Prec. performance of textual runs is close to the results that can be achieved in the visual configuration. The best visual run achieves a MiAP of 0. 388 in contrast to the best textual run, which scores with a MiAP of 0. 346. The difference of 4. 2% is rather small, especially when considering that not for all images EXIF data and Flickr user tags exist. In the example-based evaluation, the difference between visual and textual runs is more significant. Visual runs score better by about 9% in terms of F-Ex and about 6% in terms of SR-Precision. Results in the multimodal configuration outperform classification with single modality information in the visual configuration by 5. 5% and the textual configuration by about 10% in terms of MiAP. For the example-based measures F-Ex and SR-Precision, differences are very small with 1% for the visual configuration. Comparing the multi-modal to the textual configuration, differences are significant and lie by about 10% and 6. 5% for F-Ex and SR-Precision, respectively.
section_name: Comparison of achievements with different information sources

section_content: In Table 7, the results for each concept are summarized independent of the configuration. On average, the concepts could be detected with a MiAP of 0. 48 considering the best run per concept out of all configurations. In general, 79 concepts were best detected with a multi-modal approach, 17 concepts were detected best with a visual approach, and 3 concepts were detected best by a textual approach. High performance is achieved for the concepts Neutral-Illumination, No-Persons, No-Blur, and Outdoor. Following, the concepts Sky, Day, Clouds, and Plants were annotated with high scores. The worst annotation quality was achieved for the concept abstract followed by the concepts work, graffiti, technical, old-person, and boring. 

In the evaluation in 2010, a great difference in prediction quality among the concepts from 2009 and the ones newly introduced in 2010 of 0. 57 MiAP and 0. 37 MiAP could be seen. This difference is still present in this evaluation cycle. The concepts from 2009 (number 1-49) could be detected with a MiAP Table 7 : This table presents the best annotation performance per concept, achieved by any team in any configuration, in terms of iAP. It lists the concept name, the iAP score, the team, and the configuration of the run. of 0. 57 considering the best prediction for each concept out of all runs, while the 2010 concepts (numbers 50-90) improved minimally to a MiAP of 0. 38. The new sentiment concepts (numbers 91-99) can be detected with a MiAP of 0. 39. Although these are arguably very subjective concepts, the detection algorithms are capable of identifying a strong trend of sentiments correctly. Especially, the sentiments calm and inactive could be detected very well, while the sentiments scary and euphoric were annotated worst. However, one has to note that the 2010 concepts occur on average in 7. 9% of the training set images, while the 2009 and 2011 concepts are visible in 18% and 14% of the training set, respectively. Therefore, the algorithms had more example images to learn the sentiments in comparison to the more object-based concepts introduced in 2010.
section_name: Annotation performance per concept

section_content: In the following, the results of the concept-based retrieval task are presented and discussed. The participation of four teams was lower with respect to the annotation task. Despite, 31 runs have been submitted in all configurations, consisting of 10 multi-modal, 7 textual, and 14 visual runs. Approximately half of the systems used a completely automated processing. 

Table 8 depicts all runs and indicates the configuration and the degree of automation of each run. Results are sorted in terms of MAP. The MAP value ranges from 0. 1640 to 0. 0013. Overall, the task was solved best with a MAP of 0. 164 by the MLKD group, who used a multi-modal configuration with manual query formulations. It can clearly be seen that the approaches using manual processing achieve better results than the automated versions. The best performing automated run achieves a MAP of 0. 0849 (MLKD). The multi-modal and textual configurations of MLKD work significantly better than their visual ones. MLKD provides also the best MAP value for a textual configuration, which is only 0. 0094 points lower than the overall best run. The best working visual configuration, also with manual query formulation, was submitted by ISIS, achieving a MAP of 0. 0997. MEIJI provided solely automated runs, for which the multimodel approaches outperform the textual and visual runs. REGIMVID provided one automated, text-based configuration, which achieves a MAP value of 0. 0042. 

Table 9 presents the best run per team in the three configurations. If a team submitted an automated and a manual run in the same configuration, the results for both runs are illustrated. The direct comparison of automated and manual runs per team shows that the manual runs work best, independent from the configuration (textual, visual, multi-modal). This is mostly apparent in the big difference -nearly factor two -of the textual configuration of MLKD. The results of the two approaches submitted by ISIS support this interpretation. The manually processed run outperforms the automated approach with a MAP of 0. 0997 vs. 0. 0430. The performance difference between the two multi-modal configurations submitted by MKLD and MEIJI can partially be explained by the different degree of automation. MEIJI uses a fully automated system resulting Table 8 : The table shows the results for all runs sorted by MAP. Automated runs are abbreviated with "A", while manual query formulations are abbreviated with "M" in the column Automation. 9 : This table presents the best results per team in the single configurations differentiated by the degree of automation. In the case a team submitted an automated and a manual run in a particular configuration, both runs are listed. Results are sorted in terms of MAP. 

Team MAP P@10 P@20 P@100 R- The boxplots of the MAP scores of all approaches per topic in Figure 5 allow for a more detailed examination. Most topics show a wide variation among the obtained MAP values, which indicates the differences in performance per topic. This can be seen, e. g., for the topics 33 (cars and motion blur ) and 28 (fireworks). For topic 33, the lowest MAP scores are 0. The highest value (0. 5218) for this topic is achieved by ISIS with an automated, visual configuration. MAP scores for topic 28 are in the range of 0 to 0. 423. A closer look reveals that only for two topics (33 and 5) MAP values over 0. 5 are reached. This is most surprising in the case of topic 5 (riders on horse), due to the consistent low MAP values of the other approaches. The six runs performing significantly better than the rest are provided by MLKD and use textual and multi-modal information. The figure also shows that the MAP scores for topic 18 (on the far right) are homogeneously low. It can be concluded that all approaches had great difficulties identifying relevant images for female old person. The topics 13, 21 and 26 were also hard to identify (MAP close to 0) for most of the approaches, but better performing outliers clearly exist. For topic 13 (female person(s) doing sports) some approaches of MLKD reach values above 0. 14. The same observation, with values close to 0. 1, applies to topic 21 (scary dog(s)). The most striking effect can be observed for houses in mountains (26): Only two ISIS approaches were able to achieve significantly higher MAP values than 0. The best sentiment topic
section_name: Concept-based Retrieval Task: Results

section_content: The ImageCLEF 2011 Photo Annotation and Concept-based Retrieval Tasks posed two image analysis challenges that could be solved with three general configurations: textual-based analysis, visual-based analysis, and multi-modal analysis. The aim of the annotation task was to automatically annotate images with 99 concepts in a multi-label scenario. The task attracted a considerable number of international teams with a final participation of 18 teams that submitted a total of 79 runs. The results show that the annotation task could be solved reasonably well, with the best multi-modal run achieving a MiAP of 0. 443 in the multi-modal configuration, a MiAP of 0. 388 in the visual configuration, and a MiAP of 0. 346 in the textual configuration. For the evaluation per example, the best multi-modal run achieves 0. 62 F-Ex, the best visual run scores with 0. 61 F-Ex, and the best textual run with 0. 53 F-Ex. All in all, the multi-modal approaches got the best scores for 79 out of 99 concepts, followed by 17 concepts that could be detected best with the visual approach and 3 that won with a textual approach. In general, the multi-modal approaches outperformed visual and textual configurations for nearly all performance measures of the teams that submitted results for more than one configuration. The concept-based retrieval task asked participants to retrieve the most relevant images given certain topics. The topics were constructed based on user needs and query logs, and consist of a Boolean connection of several visual concepts. In total, 4 teams participated in this novel challenge and submitted 31 runs. 10 runs belong to the multi-modal configuration, 14 runs were submitted in the visual configuration, and 7 runs are based on textual information. The best multi-model configuration obtained a MAP value of 0. 164, the textual configuration scored best with 0. 1546 MAP, and the best visual run achieves a score of 0. 0997 MAP. The task was solved by 16 completely automated approaches and 14 runs which include manual intervention. It was observed that most manually processed runs work best, independent from the configuration (textual, visual, or multi-modal). They achieve MAP values in the range of 0. 164 and 0. 0295, whereas the automated solutions range between scores of 0. 0849 and 0. 0013 MAP. A closer examination showed that all approaches had great difficulties to identify relevant images for the topic female old person. Also the topics 5, 13, 21, and 26 were hard to identify as well, but here some approaches were able to reach higher MAP values. Especially, the topic riders on horse shows very strong outliers with high MAP values above 0. 5. The obtained MAP scores of the remaining topics vary widely which points to a large variation in the difficulty level of topics. Some configurations are able to achieve MAP values higher than 0. 5 for individual topics. Considering the topic female old person, all runs show nearly the same low performance. This seems to be an extremely critical topic for concept-based image retrieval.
section_name: Conclusions

section_name: Acknowledgements

section_content: We would like to thank the CLEF campaign for supporting the ImageCLEF initiative. This work was partly supported by grant 01MQ07017 of the German research program THESEUS funded by the Ministry of Economics.
section_name: Acknowledgements

section_name: Concept iAP

section_content: Team Config.
section_name: Concept iAP