nils-herrmann/clean_scirepeval huggingface dataset:
Probabilistic web image gathering

We propose a new method for automated large scale gathering of Web images relevant to specified concepts. Our main goal is to build a knowledge base associated with as many concepts as possible for large scale object recognition studies. A second goal is supporting the building of more accurate text-based indexes for Web images. In our method, good quality candidate sets of images for each keyword are gathered as a function of analysis of the surrounding HTML text. The gathered images are then segmented into regions, and a model for the probability distribution of regions for the concept is computed using an iterative algorithm based on the previous work on statistical image annotation. The learned model is then applied to identify which images are visually relevant to the concept implied by the keyword. Implicitly, which regions or the images are relevant is also determined. Our experiments reveal that the new method performs much better than Google Image Search and a simple method based on more standard content based image retrieval methods.
Probabilistic travel modeling using GPS

Recreation simulation modeling, when combined with intelligent monitoring, is becoming a valuable tool for natural resource managers. The goal of recreation simulation is to accurately model recreational use, both current and future. Models are applied to gain a thorough understanding of the characteristics of recreation. Indicator variables such as visitor experience, carrying capacity and impact on resources can be computed. If the model is valid it can be used to predict future use as well as to investigate the effect of new scenarios and management decisions.
Evaluating image retrieval

We present a comprehensive strategy for evaluating image retrieval algorithms. Because automated image retrieval is only meaningful in its service to people, performance characterization must be grounded in human evaluation. Thus we have collected a large data set of human evaluations of retrieval results, both for query by image example and query by text. The data is independent of any particular image retrieval algorithm and can be used to evaluate and compare many such algorithms without further data collection. The data and calibration software are available on-line. We develop and validate methods for generating sensible evaluation data, calibrating for disparate evaluators, mapping image retrieval system scores to the human evaluation results, and comparing retrieval systems. We demonstrate the process by providing grounded comparison results for several algorithms.
Method for comparing content based image retrieval methods

We assume that the goal of content based image retrieval is to find images which are both semantically and visually relevant to users based on image descriptors. These descriptors are often provided by an example image--the query by example paradigm. In this work we develop a very simple method for evaluating such systems based on large collections of images with associated text. Examples of such collections include the Corel image collection, annotated museum collections, news photos with captions, and web images with associated text based on heuristic reasoning on the structure of typical web pages (such as used by Google(tm)). The advantage of using such data is that it is plentiful, and the method we propose can be automatically applied to hundreds of thousands of queries. However, it is critical that such a method be verified against human usage, and to do this we evaluate over 6000 query/result pairs. Our results strongly suggest that at least in the case of the Corel image collection, the automated measure is a good proxy for human evaluation. Importantly, our human evaluation data can be reused for the evaluation of any content based image retrieval system and/or the verification of additional proxy measures.
Exploiting Text and Image Feature Co-occurrence Statistics in Large Datasets

Building tools for accessing image data is hard because users are typically interested in the semantics of the image content. For example, a user searching for a tiger image will not be satisfied with images with plausible histograms; tiger semantics are required. The requirement that image features be linked to semantics means that real progress in image data access is fundamentally bound to traditional problems in computer vision. In this paper we outline recent work in learning such relationships from large datasets of images with associated text (e.g. keywords, captions, meta data, or descriptions). Fundamental to our approach is that images and associated text are both compositional—images are composed of regions and objects, and text is composed of words, or more abstractly, topics or concepts. An important problem we consider is how to learn the correspondence between the components across the modes. Training data with the correspondences identified is rare and expensive to collect. By contrast, there i s large amounts of data for training with weak correspondence information (e.g., Corel—40,000 images; captioned news photographs on the web—20,000 images per month; web images embedded in text; video with captioning or speech recognition). The statistical models learned from such data support browsing, searching by text, image features, or both, as well as novel applications such as suggesting images for illustration of text passages (auto-illustrate), attaching words to images (auto-annotate), and attaching words to specific image regions (recognition).
Color and Color Constancy in a Translation Model for Object Recognition

Color is of interest to those working in computer vision largely because it is assumed to be helpful for recognition. This assumption has driven much work in color based image indexing, and computational color constancy. However, in many ways, indexing is a poor model for recognition. In this paper we use a recently developed statistical model of recognition which learns to link image region features with words, based on a large unstructured data set. The system is general in that it learns what is recognizable given the data. It also supports a principled testing paradigm which we exploit here to evaluate the use of color. In particular, we look at color space choice, degradation due to illumination change, and dealing with this degradation. We evaluate two general approaches to dealing with this color constancy problem. Specifically we address whether it is better to build color variation due to illumination into a recognition system, or, instead, apply color constancy preprocessing to images before they are processed by the recognition system.
The effects of segmentation and feature choice in a translation model of object recognition

We work with a model of object recognition where words must be placed on image regions. This approach means that large scale experiments are relatively easy, so we can evaluate the effects of various early and midlevel vision algorithms on recognition performance. We evaluate various image segmentation algorithms by determining word prediction accuracy for images segmented in various ways and represented by various features. We take the view that good segmentations respect object boundaries, and so word prediction should be better for a better segmentation. However, it is usually very difficult in practice to obtain segmentations that do not break up objects, so most practitioners attempt to merge segments to get better putative object representations. We demonstrate that our paradigm of word prediction easily allows us to predict potentially useful segment merges, even for segments that do not look similar (for example, merging the black and white halves of a penguin is not possible with feature-based segmentation; the main cue must be "familiar configuration"). These studies focus on unsupervised learning of recognition. However, we show that word prediction can be markedly improved by providing supervised information for a relatively small number of regions together with large quantities of unsupervised information. This supervisory information allows a better and more discriminative choice of features and breaks possible symmetries.
Word sense disambiguation with pictures

We introduce a method for using images for word sense disambiguation, either alone, or in conjunction with traditional text based methods. The approach is based in recent work on a method for predicting words for images which can be learned from image datasets with associated text. When word prediction is constrained to a narrow set of choices such as possible senses, it can be quite reliable, and we use these predictions either by themselves or to reinforce standard methods. We provide preliminary results on a subset of the Corel image database which has three to five keywords per image. The subset was automatically selected to have a greater portion of keywords with sense ambiguity and the word senses were hand labeled to provide ground truth for testing. Results on this data strongly suggest that images can help with word sense disambiguation.
Recognition as Translating Images into Text

We present an overview of a new paradigm for tackling long standing computer vision problems. Specifically our approach is to build statistical models which translate from a visual representations (images) to semantic ones (associated text). As providing optimal text for training is difficult at best, we propose working with whatever associated text is available in large quantities. Examples include large image collections with keywords, museum image collections with descriptive text, news photos, and images on the web. In this paper we discuss how the translation approach can give a handle on difficult questions such as: What counts as an object? Which objects are easy to recognize and which are hard? Which objects are indistinguishable using our features? How to integrate low level vision processes such as feature based segmentation, with high level processes such as grouping. We also summarize some of the models proposed for translating from visual information to text, and some of the methods used to evaluate their performance.
A Comparison of Computational Colour Constancy Algorithms. Part Two: Experiments on Image Data

We test a number of the leading computational color constancy algorithms using a comprehensive set of images. These were of 33 different scenes under 11 different sources representative of common illumination conditions. The algorithms studied include two gray world methods, a version of the Retinex method, several variants of Forsyth's gamut-mapping method, Cardei et al.'s neural net method, and Finlayson et al.'s Color by Correlation method. We discuss a number of issues in applying color constancy ideas to image data, and study in depth the effect of different pre-processing strategies. We compare the performance of the algorithms on image data with their performance on synthesized data. All data used for this study is available on-line at http://www.cs.sfu.ca/~colour/data, and implementations for most of the algorithms are also available (http://www.cs.sfu.ca/~colour/code). Experiments with synthesized data (part one of this paper) suggested that the methods which emphasize the use of the input data statistics, specifically Color by Correlation and the neural net algorithm, are potentially the most effective at estimating the chromaticity of the scene illuminant. Unfortunately, we were unable to realize comparable performance on real images. Here exploiting pixel intensity proved to be more beneficial than exploiting the details of image chromaticity statistics, and the three-dimensional gamut-mapping algorithms gave the best performance.
Camera characterization for color research

In this article we introduce a new method for estimating camera sensitivity functions from spectral power input and camera response data. We also show how the procedure can be extended to deal with camera nonlinearities. Linearization is an important part of camera characterization, and we argue that it is best to jointly fit the linearization and the sensor response functions. We compare our method with a number of others, both on synthetic data and for the characterization of a real camera. All data used in this study is available online at www.cs.sfu.ca/∼colour/data.
A data set for color research

We present an extensive data set for color research that has been made available online (www.cs. sfu.ca/∼colour/data). The data are especially germane to research into computational color constancy, but we have also aimed to make the data as general as possible, and we anticipate a wide range of benefits to research into computational color science and computer vision. Because data are useful only in context, we provide the details of the collection process, including the camera characterization, and the data used to determine that characterization. The most significant part of the data is 743 images of scenes taken under a carefully chosen set of 11 illuminants. The data set also has several standardized sets of spectra for synthetic data experiments, including some data for fluorescent surfaces.
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary

We describe a model of object recognition as machine translation. In this model, recognition is a process of annotating image regions with words. Firstly, images are segmented into regions, which are classified into region types using a variety of features. A mapping between region types and keywords supplied with the images, is then learned, using a method based around EM. This process is analogous with learning a lexicon from an aligned bitext. For the implementation we describe, these words are nouns taken from a large vocabulary. On a large test set, the method can predict numerous words with high accuracy. Simple methods identify words that cannot be predicted well. We show how to cluster words that individually are difficult to predict into clusters that can be predicted well -- for example, we cannot predict the distinction between train and locomotive using the current set of features, but we can predict the underlying concept. The method is trained on a substantial collection of images. Extensive experimental results illustrate the strengths and weaknesses of the approach.
A comparison of computational color constancy algorithms. I: Methodology and experiments with synthesized data

We introduce a context for testing computational color constancy, specify our approach to the implementation of a number of the leading algorithms, and report the results of three experiments using synthesized data. Experiments using synthesized data are important because the ground truth is known, possible confounds due to camera characterization and pre-processing are absent, and various factors affecting color constancy can be efficiently investigated because they can be manipulated individually and precisely. The algorithms chosen for close study include two gray world methods, a limiting case of a version of the Retinex method, a number of variants of Forsyth's gamut-mapping method, Cardei et al.'s neural net method, and Finlayson et al.'s color by correlation method. We investigate the ability of these algorithms to make estimates of three different color constancy quantities: the chromaticity of the scene illuminant, the overall magnitude of that illuminant, and a corrected, illumination invariant, image. We consider algorithm performance as a function of the number of surfaces in scenes generated from reflectance spectra, the relative effect on the algorithms of added specularities, and the effect of subsequent clipping of the data. All data is available on-line at http://www.cs.sfu.ca/(tilde)color/data, and implementations for most of the algorithms are also available (http://www.cs.sfu.ca/(tilde)color/code).
A comparison of computational color constancy Algorithms. II. Experiments with image data

We test a number of the leading computational color constancy algorithms using a comprehensive set of images. These were of 33 different scenes under 11 different sources representative of common illumination conditions. The algorithms studied include two gray world methods, a version of the Retinex method, several variants of Forsyth's gamut-mapping method, Cardei et al.'s neural net method, and Finlayson et al.'s Color by Correlation method. We discuss a number of issues in applying color constancy ideas to image data, and study in depth the effect of different preprocessing strategies. We compare the performance of the algorithms on image data with their performance on synthesized data. All data used for this study are available online at http://www.cs.sfu.ca/(tilde)color/data, and implementations for most of the algorithms are also available (http://www.cs.sfu.ca/(tilde)color/code). Experiments with synthesized data (part one of this paper) suggested that the methods which emphasize the use of the input data statistics, specifically color by correlation and the neural net algorithm, are potentially the most effective at estimating the chromaticity of the scene illuminant. Unfortunately, we were unable to realize comparable performance on real images. Here exploiting pixel intensity proved to be more beneficial than exploiting the details of image chromaticity statistics, and the three-dimensional (3-D) gamut-mapping algorithms gave the best performance.
Learning the semantics of words and pictures

We present a statistical model for organizing image collections which integrates semantic information provided by associate text and visual information provided by image features. The model is very promising for information retrieval tasks such as database browsing and searching for images based on text and/or image features. Furthermore, since the model learns relationships between text and image features, it can be used for novel applications such as associating words with pictures, and unsupervised learning for object recognition.
Sensor sharpening for computational color constancy.

Sensor sharpening [ (1994)] has been proposed as a method for improving computational color constancy, but it has not been thoroughly tested in practice with existing color constancy algorithms. In this paper we study sensor sharpening in the context of viable color constancy processing, both theoretically and empirically, and on four different cameras. Our experimental findings lead us to propose a new sharpening method that optimizes an objective function that includes terms that minimize negative sensor responses as well as the sharpening error for multiple illuminants instead of a single illuminant. Further experiments suggest that this method is more effective for use with several known color constancy algorithms.
Clustering art

We extend a recently developed method (K. Barnard and D. Forsyth, 2001) for learning the semantics of image databases using text and pictures. We incorporate statistical natural language processing in order to deal with free text. We demonstrate the current system on a difficult dataset, namely 10000 images of work from the Fine Arts Museum of San Francisco. The images include line drawings, paintings, and pictures of sculpture and ceramics. Many of the images have associated free text which varies greatly from physical description to interpretation and mood. We use WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships. This allows us to impose a natural structure on the image collection that reflects semantics to a considerable degree. Our method produces a joint probability distribution for words and picture elements. We demonstrate that this distribution can be used: (a) to provide illustrations for given captions, and (b) to generate words for images outside the training set. Results from this annotation process yield a quantitative study of our method. Finally, the annotation process can be seen as a form of object recognizer that has been learned through a partially supervised process.
Modeling the statistics of image features and associated text

We present a methodology for modeling the statistics of image features and associated text in large datasets. The models used also serve to cluster the images, as images are modeled as being produced by sampling from a limited number of combinations of mixing components. Furthermore, because our approach models the joint occurrence image features and associated text, it can be used to predict the occurrence of either, based on observations or queries. This supports an attractive approach to image search as well as novel applications such a suggesting illustrations for blocks of text (auto-illustrate) and generating words for images outside the training set (auto-annotate). In this paper we illustrate the approach on 10,000 images of work from the Fine Arts Museum of San Francisco. The images include line drawings, paintings, and pictures of sculpture and ceramics. Many of the images have associated free text whose nature varies greatly, from physical description to interpretation and mood. We incorporate statistical natural language processing in order to deal with free text. We use WordNet to provide semantic grouping information and to help disambiguate word senses, as well as emphasize the hierarchical nature of semantic relationships.
Shadow Identification using Colour Ratios

In this paper we present a comprehensive method for identifying probable shadow regions in an image. Doing so is relevant to computer vision, colour constancy, and image reproduction, specifically dynamic range compression. Our method begins with a segmentation of the image into regions of the same colour. Then the edges between the regions are analyzed with respect to the possibility that each is due to an illumination change as opposed to a material boundary. We then integrate the edge information to produce an estimate of the illumination field.
Improvements to Gamut Mapping Colour Constancy Algorithms

In his paper we introduce two improvements to the three-dimensional gamut mapping approach to computational colour constancy. This approach consist of two separate parts. First the possible solutions are constrained. This part is dependent on the diagonal model of illumination change, which in turn, is a function of the camera sensors. In this work we propose a robust method for relaxing this reliance on the diagonal model. The second part of the gamut mapping paradigm is to choose a solution from the feasible set. Currently there are two general approaches for doing so. We propose a hybrid method which embodies the benefits of both, and generally performs better than either. We provide results using both generated data and a carefully calibrated set of 321 images. In the case of the modification for diagonal model failure, we provide synthetic results using two cameras with a distinctly different degree of support for the diagonal model. Here we verify that the new method does indeed reduce error due to the diagonal model. We also verify that the new method for choosing the solution offers significant improvement, both in the case of synthetic data and with real images.