DOI: 10.5281/zenodo.1038546
title: 2015 Imageclef Webupv Collection

description: This document describes the WEBUPV dataset compiled for the ImageCLEF 2015<br>
Scalable Concept Image Annotation challenge. The data mentioned here indicates<br>
what is ready for download. However, upon request or depending on feedback<br>
from the participants, additional data may be released.

The following is the directory structure of the collection, and bellow there<br>
is a brief description of what each compressed file contains. The<br>
corresponding MD5 checksums of the files shown (for verifying a correct<br>
download) can be found in md5sums.txt.

<br>
Directory structure<br>
-------------------

.<br>
|<br>
|--- README.txt<br>
|--- md5sums.txt<br>
|--- webupv15_data_lists.zip<br>
|<br>
|--- annotations/imageclef2015.dev.bbox.*.gz<br>
|--- annotations/imageclef2015.dev.textdesc.*.gz<br>
|--- annotations/imageclef2015.subtask2cleantrack.*.input_bbox.*.gz<br>
|--- annotations/imageclef2015.subtask2cleantrack.dev.textdesc.*.gz<br>
|--- annotations/imageclef2015.dev.bbox.supplement.*.gz<br>
|--- annotations/imageclef2015.hierarchy.*.gz<br>
|--- annotations/imageclef2015.concept_to_parents.*.gz<br>
|<br>
|--- feats_textual/<br>
|      |<br>
|      |--- webupv15_data_textual_pages.zip<br>
|      |--- webupv15_data_textual.scofeat.gz<br>
|      |--- webupv15_data_textual.keywords.gz<br>
|<br>
|--- feats_visual/<br>
       |<br>
       |--- webupv15_data_visual_images.zip<br>
       |--- webupv15_data_visual_gist.dfeat.gz<br>
       |--- webupv15_data_visual_sift_1000.sfeat.gz<br>
       |--- webupv15_data_visual_csift_1000.sfeat.gz<br>
       |--- webupv15_data_visual_rgbsift_1000.sfeat.gz<br>
       |--- webupv15_data_visual_opponentsift_1000.sfeat.gz<br>
       |--- webupv15_data_visual_colorhist.sfeat.gz<br>
       |--- webupv15_data_visual_getlf.sfeat.gz<br>
       |--- webupv15_data_visual_vgg16-fc8.dfeat.gz

<br>
Contents of files<br>
-----------------

* annotations/imageclef2015.dev.bbox.*.gz

  Development set ground truth localised annotations for sub task 1.

  The format for the development set of annotated bounding boxes of<br>
  the concepts is

  &lt;image_ID&gt; &lt;seq&gt; &lt;Concept&gt;  &lt;confidence&gt; &lt;xmin&gt; &lt;ymin&gt; &lt;xmax&gt; &lt;ymax&gt;

  The development set contains 1,979 images. The bounding boxes may enclose<br>
  single instances (a single tree) or grouped instances (e.g. group of trees),<br>
  depending on the context. The annotations are not exhaustive: the emphasis<br>
  is on concepts that are interesting enough to be described in the image,<br>
  although background objects are also optionally annotated by our annotators<br>
  in many cases. Also note that a person might not be annotated if the<br>
  annotator could not decide whether the person is a man/woman/boy/girl.

<br>
* annotations/imageclef2015.dev.textdesc.*.gz

  Development set ground truth textual description annotations of images for<br>
  sub task 2

  The format is:

  &lt;image_ID&gt; \t &lt;text_description_seq&gt; \t &lt;textual_description&gt;

  The development set contains 2,000 images with 5 to 51 textual descriptions<br>
  per image (mean: 9.492, median: 8). Please note that the sentences contain a<br>
  mix of both American and British English spelling variants (e.g. color vs<br>
  colour) -- we have decided to retain this variation in the annotations to<br>
  reflect the challenge of real-world English spelling variants. Basic<br>
  spell-correction has been performed on the textual descriptions, but we cannot<br>
  guarantee that they are completely free from spelling or grammatical error.

  Changelog for v20150306: We have removed some poor textual descriptions that<br>
  managed to slip past our quality control check. Please use this latest version<br>
  (v20150306) for your development needs.

<br>
* annotations/imageclef2015.subtask2cleantrack.*.input_bbox.*.gz

  Input bounding boxes for the clean track of SubTask 2. This is just a selected<br>
  subset of 500 development images from imageclef2015.dev.bbox above (please refer<br>
  to above for file format).

<br>
* annotations/imageclef2015.subtask2cleantrack.dev.textdesc.*.gz

  Annotated textual descriptions for 500 development images, to be used to evaluate<br>
  the content selection ability of the text generation system in the clean track of<br>
  SubTask 2. The format is the same as the original imageclef2015.dev.textdesc<br>
  file, except that we further annotated textual terms with their corresponding<br>
  input bounding boxes, for example [[[dogs|0,4]]] in a textual description refers<br>
  to the two instances of dogs with the bounding box id 0 and 4 in<br>
  imageclef2015.subtask2cleantrack.dev.input_bbox.

  Note that not all descriptions from the original imageclef2015.dev.textdesc are<br>
  used in this version, and as such the sequence numbers of the descriptions may not<br>
  necessarily be contiguous as we retained the sequence numbers from the original<br>
  file for consistency.

<br>
* annotations/imageclef2015.dev.bbox.supplement.*.gz

  This file contains all bounding boxes from imageclef2015.dev.bbox, and additional<br>
  bounding boxes for additional 'general level' categories ('building', 'person',<br>
  'animal' etc.) not in the 251 concepts list. These special categories are prefixed<br>
  with an asterisk(*). We hope that you will find these useful for your development<br>
  purposes.

<br>
* annotations/imageclef2015.hierarchy.*.gz

  The hierarchy structure of the 'general level' categories. File format:

  *category \t *parent-category \t definition.

  For example, *mammal is the child of *animal. '#' represents the root node.

<br>
* annotations/imageclef2015.concept_to_parents.*.gz

  List of 'general level' category parent(s) for each 251 concept. A concept may<br>
  have multiple parents (separated by commas). File format:

  category \t *parent1,*parent2

<br>
* annotations/imageclef2015.test.groundtruth.zip

 

  Ground truth for the test.

 

 <br>
* webupv15_data_lists.zip

  -&gt; data_iids.txt : IDs of the images (IIDs) in the dataset.

  -&gt; data_rids.txt : IDs of the webpages (RIDs) in the dataset.

  -&gt; data_*urls.txt : The original URLs from where the images (iurls)<br>
       and the webpages (rurls) were downloaded. Each line in the file<br>
       corresponds to an image, starting with the IID and is followed<br>
       by one or more URLs.

  -&gt; data_rimgsrc.txt : The URLs of the images as referenced in each<br>
       of the webpages. Each line of the file is of the form: IID RID<br>
       URL1 [URL2 ...]. This information is necessary to locate the<br>
       images within the webpages and it can also be useful as a<br>
       textual feature.

    <br>
* feats_textual/webupv15_data_textual_pages.zip

  Contains all of the webpages which referenced the images in the<br>
  dataset set after being converted to valid xml. In total there are<br>
  515754 files, since each image can appear in more than one page, and<br>
  there can be several versions of same page which differ by the<br>
  method of conversion to xml. To avoid having too many files in a<br>
  single directory (which is an issue for some types of partitions),<br>
  the files are found in subdirectories named using the first two<br>
  characters of the RID, thus the paths of the files after extraction<br>
  are of the form:

    ./WEBUPV/pages/{RID:0:2}/{RID}.{CONVM}.xml.gz

  To be able to locate the images withing the webpages, the URLs of the<br>
  images as referenced are provided in the file data_rimgsrc.txt.

<br>
* feats_textual/webupv15_data_textual.scofeat.gz

  The processed text extracted from the webpages near where the images<br>
  appeared. Each line corresponds to one image, having the same order<br>
  as the data_iids.txt list. The lines start with the image ID,<br>
  followed by the number of extracted unique words and the<br>
  corresponding word-score pairs. The scores were derived taking into<br>
  account 1) the term frequency (TF), 2) the document object model<br>
  (DOM) attributes, and 3) the word distance to the image. The scores<br>
  are all integers and for each image the sum of scores is always<br>
  &lt;=100000 (i.e. it is normalized).

<br>
* feats_textual/webupv15_data_textual.keywords.gz

  The words used to find the images when querying image search<br>
  engines. Each line corresponds to an image (in the same order as in<br>
  data_iids.txt). The lines are composed of triplets:

    [keyword] [rank] [search_engine]

  where [keyword] is the word used to find the image, [rank] is the<br>
  position given to the image in the query, and [search_engine] is a<br>
  single character indicating in which search engine it was found<br>
  ('g':google, 'b':bing, 'y':yahoo).

<br>
* feats_visual/webupv15_data_visual_images.zip

  Contains thumbnails (maximum 640 pixels of either width or height)<br>
  of the images in jpeg format. To avoid having too many files in a<br>
  single directory (which is an issue for some types of partitions),<br>
  the files are found in subdirectories named using the first two<br>
  characters of the IID, thus the paths of the files after extraction<br>
  are of the form:

    ./WEBUPV/images/{IID:0:2}/{IID}.jpg

<br>
* feats_visual/webupv15_*.{s|d}feat.gz

  The visual features in a simple ASCII text format either in sparse<br>
  (*.sfeat.gz files) or dense (*.dfeat.gz files). The first<br>
  line of the file indicates the number of vectors (N) and the<br>
  dimensionality (DIMS). Then each line corresponds to one vector.<br>
  For the dense features each line has exactly DIMS values separated<br>
  by spaces, i.e., the format is:

    N DIMS<br>
    Val(1,1) Val(1,2) ... Val(1,DIMS)<br>
    Val(2,1) Val(1,2) ... Val(2,DIMS)<br>
    ...<br>
    Val(N,1) Val(N,2) ... Val(N,DIMS)

  For the sparse features, each line starts with the number of non-zero<br>
  elements and is followed by dimension-value pairs, being the first<br>
  dimension 0, i.e., the format is:

    N DIMS<br>
    nz1 Dim(1,1) Val(1,1) ... Dim(1,nz1) Val(1,nz1)<br>
    nz2 Dim(2,1) Val(2,1) ... Dim(2,nz2) Val(2,nz2)<br>
    ...<br>
    nzN Dim(N,1) Val(N,1) ... Dim(N,nzN) Val(N,nzN)

  The order of the features is the same as in the list data_iids.txt.

  The procedure to extract the SIFT based features in this<br>
  subdirectory was conducted as follows. Using the ImageMagick<br>
  software, the images were first rescaled to having a maximum of 240<br>
  pixels, of both width and height, while preserving the original<br>
  aspect ratio, employing the command:

    convert {IMGIN}.jpg -resize '240&gt;x240&gt;' {IMGOUT}.jpg

  Then the SIFT features where extracted using the ColorDescriptor<br>
  software from Koen van de Sande<br>
  (http://koen.me/research/colordescriptors). As configuration we<br>
  used, 'densesampling' detector with default parameters, and a hard<br>
  assignment codebook using a spatial pyramid as<br>
  'pyramid-1x1-2x2'. The number in the file name indicates the size of<br>
  the codebook. All of the vectors of the spatial pyramid are given in<br>
  the same line, thus keeping only the first 1/5th of the dimensions<br>
  would be like not using the spatial pyramid. The codebook was<br>
  generated using 1.25 million randomly selected features and the<br>
  k-means algorithm. The GIST features were extracted using the<br>
  LabelMe Toolbox. The images where first resized to 256x256 ignoring<br>
  original aspect ratio, using 5 scales, 6 orientations and 4<br>
  blocks. The other features colorhist and getlf, are both color<br>
  histogram based extracted using our own implementation.

<br>
* webupv15_data_visual_vgg16-relu7.dfeat.gz

  Contains the 4096 dimensional activations of the relu7 layer of Oxford<br>
  VGG’s 16-layer CNN model, extracted using the Berkeley Caffe library.<br>
  More details can be found at https://github.com/BVLC/caffe/wiki/Model-Zoo.

<br>
* webupv15_data_visual_vgg16-fc8.dfeat.gz

  Contains the 1000 dimensional activations of the fc8 layer of Oxford<br>
  VGG’s 16-layer CNN model, extracted using the Berkeley Caffe library.<br>
  More details can be found at https://github.com/BVLC/caffe/wiki/Model-Zoo.<br>
  The 1000 dimensions correspond to the 1000 categories in<br>
  webupv15_data_visual_vgg16-fc8_categories.lst: the higher the value<br>
  in the 1000d feature, the more likely the corresponding synset appears<br>
  in the image.

<br>
Contact<br>
-------

For further questions, please contact:<br>
  Mauricio Villegas &lt;mauvilsa@upv.es&gt;<br>
  Fei Yan &lt;f.yan@surrey.ac.uk&gt;