page_title: Visual Turing Test
section_title: Vocabulary
hierarchical_section_title: Visual Turing Test / Details / Query engine / Vocabulary
caption_attribution_description: English: Sample regions used as context in a Visual Turing Test. The one on the left shows regions with 1/8th the size of the image and the one on the right show regions with 1/4th size of the image
context_page_description: Computer vision research is driven by standard evaluation practices. The current systems are tested by their accuracy for tasks like object detection, segmentation and localization. Methods like the convolutional neural networks seem to be doing pretty well in these tasks, but the current systems are still not close to solving the ultimate problem of understanding images the way humans do. So motivated by the ability of humans to understand an image and even tell a story about it, Geman et al. have introduced the Visual Turing Test for computer vision systems.
  As described in, it is “an operator-assisted device that produces a stochastic sequence of binary questions from a given test image”. The query engine produces a sequence of questions that have unpredictable answers given the history of questions. The test is only about vision and does not require any natural language processing. The job of the human operator is to provide the correct answer to the question or reject it as ambiguous. The query generator produces questions such that they follow a “natural story line”, similar to what humans do when they look at a picture.
context_section_description: Vocabulary is a set of words that represent the elements of the images. This vocabulary when used with appropriate grammar leads to a set of questions. The grammar is defined in the next section in a way that it leads to a space of binary questions.
  The vocabulary  consist of three components:
  Types of Objects 
  Type-dependent attributes of objects 
  Type-dependent relationships between two objects 
  For Images of urban street scenes the types of objects include people, vehicle and buildings. Attributes refer to the properties of these objects, for e.g. female, child, wearing a hat or carrying something, for people and moving, parked, stopped, one tire visible or two tires visible for vehicles. Relationships between each pair of object classes can be either “ordered” or “unordered”. The unordered relationships may include talking, walking together and the ordered relationships include taller, closer to the camera, occluding, being occluded etc.
  Additionally all of this vocabulary is used in context of rectangular image regions w \in W which allow for the localisation of objects in the image. An extremely large number of such regions are possible and this complicates the problem, so for this test, regions at specific scales are only used which include 1/16 the size of image, 1/4 the size of image, 1/2 the size of image or larger.
caption: Sample regions used as context in a Visual Turing Test. The one on the left shows regions with 1/8 the size of the image and the one on the right show regions with 1/4 size of the image
exif: {}