Visual Turing Test

The Visual Turing Test is “an operator-assisted device that produces a stochastic sequence of binary questions from a given test image”. The query engine produces a sequence of questions that have unpredictable answers given the history of questions. The test is only about vision and does not require any natural language processing. The job of the human operator is to provide the correct answer to the question or reject it as ambiguous. The query generator produces questions such that they follow a “natural story line”, similar to what humans do when they look at a picture.
Research in computer vision dates back to the 1960s when Seymour Papert first attempted to solve the problem. This unsuccessful attempt was referred to as the Summer Vision Project. The reason why it was not successful was because computer vision is more complicated than what people think. The complexity is in alignment with the human visual system. Roughly 50% of the human brain is devoted in processing vision, which indicates that it is a difficult problem.
Later there were attempts to solve the problems with models inspired by the human brain. Perceptrons by Frank Rosenblatt, which is a form of the neural networks, was one of the first such approaches. These simple neural networks could not live up to their expectations and had certain limitations due to which they were not considered in future research.
Later with the availability of the hardware and some processing power the research shifted to image processing which involves pixel-level operations, like finding edges, de-noising images or applying filters to name a few. There was some great progress in this field but the problem of vision which was to make the machines understand the images was still not being addressed. During this time the neural networks also resurfaced as it was shown that the limitations of the perceptrons can be overcome by Multi-layer perceptrons. Also in the early 1990s convolutional neural networks were born which showed great results on digit recognition but did not scale up well on harder problems.
The late 1990s and early 2000s saw the birth of modern computer vision. One of the reasons this happened was due to the availability of key, feature extraction and representation algorithms. Features along with the already present machine learning algorithms were used to detect, localise and segment objects in Images.
While all these advancements were being made, the community felt the need to have standardised datasets and evaluation metrics so the performances can be compared. This led to the emergence of challenges like the Pascal VOC challenge and the ImageNet challenge. The availability of standard evaluation metrics and the open challenges gave directions to the research. Better algorithms were introduced for specific tasks like object detection and classification.
Visual Turing Test aims to give a new direction to the computer vision research which would lead to the introduction of systems that will be one step closer to understanding images the way humans do.
A large number of datasets have been annotated and generalised to benchmark performances of difference classes of algorithms to assess different vision tasks (e.g., object detection/recognition) on some image domain (e.g., scene images).
One of the most famous datasets in computer vision is ImageNet which is used to assess the problem of object level Image classification. ImageNet is one of the largest annotated datasets available and has over one million images. The other important vision task is object detection and localisation which refers to detecting the object instance in the image and providing the bounding box coordinates around the object instance or segmenting the object. The most popular dataset for this task is the Pascal dataset. Similarly there are other datasets for specific tasks like the H3D dataset for human pose detection, Core dataset to evaluate the quality of detected object attributes such as colour, orientation, and activity.
Having these standard datasets has helped the vision community to come up with extremely well performing algorithms for all these tasks. The next logical step is to create a larger task encompassing of these smaller subtasks. Having such a task would lead to building systems that would understand images, as understanding images would inherently involve detecting objects, localising them and segmenting them.
The Visual Turing Test (VTT) unlike the Turing test has a query engine system which interrogates a computer vision system in the presence of a human co-ordinator.
It is a system that generates a random sequence of binary questions specific to the test image, such that the answer to any question k is unpredictable given the true answers to the previous k − 1 questions (also known as history of questions).
The test happens in the presence of a human operator who serves two main purposes: removing the ambiguous questions and providing the correct answers to the unambiguous questions. Given an Image infinite possible binary questions can be asked and a lot of them are bound to be ambiguous. These questions if generated by the query engine are removed by the human moderator and instead the query engine generates another question such that the answer to it is unpredictable given the history of the questions.
The aim of the Visual Turing Test is to evaluate the Image understanding of a computer system, and an important part of image understanding is the story line of the image. When humans look at an image, they do not think that there is a car at ‘x’ pixels from the left and ‘y’ pixels from the top, but instead they look at it as a story, for e.g. they might think that there is a car parked on the road, a person is exiting the car and heading towards a building. The most important elements of the story line are the objects and so to extract any story line from an image the first and the most important task is to instantiate the objects in it, and that is what the query engine does.
The query engine is the core of the Visual Turing Test and it comprises two main parts : Vocabulary and Questions
Vocabulary is a set of words that represent the elements of the images. This vocabulary when used with appropriate grammar leads to a set of questions. The grammar is defined in the next section in a way that it leads to a space of binary questions.
The vocabulary V {\displaystyle {\mathcal {V}}} consist of three components:
For Images of urban street scenes the types of objects include people, vehicle and buildings. Attributes refer to the properties of these objects, for e.g. female, child, wearing a hat or carrying something, for people and moving, parked, stopped, one tire visible or two tires visible for vehicles. Relationships between each pair of object classes can be either “ordered” or “unordered”. The unordered relationships may include talking, walking together and the ordered relationships include taller, closer to the camera, occluding, being occluded etc.
Additionally all of this vocabulary is used in context of rectangular image regions w \in W which allow for the localisation of objects in the image. An extremely large number of such regions are possible and this complicates the problem, so for this test, regions at specific scales are only used which include 1/16 the size of image, 1/4 the size of image, 1/2 the size of image or larger.
The question space is composed of four types of questions:
The uniqueness questions along with the existence questions form the instantiation questions. As mentioned earlier instantiating objects leads to other interesting questions and eventually a story line. Uniqueness questions follow the existence questions and a positive answer to it leads to instantiation of an object.
As mentioned before the core of the Visual Turing Test is the query generator which generates a sequence of binary questions such that the answer to any question k is unpredictable given the correct answers to the previous k − 1 questions. This is a recursive process, given a history of questions and their correct answers, the query generator either stops because there are no more unpredictable questions, or randomly selects an unpredictable question and adds it to the history.
The question space defined earlier implicitly imposes a constraint on the flow of the questions. To make it more clear this means that the attribute and relationship questions can not precede the instantiation questions. Only when the objects have been instantiated, can they be queried about their attributes and relations to other previously instantiated objects. Thus given a history we can restrict the possible questions that can follow it, and this set of questions are referred to as the candidate questions Q can {\displaystyle Q_{\text{can}}} .
The task is to choose an unpredictable question from these candidate questions such that it conforms with the question flow that we will describe in the next section. For this, find the unpredictability of every question among the candidate questions.
Let H {\displaystyle H} be a binary random variable, where H ( I ) = 1 {\displaystyle H(I)=1} , if the history H {\displaystyle H} is valid for the Image I {\displaystyle I} and 0 {\displaystyle 0} otherwise. Let q ∈ Q {\displaystyle q\in Q} can be the proposed question, and X q {\displaystyle X_{q}} be the answer to the question q {\displaystyle q} .
Then, find the conditional probability of getting the answer Xq to the question q given the history H.
Given this probability the measure of the unpredictability is given by:
The closer ρ H ( q ) {\displaystyle \rho _{H}(q)} is to 0, the more unpredictable the question is. ρ H ( q ) {\displaystyle \rho _{H}(q)} for every question is calculated. The questions for which ρ H ( q ) < ϵ {\displaystyle \rho _{H}(q)<\epsilon } , are the set of almost unpredictable questions and the next question is randomly picked from these.
As discussed in the previous section there is an implicit ordering in the question space, according to which the attribute questions come after the instantiation questions and the relationship questions come after the attribute questions, once multiple objects have been instantiated.
Therefore, the query engine follows a loop structure where it first instantiates an object with the existence and uniqueness questions, then queries about its attributes, and then the relationship questions are asked for that object with all the previously instantiated objects.
It is clear that the interesting questions about the attributes and the relations come after the instantiation questions, and so the query generator aims at instantiating as many objects as possible.
Instantiation questions are composed of both the existence and the uniqueness questions, but it is the uniqueness questions that actually instantiate an object if they get a positive response. So if the query generator has to randomly pick an instantiation question, it prefers to pick an unpredictable uniqueness question if present. If such a question is not present, the query generator picks an existence question such that it will lead to a uniqueness question with a high probability in the future. Thus the query generator performs a look-ahead search in this case.
An integral part of the ultimate aim of building systems that can understand images the way humans do, is the story line. Humans try to figure out a story line in the Image they see. The query generator achieves this by a continuity in the question sequences.
This means that once the object has been instantiated it tries to explore it in more details. Apart from finding its attributes and relation to the other objects, localisation is also an important step. Thus, as a next step the query generator tries to localise the object in the region it was first identified, so it restricts the set of instantiation questions to the regions within the original region.
Simplicity preference states that the query generator should pick simpler questions over the more complicated ones. Simpler questions are the ones that have fewer attributes in them. So this gives an ordering to the questions based on the number of attributes, and the query generator prefers the simpler ones.
To select the next question in the sequence, VTT has to estimate the predictability of every proposed question. This is done using the annotated training set of Images. Each Image is annotated with bounding box around the objects and labelled with the attributes, and pairs of objects are labelled with the relations.Consider each question type separately:
Detailed example sequences can be found here.
The Images considered for the Geman et al. work are that of ‘Urban street scenes’ dataset, which has scenes of streets from different cities across the world. This why the types of objects are constrained to people and vehicles for this experiment.
Another dataset introduced by the Max Planck Institute for Informatics is known as DAQUAR dataset which has real world images of indoor scenes. But they propose a different version of the visual Turing test which takes on a holistic approach and expects the participating system to exhibit human like common sense.
This is a very recent work published on March 9, 2015, in the journal Proceedings of the National Academy of Sciences, by researchers from Brown University and Johns Hopkins University. It evaluates how the computer vision systems understand the Images as compared to humans. Currently the test is written and the interrogator is a machine because having an oral evaluation by a human interrogator gives the humans an undue advantage of being subjective, and also expects real time answers.
The Visual Turing Test is expected to give a new direction to the computer vision research. Companies like Google and Facebook are investing millions of dollars into computer vision research, and are trying to build systems that closely resemble the human visual system. Recently Facebook announced its new platform M, which looks at an image and provides a description of it to help the visually impaired. Such systems might be able to perform well on the VTT.