SlowGuess/NAACL_2022_OCR huggingface dataset:
Zechen Li
Northeastern University, US
li.zec@northeastern.edu
Anders Søgaard
soegaard@di.ku.dk
Synthetic datasets have successfully been used to probe visual question-answering datasets for their reasoning abilities. CLEVR (Johnson et al., 2017), for example, tests a range of visual reasoning abilities. The questions in CLEVR focus on comparisons of shapes, colors, and sizes, numerical reasoning, and existence claims. This paper introduces a minimally biased, diagnostic visual question-answering dataset, QLEVR, that goes beyond existential and numerical quantification and focus on more complex quantifiers and their combinations, e.g., asking whether there are more than two red balls that are smaller than at least three blue balls in an image. We describe how the dataset was created and present a first evaluation of state-of-the-art visual question-answering models, showing that QLEVR presents a formidable challenge to our current models. Code and Dataset are available at https://github.com/zechenli03/QLEVR
Visual question answering is at the locus of computer vision and natural language processing, and its objective is developing computer vision systems that can answer arbitrary natural language questions about images (Lu et al., 2016; Schwartz et al., 2017; Ramakrishnan et al., 2018; Gat et al., 2020). This is useful across a range of applications, including medical image analysis, accessibility for visually impaired, video surveillance, art and advertisement (Barra et al., 2021).
The complexity of visual question answering naturally depends on the complexity of the images and the complexity of the natural language questions. The task reduces to object recognition for very simple questions of the form:
(1) Is there a triangle in this image?
Question: Are all the cyan metallic triangular prisms on the brown plane?
Answer: True
Question: On the non-white planes on the left rear side of the black wood rectangular plane, all the cyan metallic cubes but at least 2 are larger than at most 7 cubes; is it right?
Answer: False
Figure 1: A sample image and questions from QLEVR. Tasks involve attribute recognition, counting, comparing numbers, spatial relationships, and understanding of quantifiers.
Object recognition can of course be a very complex task on its own, depending on the types of objects, the number of possible objects to be recognized, the amount of supervision for inducing a good model, general image quality, etc. However, more complex queries such as (2) make visual question answering much harder:
(2) Is there a triangle inside a circle in this image?
Answering such a question in the presence of an image requires a computer vision system that not only recognizes objects, but also relations between them. CLEVR (Johnson et al., 2017) probes computer vision systems's ability to answer even more complex queries, such as, for instance:
(3) Is there a cyan cube to the right of the yellow sphere?
Findings of the Association for Computational Linguistics: NAACL 2022, pages 980 - 996
Question (3) involves reasoning about the relation between two objects, as well as the compositional semantics of color adjectives. In addition to shapes and colors, CLEVR also includes questions about sizes and quantities.
In this paper, we present a novel visual question-answering dataset that goes beyond CLEVR in focusing specifically on quantificational language, e.g.:
(4) Are most of the cyan cubes to the right of the yellow sphere?
Given the complexity of quantificational language, the rich typology of expressions of quantification across different languages, and the interest from philosophy, it is perhaps surprising that quantificational language has received relatively little attention in the NLP community (see §2), but we believe it is a crucial step in pushing the research horizons in (visual) question-answering.
Contributions Based on a comprehensive typology of English quantifiers, we build a dataset of 100,000 synthetic images and 999,446 unique questions to these images. This is roughly the same size as or a little bigger than CLEVR (Johnson et al., 2017). Our questions are on average longer than previous datasets. We evaluate three baselines from Johnson et al. (2017), a text-only baseline based on BERT (Devlin et al., 2019), and MAC (Hudson and Manning, 2018) on QLEVR and analyze performance across quantifier types.
Visual Question Answering Challenge Datasets
Several synthetic challenge datasets for visual question answering exist: Andreas et al. (2016) presents SHAPES, a predecessor to CLEVR and QLEVR, relying also on synthetic constellations of colored geometric shapes and template-driven question generation. Pezzelle and Fernandez (2019) create a similar dataset to probe visual question answering models for knowledge of adjectival semantics. A portion of the visual question answering dataset (Agrawal et al., 2017) contains synthetic cartoon imagery. Sampat et al. (2021) present an extension of CLEVR that probes for hypothetical reasoning of the form: If someone removed three triangles from this image, how many would be left? Malinowski and Fritz (2014) combined natural images with synthetic, template-driven question generation.
Finally, Parfenova et al. (2021) recently created a dataset of three-image scenes to probe two-step reasoning.
Synthetic visual question answering datasets have several advantages over ones based on real images and questions that tend to suffer from selection biases (Liu et al., 2021), but of course they are limited in what can be induced from them. They are therefore mostly useful for probing the limitations of visual question answering architectures and off-the-shelf models. Showing results only on synthetic data is often seen as a weakness in the literature (Hassantabar, 2018), but synthetic data is useful for diagnosing the errors of visual question answering systems, in our case highlighting the challenges posed by quantifiers.
Quantifiers Quantifiers have been largely ignored in the NLP community. Question-answering datasets have been developed for numerical reasoning in English (Dua et al., 2019), and some have identified quantifier words as important sources of errors for textual entailment systems (Joshi et al., 2020). Fang and Lou (2021) recently focused on the two quantifier words part and whole in an error analysis for named entity recognition.
3 QLEVR
We design a challenge dataset called QLEVR (for Quantificational Language and Elementary Visual Reasoning) that requires more complex reasoning than previous visual question-answering datasets. QLEVR is designed to probe the visual reasoning capabilities of visual question-answering systems with respect to quantificational language, including detecting members of sets, quantifying sets, and reasoning about the relationships between sets. To this end, we automatically construct scene graphs (Johnson et al., 2015) and use these to generate synthetic images with ground-truth locations, attributes, and relationships for planes and objects. Each scene graph can be queried in a number of way, and we design query templates to render natural language questions involving complex reasoning about sets of such planes and objects. We describe each of these steps in detail:
Image Generation All images in QLEVR are images of objects organized in a particular way on a desk-like surface. Figure 2 shows how the images are generated. We construct a scene graph for a two-dimensional image containing areas and ob
Q: On the planes where there are not exactly 3 tiny objects on each plane, are there at least 6 gray metallic objects to the left front of the gray dappled triangle-based pyramid?
Figure 2: An overview of our dataset. Top: Image generation process and bounding box information. The top two-dimensional image records the scene graph, and the bottom gray-scale color map records roughness of each plane. Center: Examples of questions and their associated operators. Bottom: Ideal visual attention as the operator proceeds.
jects of different sizes and shapes. Scene graphs determine the ground-truth locations, bounding boxes, attributes and relationships for the planes and objects in the form of a graph or tree structure. Nodes are planes or objects annotated with attributes, each of which is connected to its spatially related nodes.
Each image contains one to five areas or geometric planes. These can be either triangular, rectangular or circular. The rest of the desk area we refer to as the white non-geometric plane. Geometric planes come in two materials (marble and wood), three colors (black, gray, and brown), and random sizes.
Each geometric plane contains one to ten (1-10) objects, with different sizes and shapes, and the non-geometric plane contains one to twelve (1-12) objects, with different sizes and shapes. Object come in seven shapes (cone, cube, cylinder, pentahedron, sphere, triangular prism, and tetrahedron),
two absolute sizes (small and large), five materials (metal, rubber, leather, marble, and wood), and eight colors (blue, brown, cyan, gray, green, purple, red and yellow). The spatial relationships between planes and objects include front, back, left and right, as well as right front, right rear, left front and left rear.
We render three-dimensional images of the scene graphs with Blender (Community, 2018). Light settings and three preset camera positions were chosen at random, after validating that all objects were at least partially visible. Since the depth of the scene can affect the judgment of the spatial relationship in the three-dimensional image, the desk boundary is always visible as a reference for determining the depth of the scene. Minimum distances between objects and planes were kept to reduce the ambiguity of spatial relationships. See Appendix B for more details.
Question Generation Quantifiers are often said to be among the most important and complex constructs of natural languages (Hintikka, 1977; Barwise and Cooper, 1981). As pointed out by by Bernardi and Pezzelle (2021), visual question-answering models need to master a wide range of linguistic phenomena, including negation, entailment, mutual exclusivity and so on. We add (generalized) quantifiers to this list and design a dataset to probe the ability of visual question-answering systems to handle quantifiers in combination with other linguistic phenomena. See Table 1 for the quantifiers included in QLEVR.
See Figure 2 for how questions are formed from scene graphs. In brief, we think of the scene graph as a model and evaluate various combinations of logical operators, including quantifiers, on the scene graph, i.e., performing a model checking (Clarke et al., 2009) procedure.
We introduce the notion of a question family, defined by a set of operators and a scene graph. Each question family is associated with 2-6 text templates and a set of synonyms (for shapes, colors, materials, and spatial relationships). The templates were written by hand. Each question template can thus generate multiple questions. For example, the
Table 1: Quantifiers included in QLEVR.  $P$  denotes the set of all the objects on the target plane(s).  $A$  or  $B$  denotes a subset of  $P$  with the same attributes.  $|A|$  is the cardinality of the set  $A$ .
Question: On the geometric plane whose color is different from that of other planes, all yellow spheres but 1 are smaller than fewer than 1 or more than 3 spheres; is it right?
Answer: True Quantifiers: exactly N (all but  $\mathbf{N}\neg$  ), between  $(\neg$  between)
Figure 3: Example image-question pair
(5) Are there exactly  $<\mathrm{OC}> <\mathrm{Z}> <\mathrm{C}> <\mathrm{M}> <\mathrm{S}> <\mathrm{os}>$  on the plane  $<\mathrm{ps}>?$
where upper-cased variables refer to words, and lower-cased variables to suffixes, can generate the question
(6) Are there exactly 2 small red rubber objects on the black wooden triangular plane?
We construct a total of 671 different templates, which are randomly constructed from 11 plane templates and 61 object templates. Our questions involve attribute recognition, counting, comparing numbers or attributes, spatial relationships, and understanding of quantifiers. Figure 2 shows the operators built in the given question family, such as filter, relate, and at least.
Note that many (generalized) quantifiers are related by entailment. The question
(7) Are all the red cubes on the marble planes?
is, assuming an image with red cubes, semantically equivalent to
(8) Are no red cubes not on the marble planes?
The semantics of combinations of quantifiers can be derived using squares of opposition (Westerståhl, 2012). We exploit these entailment relations in creating QLEVR.
Split	Images	Questions	Unique questions	Overlap with train	Overlap with val
Train	70,000	700,000	699,498	-	-
Val	15,000	150,000	149,968	199 | 148	-
Test	15,000	150,000	149,980	194 | 145	49 | 39
Total	100,000	1,000,000	999,446	-	-
Table 2: Statistics for our dataset. In each Overlap column, the number on the right represents the number of overlapping questions with the same answer.
Some combinations of key values may generate unreasonable questions. We therefore define restrictions for each question family to avoid the generation of pragmatically odd, ill-posed or trivial questions. For example, the phrase on the marble plane where there are at least 5 red objects would be pragmatically odd if there was only one marble plane in the scene. The sentence
(9) On the marble plane, do between 2 and 4 cubes have the same size as most of the cylinders?
is ill-posed if there are no cubes on the marble plane. Finally, questions like Are there more red cubes than cubes? are trivial, because they can be answered in the absence of the image. The assertion is always true. The opposite would, for example, be true of
(10) On the plane with 8 balls, are there exactly 3 balls?
We present many examples of images and questions in the Appendix, but see also Figure 3 for a complex question with embedded quantifiers.
Dataset Characteristics QLEVR has 1,000,000 questions for 100,000 images, with each image having 10 questions generated from different question families. The dataset is balanced, preventing answering in the absence of the images. In addition, the answer distribution across question families is constrained by acceptance-rejection sampling. The data is randomly split, with  $70\%$  for training data,  $15\%$  for validation and  $15\%$  for heldout evaluation data (the test set). As shown in figure 4, QLEVR includes 27 different quantifiers. Questions contain 1-4 quantifiers. Table 2 shows the diversity and complexity of the QLEVR questions. Almost all the questions are unique. Very few questions appear in several splits, and always in conjunction with new scene graphs.
In this section, we evaluate the performance of baselines and near-state-of-the-art models on the QLEVR dataset and perform detailed error analysis. We ran each method three times with different random seeds and report the test set performance for the model that achieved the best performance on the validation data.
We first present three purely text-based models, Q-type (Antol et al., 2015), LSTM (Hochreiter and Schmidhuber, 1997) and BERT (Devlin et al., 2019), to evaluate the level of visual reasoning needed for QLEVR. If these perform at random (0.5), we have successfully constructed a dataset in which questions cannot be answered in the absence of images. It is important to include text-only models as baselines in visual question answering to control for spurious correlations (Gat et al., 2020). We shall see in §4.2 that while Q-type performs at chance level, the BERT and LSTM baselines are able to pick up on some spurious correlations. We also evaluate two standard visual question answering architectures, one based on a combination of convolutional and recurrent neural networks (CNN+LSTM), and one attention-based architecture (Hudson and Manning, 2018). The latter performs best on the counting and number comparison tasks in the CLEVR dataset (Johnson et al., 2017) compared with other approaches, such as Bottom-Up-Attention and Top-Down (UpDn) (Anderson et al., 2018), Question-Conditioned Graph (QCG) (Norcliffe-Brown et al., 2018), Bilinear Attention Network (BAN) (Kim et al., 2018), Relation Network (RN) (Santoro et al., 2017) and Recurrent Aggregation of Multimodal Embeddings Network (RAMEN) (Shrestha et al., 2019). We describe each system in detail:
- Q-type (Antol et al., 2015): Similar to the "per Q-type prior" method in (Antol et al., 2015), this baseline predicts the most popular answer for each question type.
- LSTM (Hochreiter and Schmidhuber, 1997): Question words are embedded as 300-dimensional vector sand fed into an LSTM network. The last hidden state representation is passed into a multi-layer perceptron (MLP) to predict the final answer. All experiments use
Figure 4: Statistics for our dataset. Left: Question length distribution for different popular VQA datasets; most of the QLEVR questions have 30 to 40 words, which is longer than other datasets. Middle: Distribution of the number of quantifiers in QLEVR questions. Right: Frequency distribution of quantifiers in QLEVR, where  $N$  stands for Number and  $F$  stands for Fraction; each, total, no, at most  $N$ , at least  $N$ , exactly  $N$ , between, not between, and not exactly  $N$  are also used in the plane templates, so they appear more frequently. For the quantifiers in the texts of the same question family, we consider each quantifier in square of opposition  $\{Q, Q^{\neg}, \neg Q, Q^{d}\}$  as quantifier  $Q$ .
a bi-directional LSTM with 512 units in the hidden layer per direction.
- BERT (Devlin et al., 2019): We fine-tune BERT (Devlin et al., 2019) augmented with a sentence-level classification head: The special classification token [CLS] is passed to a feed-forward layer and used for sentence class prediction.
- CNN+LSTM: The images are encoded using a convolutional neural network and questions as the last hidden state produced by an LSTM network. The convolutional network uses spatial features produced by ResNet-101 (He et al., 2015) pre-trained on ImageNet (Deng et al., 2009). We resize all images to  $448 \times 448$ , and use the final average pooling layer to extracts features of the shape (1, 14, 14, 2048). The question and image features are concatenated and passed to a multi-layered perceptron to predict the final answer.
- MAC (Hudson and Manning, 2018): The MAC network is a recurrent attention network, which uses a Memory, Attention, and Composition (MAC) cell in each attention-based reasoning step to learn to perform iterative reasoning processes. MAC learns compositional reasoning directly from the questions and the images in an end-to-end approach. The word vectors have a dimension of 300 and are initialized randomly using a standard uniform distribution. The images are resized to  $448 \times 448$ , and 2048-dimensional features are produced by ResNet-101. The model uses a hidden state size of 512 and a length of 12 MAC cells.
4.2 Analysis by Quantifier Type
Table 3 shows the results of the five methods described in §4.1 on the test set of QLEVR. We make the following observations.
1. Q-type exhibits performance levels around  $50\%$  for every quantifier type, showing that the answer distribution of QLEVR is uniform.
2. Text-only LSTM and BERT achieve an average accuracies of  $64.6\%$  and  $65.8\%$ , respectively. These results suggest that even if the answers of each question family are distributed uniformly, there may still be spurious correlations: Objects with more detailed attribute descriptions may be more likely to get a false answer. For example, the question "Are there more than 3 small blue cubes on the black planes?" is more likely to get a false answer than "Are there more than 3 blue objects on the black planes?").
3. The CNN+LSTM architecture performs better than LSTM on 24 out of 27 quantifier types and on par with BERT; MAC performs better than LSTM on 26 out of 27 quantifier types, better than BERT on 24 out of 27 quantifier types and better than CNN+LSTM on 23 out of 27 quantifier types. In general, however, CNN+LSTM and MAC do not improve much over text-only LSTM and BERT, suggesting that the image features extracted by ResNet-101 contain little information relevant to counting in complex scenes.
4. Accuracies for quantifiers that present thresholds (e.g., more than  $N$ , at least  $F$ , etc.) are
Q-type	LSTM	BERT	CNN+LSTM	MAC
each	50.0	63.9	65.4	65.3	66.2
total	50.0	63.4	64.7	65.1	66.3
all	50.0	59.5	60.5	60.7	61.3
most	50.0	61.7	63.6	63.5	64.0
not all	50.0	60.7	62.2	61.3	62.6
no	50.0	63.9	64.6	64.6	65.3
some	50.0	58.2	58.9	58.7	58.7
some but not all	50.0	59.9	61.5	61.0	61.7
exactly N	50.0	62.6	63.8	63.5	64.0
not exactly N	50.0	64.9	65.7	65.9	66.7
between	50.0	65.2	66.6	67.0	67.9
not between	50.0	64.4	65.8	66.0	66.0
all but at most	50.0	66.9	68.8	68.2	68.9
all but at least	50.1	62.7	63.7	63.3	65.2
Q-type	LSTM	BERT	CNN+LSTM	MAC
more than N	50.0	65.5	67.3	67.1	67.2
at least N	50.0	64.7	66.2	65.8	66.5
fewer than N	50.0	66.7	67.6	67.7	68.3
at most N	50.0	64.3	65.4	64.9	65.7
more than F	50.0	69.0	69.6	71.4	71.4
at least F	50.0	67.9	70.1	71.0	72.0
fewer than F	50.0	67.6	68.8	70.2	71.6
at most F	50.0	65.8	68.9	70.3	70.8
every _ except _	50.1	70.9	72.0	70.9	72.1
no _ except _	50.1	78.4	78.1	77.9	78.2
more _ than _	50.0	68.0	69.4	69.1	68.7
fewer _ than _	50.0	68.9	69.9	68.8	69.0
equal _ and _	50.0	61.0	62.2	62.0	62.2
Overall	50.0	64.6	65.8	65.9	66.5
Table 3: Test set results of baselines and state-of-the-art models on the QLEVR dataset. Models are evaluated for both overall accuracy and accuracy per quantifier type. In quantifier type,  $N$  stands for Number and  $F$  stands for Fraction. Refer to Figure 4 for the number distribution of quantifiers.
Figure 5: The effect of different number of quantifiers in questions on the accuracy of the answers. Figure 4 shows the distribution of the number of quantifiers in each QLEVR question.
higher than for quantifiers that require a number of objects to match exact values (e.g., exactly  $N$ ).
5. Quantifiers without numerals (e.g., all, most, not all, some and some but not all) lead to lower accuracies than other quantifiers, showing that reasoning with these quantifiers is harder. This highlights the need for including such quantifiers in challenge datasets to push advancements in visual question answering.
Number of Quantifiers in Questions Figure 5 shows how accuracy varies as the number of quantifiers in the questions increases. The more quantifiers in a question, the more complex its semantics will be.
Number of Planes We also test the visual reasoning abilities of these models by examining error across the number of planes involved in answering the question. Appendix A introduces all the plane templates in our question families. We use the plane template "on the  $<PC>$ $<PM>$ $<PS>$  plane" for our analysis, because this template has no influence of quantifiers or spatial relationships in targeting planes. QLEVR test set has 13,612 questions with this plane template. The left graph in Figure 6 shows how the accuracy varies with the increase in the number of target planes that need to be reasoned with. Among the 13,612 questions, 10,288 of them involve a single plane and 3,324 of them involve multiple planes. We can see that for language-only models Q-type, BERT and LSTM, the number of target planes does not significantly affect the accuracy. However, for CNN+LSTM and MAC, questions involving just a single plane are harder to answer than those involving multiple planes. This is because for visual models, planes enable disambiguation and thereby reduce the required reasoning. The right graph in Figure 6 compares accuracy on questions that do not refer to specific planes (no attribute), to questions that refer to specific planes (with attributes). Among the 13,612 questions, 1,340 questions do not refer to specific planes, whereas 12,272 do. This distinction has little impact on the perfor