SlowGuess/NAACL_2022_OCR huggingface dataset:
Figure 6: The results of questions contains "on the  $<PC>$ $<PM>$ $<PS>$  plane". Left: The effect of different number of target planes on the accuracy of the answers; single means that the reasoning process basically only needs to consider one plane in the image, while multiple means that multiple planes need to be considered. Right: The effect of whether the plane has attribute description on the accuracy of the answer; no attribute ("on the planes") means that the reasoning process does not need to consider planes in the image and see the image as a whole, while with attributes ("e.g., on the wooden plane") means that specific plane(s) needs to be considered.
mance of our text-only models. For CNN+LSTM and MAC, however, examples in the no attribute class exhibit higher accuracies than those in with attributes. This, again, shows performance is better when less visual reasoning is required.
In this paper, we proposed a dataset, which we call QLEVR – for Quantificational Language and Elementary Visual Reasoning. QLEVR probes the ability of visual question-answering systems to reason with quantificational language, including 27 different quantifiers and combinations thereof. It requires complex visual reasoning to locate the specific planes and understand various relationships between objects. We increase the semantic diversity of the questions by negating quantifiers and by using different templates for semantically equivalent questions. Our analysis highlights how challenging such examples are to visual question-answering systems, and we hope that QLEVR will help guide push research horizons in visual question-answering by zooming in on the challenges posed by quantificational language.
One fundamental limitation is that QLEVR only considers English questions, and we plan to extent it to other, typologically unrelated languages. Besides, QLEVR can easily be extended by adding new question families, and questions whose answers are not limited to true or false, e.g., with numbers or attributes as answer types. In addition to the three-dimensional images, we also provide two-dimensional images and scene graphs recording the ground-truth information (see Figure 2). It is also possible to generate questions about 2D im
ages by simply modifying our question families. We hope these two datasets can be used for transfer learning for visual question answering in the future.
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2017. Vqa: Visual question answering. Int. J. Comput. Vision, 123(1):4-31.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39-48.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV).
Silvio Barra, Carmen Bisogni, Maria De Marsico, and Stefano Ricciardi. 2021. Visual question answering: which investigated applications? Pattern Recognit. Lett., 151:325-331.
Jon Barwise and Robin Cooper. 1981. Generalized quantifiers and natural language. Linguistics and Philosophy, 4(2):159-219.
Raffaella Bernardi and Sandro Pezzelle. 2021. Linguistic issues behind visual question answering. Language and Linguistics Compass, 15(6):e12417.
Edmund M. Clarke, E. Allen Emerson, and Joseph Sifakis. 2009. Model checking: Algorithmic verification and debugging. Commun. ACM, 52(11):74-84.
Blender Online Community. 2018. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Lei Fang and Jian-Guang Lou. 2021. Part & whole extraction: Towards a deep understanding of quantitative facts for percentages in text. ArXiv, abs/2110.13505.
Itai Gat, Idan Schwartz, Alexander Schwing, and Tamir Hazan. 2020. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies. In Advances in Neural Information Processing Systems, volume 33, pages 3197-3208. Curran Associates, Inc.
Shayan Hassantabar. 2018. Visual question answering: Datasets, methods, challenges and opportunities.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition.
Jaakko Hintikka. 1977. Quantifiers in natural languages: Some logical problems ii. Linguistics and Philosophy, 1(2):153-172.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation.
Drew A Hudson and Christopher D Manning. 2018. Compositional attention networks for machine reasoning.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. 2015. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668-3678.
Pratik Joshi, Somak Aditya, Aalok Sathe, and Monojit Choudhury. 2020. TaxiNLI: Taking a ride up the NLU hill. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 41-55, Online. Association for Computational Linguistics.
Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018. Bilinear Attention Networks. In Advances in Neural Information Processing Systems 31, pages 1571-1581.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Eliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Mateusz Malinowski and Mario Fritz. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In NeurIPS, pages 1682-1690.
Will Norcliffe-Brown, Efstathios Vafeias, and Sarah Parisot. 2018. Learning conditioned graph structures for interpretable visual question answering. arXiv preprint arXiv:1806.07243.
Iuliia Parfenova, Desmond Elliott, Raquel Fernandez, and Sandro Pezzelle. 2021. Probing cross-modal representations in multi-step relational reasoning. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP-2021), pages 152-162, Online. Association for Computational Linguistics.
Sandro Pezzelle and Raquel Fernández. 2019. Is the red square big? MALeViC: Modeling adjectives leveraging visual contexts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2865-2876, Hong Kong, China. Association for Computational Linguistics.
Sainandan Ramakrishnan, Aishwarya Agrawal, and Stefan Lee. 2018. Overcoming language priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Shailaja Keyur Sampat, Akshay Kumar, Yezhou Yang, and Chitta Baral. 2021. CLEVR_HYP: A challenge dataset and baselines for visual question answering with hypothetical actions over images. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, pages 3692-3709, Online. Association for Computational Linguistics.
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Timothy P. Lillicrap. 2017. A simple neural network module for relational reasoning. In NIPS.
Idan Schwartz, Alexander Schwing, and Tamir Hazan. 2017. High-order attention models for visual question answering. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Robik Shrestha, Kushal Kafle, and Christopher Kanan. 2019. Answer them all! toward universal visual question answering models. In CVPR.
Dag Westerstahl. 2012. Classical vs. modern squares of opposition, and beyond.
A Question Templates
As described in Section 3, QLEVR question templates are composed of 11 plane templates and 61 object templates randomly paired. In this section we detail the difference between these templates.
Plane Templates. The role of the plane templates is to raise our question for specific planes (regions) in the image through some restrictions (attributes, spatial relations and explicitly restricted quantifier phrases). Basically, the plane templates can generate questions with following types:
- On the white non-geometric planes.
- On the geometric plane with a different shape (color/material) from other planes.
- On the black planes to the left rear of the circular plane.
- On the planes where there are at least 3 red cubes on each plane.
- On the quadrilateral plane where there are at most 5 blue balls.
- On the brown planes where there are between 1 and 4 triangular prisms on each plane.
- On the triangular plane where there is exactly 1 leathery object.
- On the plane where there are not 2 to 4 triangular prisms.
- On the gray planes where there are not exactly 3 items on each plane.
- On the marble plane where there are not any wooden cones.
- On the wooden plane where there is a total of 7 small rubber objects.
To avoid pragmatically odd questions, we ensure that the number of planes obtained by the plane templates with restrictions of spatial relations and explicitly restricted quantifier phrases (e.g. On the brown planes behind the gray plane, or On the brown plane where there are exactly 3 balls) is less than the number of planes obtained by the templates without these restrictions (e.g. On the brown planes) for the same scene graph.
Object Templates. We can use the operators representation of the questions templates to analyze model performance on the following forms of reasoning:
Figure 7: Accuracy per question type on the QLEVR dataset.
- Existence type 1: Questions ask whether a certain type of quantifier-restricted object exists on one or some specific planes (e.g., "Whether all the cyan cubes [Plane Template]?").
- Existence type 2: Questions ask whether a certain type of quantifier-restricted object exists in a certain direction of a unique object (e.g., "[Plane Template], are there fewer than 3 balls behind the cyan cube?").
- Comparing attributes: Questions ask whether two types of quantifier-restricted objects have the same value for some attributes (e.g., "[Plane Template], is there any small cylinders that has the same color as most leathery tetrahedrons?").
- Quantity comparison: Questions compare the size of two sets of objects (e.g., "[Plane Template], are there more big blocks than rubber balls?").
- Size comparison: Questions ask which of two quantifier-restricted objects has a larger size (e.g., "[Plane Template], some red cones are larger than some but not all of the metal cones; is it right?").
- Spatial relations: Questions involves the spatial relationship between objects (e.g., "[Plane Template], are there more big blocks in front of the yellow cylinder than rubber balls to the left rear of the small block?").
Figure 7 shows the performance on above question types. As can be seen, MAC outperforms
other models on most question types. The only exception is: on quantity comparison task, BERT performs slightly better than MAC, showing that MAC has better reasoning ability in complex scenes. Questions of Existence type 1 obtain better results than Existence type 2 for vision-language model CNN+LSTM and MAC, suggesting that the position relationship between object and plane is easier to be inferred by the models than the spatial relationship between the objects. For questions of Quantity comparison, MAC and CNN+LSTM performs on par with LSTM, suggesting that the image features extracted by ResNet-101 may contain little information related to counting in complex scenes.
B 3D Modeling and Design
Figure 8 shows the materials and object models made through Blender (Community, 2018), as well as the performance of different colors on these materials. Two different materials of leather, marble, and wood were made respectively to further enrich the diversity of objects in the dataset. The images of the plane materials were made by modifying the images under CC0 1.0 Universal. $^{1}$  Note that after the overall scene rendering, objects of certain materials will produce different effects according to the color and material of the plane in contact with them, as well as the position of the camera and lights.
C Example Images and Questions
The remaining pages show some images and questions generated by the combination of our different plane templates and object templates. Each question is annotated with its answer and contained quantifiers, where  $N$  stands for Number,  $F$  stands for Fraction and  $O$  stands for Object.
(a) object shapes
(b) materials and colors of the planes
(c) metal
(d) rubber
(e) leather
(f) marble
(g) wood
Figure 8: From left to right, the object shapes in (a) are cone, cube, cylinder, pentahedron, sphere, triangular prism, and tetrahedron; the plane attributes in (b) are black marble, black wood, brown marble, brown wood, gray marble and gray wood; the colors in (c)  $\sim$  (g) are blue, brown, cyan, gray, green, purple, red and yellow.
$^{1}$  Creative Commons - CC0 1.0 Universal
Question: Whether all the large brown objects are on the white plane?
Quantifiers: all
Question: Some large rubber tetrahedron is not on the gray marble plane; is it right?
Quantifiers: not all (some  $\neg$ )
Question: It is not the case that all the big blue rubbery spheres are not on the gray rectangular plane; is it right?
Quantifiers: some  $(\neg \text{all} \neg)$
Question: All the big wooden blocks but at least 2 are not on the planes where there are exactly 2 cylinders on each plane; is it right?
Quantifiers: each, exactly N, at least N (all but at least  $\mathbf{N}\neg$
Question: It is not the case that at most 2 wood cylinders are on the quadrilateral plane where there are exactly 2 purple wood cylinders; is it right?
Quantifiers: each, exactly N, more than N (at most N)
Question: Are there fewer than 2 small purple dappled cylinders on the planes where there is exactly 1 purple block on each plane?
Quantifiers: each, exactly N, fewer than N
Question: It's not the case that some large purple metallic triangular prism is on the planes where there are 9 items in total; is it right?
Quantifiers: total, no (  $\neg$  some)
Question: Whether some but not all of the large purple rubber objects are on the marble planes where there are 4 blue objects in total?
Quantifiers: total, some but not all
Question: Are there at most 3 small blue objects on the dappled planes where there are 5 big triangular prisms in total?
Quantifiers: total, at most N
Question: All the tiny red wood objects but 1 are not on the non-white plane where the shape of the plane is different from that of other planes; is it right?
Quantifiers: exactly N (all but  $\mathbf{N}\neg$
Question: Are there between 1 and 3 small red leathery cubes on the circular plane to the left rear of the brown quadrilateral plane?
Quantifiers: between
Question: All the red leather objects but at most 3 are on the geometric plane where the material of the plane is different from that of other planes; is it right?
Quantifiers: all but at most N
Question: More than two thirds of the green objects are on the brown marble plane to the left front of the black wood plane; is it right?
Quantifiers: more than F
Question: Are most brown metallic objects on the geometric plane on the right side of the brown wooden round plane?
Quantifiers: most
Question: It is not the case that fewer than 4 brown metallic triangular prisms are not on the non-white plane where the color of the plane is different from that of other planes; is it right?
Quantifiers: all but at least  $\mathbf{N}$  ( $\neg$  fewer than  $\mathbf{N} \neg$ )
Question: On the gray marble plane where there are between 3 and 5 tiny cyan objects, is there any wooden triangular prism that has the same size as most metallic objects?
Quantifiers: between, some (any), most
Question: On the gray plane where there are between 1 and 4 cyan triangular prisms, 2 to 5 cyan items are the same material as most small items; is it right?
Quantifiers: between, between, most
Question: On the planes where there are between 3 and 6 cyan items on each plane, are there exactly 3 small triangular prisms that have the same color as most small metallic spheres?
Quantifiers: each, between, exactly N, most
Question: Fewer than three-quarters of the small yellow objects are on the wood three-sided plane where there are not any large metallic square-based pyramids; is it right?
Quantifiers: no (  $\neg$  any), fewer than F
Question: It is not the case that fewer than 11/15 of the metallic items are on the planes where there are 0 tiny rubbery tetrahedrons on each plane; is it right?
Quantifiers: each, no (0), at least F ( $\neg$  fewer than F)
Question: At most 7/8 of the yellow rubber triangular pyramids are on the planes where there is no big yellow wooden cone on each plane; is it right?
Quantifiers: each, no, at most F
Question: On the gray quadrilateral plane where there are not between 1 and 3 small gray items, are all the large items but at most 1 the same color as most leather items?
Quantifiers: not between, all but at most N, most
Question: On the plane where there are not between 2 and 4 green leather objects, fewer than a half of the gray cylinders are the same size as most gray rubbery objects; is it right?
Quantifiers: not between, fewer than F, most
Question: On the planes where there are not between 0 and 3 big objects on each plane, more than five twelfths of the big gray objects have the same shape as most gray rubber objects; is it right?
993Quantifiers: each, not between, more than F, most
Question: On the planes where there are not exactly 2 spheres on each plane, at most 4 red leather objects are the same shape as most objects; is it right?
Quantifiers: each, not exactly N, at most N, most
Question: On the triangular plane where there is not exactly 1 small marbled square-based pyramid, at least 2 red marbled items have the same size as most red items; is it right?
Quantifiers: not exactly N, at least N, most
Question: On the marbled planes where there is at least 1 large marbled item on each plane, at least 2 large items are not the same material as most gray items; is it right?
Quantifiers: each, at least N, all but at least N (at least N  $\neg$ ), most
Question: On the dappled planes, is there the same number of tiny objects on the left side of the big sphere and tiny red spheres right of the red leathery triangular prism?
Quantifiers: equal  $\mathrm{O}_1$  and  $\mathrm{O}_2$
Question: On the white non-geometric plane, are there fewer big objects to the right rear of the big red dappled object than tiny spheres in front of the big red dappled object?
Quantifiers: fewer  $\mathrm{O_1}$  than  $\mathrm{O}_2$
Question: On the planes, is the number of triangular prisms to the right front of the red leathery tetrahedron greater than the number of leathery tetrahedrons on the left side of the large marble sphere?
Quantifiers: more  $\mathrm{O_1}$  than  $\mathrm{O_2}$
Question: On the planes where there are more than or equal to 2 cylinders on each plane, are there more brown metallic cones than brown leathery cylinders?
Quantifiers: each, at least N, more  $\mathrm{O_1}$  than  $\mathrm{O_2}$
Question: On the wood plane where there is not exactly 1 small yellow leathery object, is the number of small objects less than the number of brown leathery cones?
Quantifiers: not exactly N, fewer  $\mathrm{O_1}$  than  $\mathrm{O}_2$
Question: On the planes where there are no fewer than 2 big cones on each plane, is the number of circular cylinders the same as the number of big circular cylinders?
Quantifiers: each, at least N, equal  $\mathrm{O_1}$  and  $\mathrm{O}_2$
Question: On the planes where there are no more than 3 tiny marbled objects on each plane, all the tiny marbled objects are in front of the red metallic triangular prism; is it right?
Quantifiers: each, at most N, all
Question: On the plane where there are at most 4 marble items, is there any tiny ball in front of the tiny red metal three-sided prism?
Quantifiers: at most N, some
Question: On the wood plane where there are fewer than or equal to 3 small marble objects, it is not the case that no large marble triangular prism is not on the left side of the green sphere; is it right?
994 Quantifiers: at most N, not all  $(\neg \mathrm{no}\neg)$
Question: On the planes where there is exactly 1 small blue marbled object on each plane, every green metal object is not behind the green rubber cylinder; is it right?
Quantifiers: each, exactly N, no (every  $\neg$
Question: On the plane where there is not exactly 1 big green metal tetrahedron, some but not all of the marble objects are to the right of the blue tetrahedron; is it right?
Quantifiers: not exactly N, some but not all
Question: On the planes where there is a total of 5 big green items, all the tiny green blocks but at most 1 are not right rear of the big green marble block; is it right?
Quantifiers: total, at most  $\mathrm{N}$  (all but at most  $\mathrm{N} \neg$ )
Question: On the planes where there are not any small red rubber objects on each plane, there are exactly 2 small gray pentahedrons left rear of the large cyan pentahedron; is it right?
Quantifiers: each, no (  $\neg$  any), exactly N
Question: On the planes where there is not exactly 1 wood ball on each plane, are there 1 to 3 wood objects to the left front of the gray marbled square-based pyramid?
Quantifiers: each, not exactly N, between
Question: On the wooden planes where there are not between 1 and 3 red rubber items on each plane, at most 3 small items are not on the right front side of the big red sphere; is it right?
Quantifiers: each, not between, all but at most N (at most N
Question: On the planes where there are at least 2 cyan items on each plane, all the tiny cones but at least 4 are not right rear of the tiny red rubbery pentahedron; is it right?
Quantifiers: each, at least N, at least N (all but at least  $\mathbf{N}\neg$
Question: On the quadrilateral plane where there is at most 1 yellow circular cylinder, it is not the case that at most 1 red rubber object is left rear of the leather object; is it right?
Quantifiers: at most N, more than N (at most N)
Question: On the geometric plane whose material is different from that of other planes, it is not the case that all the red items but at least 2 are not right front of the tiny yellow cylinder; is it right?
Quantifiers: fewer than N (  $\neg$  all but at least N  $\neg$
Question: On the planes where there are between 1 and 3 tiny gray items on each plane, are all the tiny items but at least 5 right of the large gray metal circular cylinder?
Quantifiers: each, between, all but at least N