SlowGuess/ACL_2020_OCR huggingface dataset:
for image captioning and visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition, pages 6077-6086.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sunderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018b. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In The IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683.
Mark C. Baker. 1987. The atoms of language: The mind's hidden rules of grammar. Basic books.
Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A. Knepper, and Yoav Artzi. 2018. Following high-level navigation instructions on a simulated quadcopter with imitation learning. In Proceedings of the Robotics: Science and Systems Conference.
Valts Blukis, Eyvind Niklasson, Ross A. Knepper, and Yoav Artzi. 2019. Learning to map natural language instructions to physical quadcopter control using simulated flight. In Proceedings of the Conference on Robot Learning.
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46(3):904-911.
Volkan Cirik, Louis-Philippe Morency, and Taylor Berg-Kirkpatrick. 2018. Visual referring expression recognition: What do systems actually learn? In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 781-787.
Michael John Collins. 1996. A new statistical parser based on bigram lexical dependencies. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 184-191.
Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1129-1141.
Chris Dyer, Gábor Melis, and Phil Blunsom. 2019. A critical analysis of biased parsers in unsupervised parsing. arXiv preprint arXiv:1909.09428.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 457-468.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In The IEEE Conference on Computer Vision and Pattern Recognition, pages 6325-6334.
Serhii Havrylov, Germán Kruszewski, and Armand Joulin. 2019. Cooperative learning of disjoint syntax and semantics. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1118-1128.
Jack Hessel, David Mimno, and Lillian Lee. 2018. Quantifying the visual concreteness of words and topics in multimodal datasets. In Proceedings of North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2194-2205.
Phu Mon Htut, Kyunghyun Cho, and Samuel Bowman. 2018. Grammar induction with neural language models: An unusual replication. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 4998-5003.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In *The IEEE International Conference on Computer Vision*, pages 804-813.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. 2019. Stay on the path: Instruction fidelity in vision-and-language navigation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1862-1872.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Yoon Kim, Alexander M Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gabor Melis. 2019. Unsupervised recurrent neural network grammars. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1105-1117.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2676-2686.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco:
Common objects in context. In European conference on computer vision, pages 740-755.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In The IEEE Conference on Computer Vision and Pattern Recognition, pages 11-20.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2667-2678.
Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to actions with reinforcement learning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1004-1015.
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, pages 2089-2096.
Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron Courville. 2018a. Neural language modeling by jointly learning syntax and lexicon. In Proceedings of International Conference on Learning Representations.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In Proceedings of International Conference on Learning Representations.
Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019. Visually grounded neural syntax acquisition. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1842-1861.
Peter D Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 680-690.
Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, and Jia Deng. 2016. Structured matching for phrase localization. In The European Conference on Computer Vision, pages 696-711.
Adina Williams, Andrew Drozdov*, and Samuel R Bowman. 2018. Do latent tree learning models identify meaningful structure in sentences? In Transactions of the Association for Computational Linguistics, volume 6, pages 253-267. MIT Press.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling context in referring expressions. In The European Conference on Computer Vision, pages 69-85.
A Additional Results and Visualizations
Table 5 is an extended version of Table 1 from Section 5. We include standard deviation for the phrasal category recall and self- $F_{1}$  scores evaluated across different parameter initializations. Figure 3 is a larger version of Figure 1 from Section 5. It visualizes the token embeddings of  $1, \mathrm{s}_{\mathrm{WS}}$ ,  $\mathrm{c}_{\mathrm{ME}}$  and  $2, \mathrm{s}_{\mathrm{WS}}$ ,  $\mathrm{c}_{\mathrm{ME}}$  for all universal parts-of-speech categories (Petrov et al., 2012). Figures 4 and 5 show several examples visualizing our learned representations with the  $1, \mathrm{s}_{\mathrm{WS}}$ ,  $\mathrm{c}_{\mathrm{ME}}$  variant, the  $1d$  variant closest to the original model, as a concreteness estimate. Figure 4 shows the most concrete nouns, and Figure 5 shows the least concrete nouns. We selected nouns from the top (bottom)  $5\%$  of the data as most (least) concrete. We randomly selected image-caption pairs for these nouns.
At the end of the supplementary material, we include tree visualizations, comparing gold trees with phrasal categories, trees generated by the original VG-NSL, and trees generated by our best performing, simplified  $1, \mathrm{s}_{\mathrm{MHI}}, \mathrm{c}_{\mathrm{MX}}$  variant. We select the trees to highlight the difference between VG-NSL and our variant. First, we select all development trees where all five VG-NSL models agree to avoid results that are likely due to initialization differences. We do the same for our variant. Finally, we select all trees where the two sets, from VG-NSL and our variant, disagree. This process leaves us with 814 development examples, out of the original 5,000 examples. We display ten examples from this final set.
Model	NP	VP	PP	ADJP	Avg. F1	Self-F1
Shi2019	79.6 ± 0.4	26.2 ± 0.4	42.0 ± 0.6	22.0 ± 0.4	50.4 ± 0.3	87.1
Shi2019*	80.5 ± 1.5	26.9 ± 0.9	45.0 ± 2.9	21.3 ± 1.2	51.4 ± 1.1	87.3
1, sWS, cME	77.2 ± 5.3	17.0 ± 5.2	53.4 ± 12.8	18.2 ± 1.0	49.7 ± 5.9	76.0
2, sWS, cME	80.8 ± 1.1	19.1 ± 1.1	52.3 ± 3.5	17.1 ± 1.0	51.6 ± 0.6	88.1
+HI
Shi2019	74.6 ± 0.5	32.5 ± 1.5	66.5 ± 1.2	21.7 ± 1.1	53.3 ± 0.2	90.2
Shi2019*	73.1 ± 0.3	33.9 ± 0.8	64.5 ± 0.2	22.5 ± 0.4	51.8 ± 0.3	91.6
1, sWS, cME	74.0 ± 0.4	35.2 ± 2.0	62.0 ± 1.1	24.2 ± 0.9	51.8 ± 0.4	87.3
2, sWS, cME	73.8 ± 0.3	30.2 ± 0.4	63.7 ± 0.3	21.9 ± 0.3	51.3 ± 0.1	93.3
+HI+FastText
Shi2019	78.8 ± 0.5	24.4 ± 0.9	65.6 ± 0.1	22.0 ± 0.7	54.4 ± 0.3	89.8
Shi2019*	77.3 ± 0.1	23.9 ± 0.5	64.3 ± 0.3	21.9 ± 0.3	53.3 ± 0.1	92.2
1, sWS, cME	76.6 ± 0.3	21.9 ± 2.3	68.7 ± 4.1	20.6 ± 0.9	53.5 ± 1.4	87.8
2, sWS, cME	77.5 ± 0.2	22.8 ± 0.4	66.3 ± 0.6	19.3 ± 0.7	53.6 ± 0.2	93.6
+HI+FastText-IN
Shi2019*	78.3 ± 0.2	26.6 ± 0.3	67.5 ± 0.5	22.1 ± 1.0	54.9 ± 0.1	92.6
1, sM, cMX	79.6 ± 0.2	29.0 ± 0.7	38.3 ± 0.3	23.5 ± 0.6	49.7 ± 0.2	95.5
1, sMHI, cMX	77.6 ± 0.2	45.0 ± 0.8	72.3 ± 0.2	24.3 ± 1.0	57.5 ± 0.1	93.4
1, sM, cME	80.0 ± 0.2	26.9 ± 0.2	62.2 ± 0.4	23.2 ± 0.4	54.3 ± 0.2	95.7
1, sMHI, cME	76.5 ± 0.1	20.5 ± 0.8	63.6 ± 0.6	22.7 ± 0.7	52.2 ± 0.3	94.7
1, sWS, cME	77.7 ± 0.1	26.3 ± 0.4	72.5 ± 0.2	22.0 ± 0.6	55.5 ± 0.1	95.5
2, sWS, cME	78.5 ± 0.4	26.3 ± 0.6	69.5 ± 1.2	21.1 ± 0.5	55.2 ± 0.1	93.7
Table 5: Test results. We report the results from Shi et al. (2019) as Shi2019 and our reproduction as Shi2019*. We report mean  $F_{1}$  and standard deviation for each system and mean recall and standard deviation for four phrasal categories. Our variants are specified using a representation embedding  $(d \in \{1,2\})$ , a score function  $(\mathrm{s}_{\mathrm{M}}$ : mean,  $\mathrm{s}_{\mathrm{MHI}}$ : mean+HI,  $\mathrm{s}_{\mathrm{WS}}$ : weighted sum), and a combine function  $(\mathrm{c}_{\mathrm{MX}}$ : max,  $\mathrm{c}_{\mathrm{ME}}$ : mean).
Figure 3: Token embedding visualization for  $2, \mathrm{s}_{\mathrm{WS}}, \mathrm{c}_{\mathrm{ME}}$  (top) and  $1, \mathrm{s}_{\mathrm{WS}}, \mathrm{c}_{\mathrm{ME}}$  (bottom) colored by universal POS tags (Petrov et al., 2012).
Elephant (4633 occurrences):
(a) A person riding an elephant and carrying gas cylinders.
(b) An elephant is in some brown grass and some trees.
(c) A captive elephant stands amid the branches of a tree in his park-like enclosure.
(d) Two baby gray elephant standing in front of each other.
(e) The older elephant is standing next to the younger elephant.
Giraffe (5546 occurrences):
(a) Two giraffe standing next to each other on a grassy field.
(b) A giraffe laying down on the dirt ground.
(c) A herd of giraffe standing next to each other on a field.
(d) A giraffe stands beneath a tree beside a marina.
(e) A giraffe rests its neck on a bunch of rocks.
Pizza (8340 occurrences):
(a) A woman holding a pizza up in the air.
(b) A slice of pizza sitting on top of a white plate.
(c) A pizza sitting on top of a plate covered in cheese and tomatoes.
(d) Three pieces of sliced pizza on a wooden surface.
(e) Some boxes of frozen pizzas are in the store.
(f) A pizza topped with cheese and pepperoni with veggies.
(g) A large pizza is in a cardboard box.
Snowboarder (922 occurrences):
(a) A snowboarder practicing his moves at a snow facility.
(b) A snowboarder is coming down a hill and some trees.
(c) A snowboarder rests in the snow on the snowboard.
(d) A snowboarder jumps off of a hill instead of just sliding down it.
(e) A snowboarder is jumping in the air with their board held to the side.
(f) The snowboard is almost as big as the snowboarder.
(e)
Figure 4: Image-caption pairs corresponding to noun tokens estimated as most concrete (bottom  $5\%$ ) in our  $1, \mathrm{s}_{\mathrm{WS}}, \mathrm{c}_{\mathrm{ME}}$  variant. We also report the number of occurrences in the MSCOCO training set.
Metal (1630 occurrences):
(a) A pink piece of metal with a bolt and nut on top.
(b) Wilting roses and greenery in a metal vase.
(c) A couple of street signs sitting on top of a metal pole.
(d) Kitchen with wooden cabinets and a metal sink.
(e) A metal toilet and some tissue in a bathroom.
Palm (321 occurrences):
(a) A motorcycle sits parked in palm tree lined driveway.
(b) Two people in helmets on a parked motorcycle and a small palm tree to the side of them.
(c) Two flat bed work trucks among palm trees.
(d) A cake with palm trees, and a person on a surf board.
(e) A pink cellphone and white palm pilot on a table.
Picture (5932 occurrences):
(a) A blurry picture of a cat standing on a toilet.
(b) Picture of a church and its tall steeple.
(c) The street sign at the intersection of Broadway and 7th avenue is the star of this picture.
(d) A picture of some people playing with a frisbee.
(e) A little girl sitting in the middle of a restaurant and smiling for picture.
Time (1184 occurrences):
(a) A time lapse photo of a skier skiing down a hill.
(b) A skaterboarder getting major air over some stairs during a night time shoot.
(c) The man is trying to eat three hot dogs are the same time.
(d) A boy playing a Wii game at Christmas time.
(e) A large display of a hand holding a cell phone to tell the time.
(d)
Figure 5: Image-caption pairs corresponding to noun tokens estimated as least concrete (bottom  $5\%$ ) in our 1, sWS, cME variant. We also report the number of occurrences in the MSCOCO training set.
Gold Tree
VG-NSL (original)
Simplified VG-NSL  $(1, s_{MHI}, c_{MX})$
Simplified VG-NSL  $(1,s_{MHI},c_{MX})$
2629
Gold Tree
VG-NSL (original)