Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a task of joint visual-linguistic grammar induction from parallel image-text data, presents models and metrics for the task, and shows strong empirical results.

main_review: 
### Strengths 

- As far as I know, this is the first paper that proposes joint visual-linguistic grammar induction in a real-world setting (in contrast to synthetic settings; Hong et al., 2021).

- The approach and the evaluation process are solid and make a lot of sense to me.  

- The visually grounded parsing results are quite impressive. 

### Weakness

- My major concern is about the model selection process and the potential unfair comparisons to existing work. 
    - Model selection: If I understood correctly, for text parsing, the best models are selected w.r.t. to the parsing performance on a 1000-example dev set (Appendix F). \
This is an unrealistic setting (see https://aclanthology.org/2020.emnlp-main.614.pdf for discussions; in short, for any fancy unsupervised parsing model that uses a labeled development set, a supervised model trained on these development examples should be considered as a strong baseline) -- introducing unsupervised criteria for model selection is more important than our initial impression.

    - Unfair comparison: CLIORA, the model proposed in this paper, uses DIORA as initialization, which uses ELMo to initialize word embeddings and the PTB labeled development set for model selection. This means that CLIORA has seen far more text than other baselines (VG-NSL, C-PCFG, VC-PCFG, and so on) and human language learners. \
This issue also undermines the authors' arguments about potential links to how humans learn language. I expect either a CLIORA trained from scratch (without DIORA initialization) or weakened arguments about the relationship between the current CLIORA and human language learning.

- There seem to be some confusion on basic linguistic concepts, e.g., nonterminal vs. terminal symbols, and a few typos that affects smooth understanding (please see also detailed comments below). 

### Other comments and questions

- Introduction: "These works, however, fail to consider a unified VL structure, nor have they demonstrated impact on visual understanding."  \
I don't think I necessarily agree with this statement, especially regarding Hong et al. (2021). Despite that there is a clear gap between their dataset and the real world settings, they are aligning the "visual grammars" to language grammars, yielding an arguably unified VL structure. 

- Introduction: "The non-terminal symbol of a conventional constituency structure is a category label from a limited set (e.g., the set of part-of-speech (POS) tags) (Hopcroft et al., 2001). " \
Do you mean *terminal* symbols here? We usually refer to POS tags (to clarify, phrase tags are not POS tags) by preterminal or terminal (depending on whether the phrase-structure grammar is lexicalized, i.e., whether it's considering real words or just POS tags), and refer to the phrase nodes by nonterminal nodes/symbols (e.g., NP, PP). It seems that this is not a typo -- I have the same questions for the following task definition section on page 3.

- Task definition, evaluation metrics: if I understood correctly, CCRA requires some extra annotation of critical concepts -- how did you collect such annotations to determine which NPs are critical? \
(Very minor) based on the full name, CCRA should really be CCRR -- what does A stand for here? 

- Section 3.2, feature extraction: the Yoon Kim et al. (2019b) paper is not relevant to image features at all -- did you mean Shi et al., (2019)? 

- Table 1: what is the dagger after VGNSL-HI? 

- Section 4.3: did you mean "augments" by "arguments"? 

- Some more thoughts regarding motivation limitations: humans arguably learns how to parse concrete sentences first, and can then generalize to abstract domains that are not visually groundable. In this work, it seems that the model only works when both the text and image are available, as there is a need to infuse visual features into text spans. Do you have any thoughts on enabling a trained CLIORA model to parse pure text without grounding signals?

### Missing Reference

Kojima et al. [1] has strengthened the VG-NSL model by simplifying the architecture, and argued that such visually grounded models are potentially biased towards concrete noun phrases. However, the paper neither cited it nor discussed the relevant issues. 

[1] https://aclanthology.org/2020.acl-main.234.pdf 

There have been a lot of relevant work earlier than 2019 on visual-semantic embeddings or structured visual understanding with text. To name a few, 

**Older work on structured image-text representations**

[2] https://openaccess.thecvf.com/content_iccv_2015/papers/Ma_Multimodal_Convolutional_Neural_ICCV_2015_paper.pdf

[3] https://openaccess.thecvf.com/content_cvpr_2018/papers/You_End-to-End_Convolutional_Semantic_CVPR_2018_paper.pdf

**Contrastive loss for visual-semantic embeddings**

[4] https://arxiv.org/pdf/1411.2539.pdf


### Minor Editing Comments
- I was confused about what CCRA is when reading the abstract -- would be good to include the full name and give an intuitive description of the metric. 

- Yoon et al. $\rightarrow$ Kim et al. 

- Shi et al. (2019) proposes $\rightarrow$ Shi et al. (2019) propose

- In my opinion, putting Section 3.4 before 3.3 would better streamline the paper. 

summary_of_the_review: This paper introduces the task of joint visual-linguistic grammar induction, and presents models, metrics and empirical results on it. While I appreciate the impressive results, I am concerned about the unrealistic model selection process (comparing model outputs to a large set of ground-truth parse trees) and the unfair comparison (the proposed model has access to much more unlabeled text data than baselines).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new model for grammar induction for text, with help from the coupled images. The model was built on top of an existing unsupervised grammar induction model used for text without image information. The experimental results show the approach was effective. The work essentially demonstrates some effective ways of leveraging the additional image information for improving the grammar induction task. The paper also discussed some weaknesses of the approach and future work.
main_review: The topic of grammar induction has been there for a very long time in NLP and is a very fundamental topic.  The model was largely built based on an existing model for purely text-based grammar induction. The model essentially makes use of neural networks to learn good latent representations (using a reconstruction loss) where the latent representation is defined with neural networks which yield scores for constituents and vector representations of them. The approach adopts the classical inside-outside process for the computing of the scores.

The paper essentially investigates what might be the effective methods for integrating image information into text for improved grammar induction. The execution of the paper was quite good, and the results are convincing. However, I feel the overall model is essentially a way to use image information to regularize the grammar induction process. Little can be said about in what precise manner the image is actively contributing to the induction process. Indeed, the authors also acknowledged something along with what I thought in the final section. Nevertheless, I think it is an interesting piece that might inspire future research on multimodal processing (for image + language).
summary_of_the_review: I think this is a reasonable piece, with good writing and a nice set of experiments. It would be helpful for future research in this domain.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a new method CLIORA to do unsupervised parsing and vision-language grounding. 
CLIORA is based on DIORA model. 
But different from previous unsupervised parsing methods, CLIORA also induces alignment between constituents and image regions. 
In order to train the model, the author introduces a contrastive loss.
Experiment results show that the proposed method outperforms baseline unsupervised parsing methods and it also induces meaningful alignment between image regions and constituents.
main_review: Strengths:
The idea of jointly inducing structure in natural language and grounding the constituents with real-world images is intuitively correct.
The ablation study also shows that both feature-level fusion and score-level fusion (including the contrastive loss, if I understand correctly) helps in improving the parsing results.

Weakness:
1) The image features are only used for computing the inside pass. The image feature should contain information that can help predict the missing word, such that it could be used in the outside pass too. Selecting the best image region for predicting the missing word is also an intuitively correct way to build the vision-language alignment.
2) The compute of sim(I, c_ij) includes a max operator, this could lead to a biased gradient.
3) As the author mentioned in the discussion section, the model doesn't consider the latent hierarchical structure of the image. For example, the sentence describes the entire image, while each phrase describes part of the image.
summary_of_the_review: Overall, the proposed method is interesting and inspiring. 
The idea should be interesting to both unsupervised parsing and multimodel communities.
review_title: null

Meta-review: This paper proposes to perform unsupervised grammar induction over image-text pairs and used shared structure between the modalities to improve grammar induction on both sides. Authors find the paper clear, creative and interesting and recommend acceptance without hesitation.
Decision: Accept (Oral)