Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper aims at learning whether a certain text is visualizable or not as generative models such as DALLE-2 are becoming very popular. To this effort, the authors curate a dataset called Text Imageability Dataset (TIMED) which contains 3620 sentences with label for whether it is a visual or non-visual text. The authors also curate an automatic dataset with similar labels and show that when a customized CLIP model is trained on a combination of these, the model is able to perform better than a lot of other baselines. Overall, this paper focuses on a niche area of research of visualness of text and provides a solution for it but there a lot of caveats as discussed in rest of the review.
strength_and_weaknesses: ### Strengths
- The paper provides a new dataset for evaluation and an automatic dataset for training a model for the task of testing the visualness.
- The paper also provides a method to train CLIP model to detect whether a text is visual or not. 
- Provides ablations on the choice of the training objective for the purpose of testing visualness.
- Clearly written and easy to read.
- Ablations on MRC-I score and efforts to improve them using word embeddings.
- Disentangles the effect of multi-stage training and shows that both datasets are needed to reach optimum performance.

### Weaknesses
- The motivation of problem statement is weak. The paper doesn't focus on why we need to solve the problem but mostly focuses on whats and the hows. Is it even needed to solve this problem? There is no discussion on how this approach can actually improve the current generative models by incorporating the visualness into the pipeline. The data stable diffusion is trained on (LAION) is already filtered out using CLIP scores which makes this process redundant. The question is whether we want to do automatic filtering on the fly during the training or beforehand as done in stable diffusion. I also don't understand why a human will ask the model to generate a non-visual text. Overall, I am not sure why this problem is important to solve.
- I expected more analysis on the different backbones. For example, DeCLIP [1] performs better overall, FLAVA [2] and Unified-IO [3] have a better text encoder. It is hard to take away any conclusions from a single backbone test.
- BERT (full) performs better than TIP-CLIP model when only trained on auto-labeled data. It is unclear whether the difference is due to pretraining objective? Maybe using VisualBERT-style [4] model would have disentangled that?
- The downstream impact of the pretraining CLIP as TIP-CLIP-style objective is unclear as the only evaluation that has been conducted is on text-to-image retrieval and that too again on a subset that has been already filtered out by CLIP. There is a compounding effect of problems here. A clear comparison would be on rather MS-COCO or Flickr30k retrieval along with other tasks used in CLIP paper. The claim that the model has almost same performance on retrieval would strengthen if we have those results.
- No other stronger text backbones have been used except BERT. Comparisons with RoBERTa/T5 would be useful to take a call.
- Another compounding effect in the dataset is the fact that CLIP was used to filter out the matching images with custom threshold. 
- It is unclear how scene text fits into this picture. If the models get good at generating scene text, they can directly replicate non-visual text as scene text on a pdf page or a bland background. Overall, visualness is a subjective term changing from person to person and formulation of the problem.

[1] Li, Yangguang, et al. "Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm." arXiv preprint arXiv:2110.05208 (2021).

[2] Singh, Amanpreet, et al. "Flava: A foundational language and vision alignment model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Lu, Jiasen, et al. "Unified-io: A unified model for vision, language, and multi-modal tasks." arXiv preprint arXiv:2206.08916 (2022).

[4] Li, Liunian Harold, et al. "Visualbert: A simple and performant baseline for vision and language." arXiv preprint arXiv:1908.03557 (2019).
clarity,_quality,_novelty_and_reproducibility: - How were the documents filtered out to only include brochures, flyers, and inforgraphic and academic figures?
- Clustering of all of the non-visual text into a single NULL cluster may have downstream impact on text encoder. Does image-to-text retrieval performance decrease on other tests? FLAVA-style backbone where text encoder is stronger would be a better test.
- Selection of pdfs vs webpages is ambiguous. Could it be done with web pages as well with dom diff 1? This might cause the results to be limited to a specific domain?
- Was there a hyperparameter search conducted on BERT? If not, it is hard to conclude anything from one particular set of hyperparameter when number are very close. Was there a statistical significance test run?
- Why are the numbers for BERT in Table 4 row 3 different from BERT in Table 2.

summary_of_the_review: The paper focuses on a niche problem but doesn't discuss its significance and how it can help us build better model instead focuses on how to solve the problem. There is a lack of ablation study around backbones, objectives and text encoders which makes it hard to take away any significant lessons. The limitations imposed by the compounding effect of filtering using CLIP are prevalent and need to be disentangled. Given these concerns, I suggest a rating of
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is very interesting. It studies the visualness of text based on a simple example. Some words can incur our imagination for the corresponding image, while the others might not. With this motivation, the author proposed a new dataset called TimeD with both manually labeled data and the automatically model labeled data. The author also proposed a fine-tuning strategy that tunes the CLIP model to predict the visualness given a text. The whole paper is very fun to read.
strength_and_weaknesses: The idea is very interesting and novel to me. I like the motivation example and the dataset collection part. Especially using the CLIP to automatically labels the relatedness between image and text and the human annotated data to further enhance the labeling quality is pretty clever. 

But I have concerns with the fine-tuning stage.
1. The negative text is aligned with a NULL image. Given a batch of positive and negative text, and a set of positive images and NULL images. I wonder are the NULL images identical to each other?
2. If no, during different training iterations, suppose we sample a text which has been sampled before, would the NULL images looks different comparing to the previous iteration?
3. If all the negative images looks the same in the training batch, would the contrastive learning loss work? Basically the negative text is asked to aligned with a NULL image and not aligned with the same NULL image in this batch.
4. If all the negative images are different in the training batch, I doubt the contrastive learning would work. Basically, the logic follows. The negative text is asked to be close to one of the NULL image, but be far away from the other NULL images.
5. As the TIP-CLIP is always fine-tuned on the pre-trained CLIP model. I wonder how much would the pre-trained CLIP contribute? Did the author try to train from scratch on the proposed dataset? Due to lack of the data, this might be done with a smaller model.
6. For implementation detail, the model is fine-tuned on a batch size of 32 with one GPU. As the CLIP would achieve quite low performance on smaller batch size. I wonder if this is a typo or this is designed on purpose?

For Fig 1.a, the figure is rendered incorrectly on my PDF reader (I tried multiple ones) and my web browser. Only part of the image has been rendered. I wonder is this a PDF issue or the figure was drew on that way on purpose.
clarity,_quality,_novelty_and_reproducibility: I think the paper is easy and interesting to read.
The proposed dataset is pretty novel to me. Although I have concerns with the proposed fine-tuning stage approach.
With the publicity of the dataset, I think this paper is pretty straightforward to implement.
summary_of_the_review: I think the paper is interesting and the idea is novel. However I have concerns with the proposed fine-tuning stage approach. I am happy to raise my rating if my concern is resolved during rebuttal. For this stage, I will give the rating of 5.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors construct a dataset of sentences paired with a “visualness” label, indicating if the sentence evokes visual imagery. One part of this dataset is automatically synthesized by processing PDFs and computing CLIP scores between sentences and images in the PDFs. Another part of the dataset consists of sentences (also sourced from PDFs), that are then labeled by AMT workers for “visualness.” 

The authors then use this dataset to further finetune a CLIP model to match “non-visual” sentences with a “NULL” image. This model is compared to various strong baselines, and shown to be the best of all considered methods. Various ablations and analysis are also conducted, such as measuring the correlation of BERT and CLIP models trained on the dataset to MRC scores, and ablating out automatic or human-labeled portions of the dataset for finetuning. Qualitative analyses of the attention of TIP-CLIP are also presented, as well as DALLE generations conditioned on “visual” and “non-visual” text.
strength_and_weaknesses: Strengths:

Comprehensive analysis of TIP-CLIP, with good ablations of training methods, good comparison to existing MRC scores, and interesting visualizations. Appendix section contains even more analyses of the dataset, the ablations, the NULL image, and the TIP-CLIP model.

The baselines seem quite strong, and TIP-CLIP is able to outperform all of them. 

A dataset of sentences with "visualness" labels is novel (to the best of my knowledge). 

Weaknesses:

It's not clear to me that the data collection procedure should produce sentences that are really "non-visual" if the criteria for "non-visualness" is "CLIP similarity with all images on page less than T_{neg}." For example, it seems possible to me that I can have a very visual sentence that just happens to not match any image on a page in a pdf.

The novelty and usefulness of the TIP-CLIP model is a tad unclear. It's just CLIP trained with an additional NULL image tag, and as mentioned in the paper one could easily just train a binary classifier. The justification for using TIP-CLIP rather than a binary classifier is that TIP-CLIP gives us representations that can be used for image retrieval, but TIP-CLIP actually underperforms CLIP in this task (MRR of 0.937 against 0.989). Relatedly, it seems that a binary classifier was trained and the F_1 score is presented, but not any other results. Could this be added to table 2?

The baselines are all very reasonable, but I'm surprised that the pre-trained CLIP model performs so well. My understanding is that it's a vanilla CLIP model and has therefore never seen the NULL image, which is why I find it interesting that it should perform so well. Is my understanding correct? And if so how come this model does so well?

The analysis of the TIP-CLIP model is well done, but I have a few questions about the qualitative results. In figure 2 the image embeddings and the text embeddings in the T-SNE plot seem disjoint, whereas I was under the impression that the contrastive loss used to train the CLIP model would bring these two clusters together. Is this simply how CLIP representations look? Or does this indicate a problem with the dataset? Or perhaps this is a problem with T-SNE hyperparameters?

Additionally, how was the attention calculated for figure 3 and 4? Anecdotally, I think attention in language models are all over the place, so I don't know how much weight should be put into these visualizations.

Very small nitpick: I^e_m in equation 1 is defined self-referentially
clarity,_quality,_novelty_and_reproducibility: Clarity: very clear, well written and well motivated paper.

Novelty: I think the dataset and the idea of sentence "visual-ness" is novel. I think the TIP-CLIP model has more marginal novelty.

Reproducibility: I did not find any code, but I believe the details in the paper are sufficient to reproduce the TIP-CLIP result. The details of the construction of the dataset are also well documented in the paper.
summary_of_the_review: This paper poses an interesting problem: measuring the "visualness" of a sentence. The dataset presented is novel, but I have some slight concerns on the TIP-CLIP model and the dataset construction (see weaknesses section) that I would like the authors to address.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes the problem of identifying the visualness of a sentence with the motivation of complementing text in a document with images that may be generated by a text-to-image generator like DALL-E. The authors propose a manually curated and an automatically curated dataset for the this purpose. A CLIP model's objective is modified to create a TIP-CLIP which is effective as a sentence visualness classifier.

Three reviewers reviewed this work. Some of them found the motivation to be interesting, some found the data to be useful and some found the writing to be clear. However, they identified some strong concerns with the paper. First, the automatically constructed dataset seems quite noisy. If a piece of text does not correspond to an image in a document, it is currently considered as a negative, but in fact, it may be a visual sentence without a corresponding image in the document. Second, the usefulness of building TIP-CLIP was unclear and not motivated well. Yes, the TIP-CLIP model retains its representations fairly well (not surprising since it has the original objective), but what is the need for having a joint model for representations and this fairly restricted task of visualness ? Why not just train a finetuned binary classifier ? Third, although the authors provided comparisons to text models like Bert and Roberta, they did not provide any V+L models as comparisons, which would have been quite interesting. For instance, are the text representations of a text-to-image model powerful enough to solve this task ?

In addition to reviewer concerns, I wanted to note that the overall problem of complementing a page of text with images is very interesting. But this work solves just a first part of this overall problem. I think what is missing is to show that identifying visualness can help improve the quality of a document by adding images into it. There are many interesting things to consider here, such as which sentences are most visual, which images would help the reader the most, how does one dedup images that may be added, etc. Although an important component of this overall task, I think the paper tackles a very small part of the problem, which I think is a bit insufficient for this conference.

The reviewers, after discussion have provided a consistent rating to reject the paper. Given the above comments and discussion and my own reading of the paper, I am recommending a reject.
justification_for_why_not_higher_score: I have not provided a higher score due to the limitations of the scope of the work presented (as discussed above) and the concerns of the reviewers that remained even after the rebuttal including noisy automatic dataset, motivation for TIP-CLIP and lack of some comparisons to V+L work.
Decision: Reject