Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper focuses on Chinese Vision-Language Pre-training (VLP) and argues the lack of large-scale datasets and benchmarks in Chinese, thus making the following contributions:
- Release a large-scale Chinese VLP dataset  named Wukong with 100 million image-text pairs.
- Release a group of large-scale VLP models pre-trained with various popular architectures as backbone.
- Provide a benchmark on downstream tasks and show state-of-the-art performance.
strengths: 1. It provides the largest vision-language dataset in the Chinese field that is completely open to the community.
2. For the pre-training process, the combination of different training methods and training frameworks in visual-language pre-training is explored, and the experiment is sufficient.
3. The reduced token interaction mechanism is designed to reduce the pre-training cost to a certain extent.
weaknesses: 1. The dataset construction process is relatively simple, and there is few statistical analysis of it.
2. More visual-language downstream tasks need to be conducted.
correctness: There are some concerns about the construction of the dataset mentioned in the weakness above, such as the consistency between text and image.
clarity: Yes.
relation to prior work: Yes.
additional feedback: 1. When using search engines to search for keywords, the retrieved images tend to be less relevant. How can the method of building this dataset ensure the relevance of images and text?
2. Suggest adding the analysis of consistency between text and image.
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work releases a large-scale chinese cross-modal dataset,Wukong, containing about 100M image-text pairs. These image-text pairs are filtered according to image resolution and text contents(i.e. person name are replaced, sensiteive contents are discarded).   The authors also propose a new method for training(token-wise similarity+ token reduction) .   With the proposed dataset and training method, the authors pretrained and released a series of foundation models that obtain state-of-the-art results.   This work also provides a benchmark for VLP community.
strengths: The first public chinese cross-modal dataset that attain a scale of 100M. 
Extensive expriments have been conducted and show the effectiveness of the dataset.
The released pretrained models will benifit both the industrial  community and the VLP community.
weaknesses: Some details for training different variants are missed. 
correctness: I have the following questions for the authors:
1) It seems that you don't mention which pretrained visual encoders are used for LiT-tuning.  It's important for the replicability. 
2) I doubt that there might be something wrong in the hyperparameters given in the suplementary material.  Are you sure that the linear projected embeddings  have a dimmension of 256?  it's inconsistent with the parameters presented in your project web page. If you use the visual encoders pretrained by CLIP, that dimmension should be 512 or 768.  And I find that the results are the same with  your second version of paper preprinted on arxiv, and in that paper, the embedding dimmensions are 512 and 768. 
3) I have read your previous two versions of Wukong's paper, I notice that the performances of CLIP-style models have been imporved significantly compared to the results in version1. Could you please tell me what leads to this improvement?
clarity: In general, this paper is well writen. 
relation to prior work: Yes, the main contributions are clearly claimed. This work proposes the first public chinese cross-modal dataset that attain a scale of 100M.  
additional feedback: None
review_title: a great large-scale chinese cross-modal dataset and some pretrained foundation models.   

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: - Summary

In this paper, a large-scale Chinese VLP dataset containing 100 million image-text pairs and a series of benchmark models are constructed and released online. Meanwhile, using this dataset, a series of dual-encoder models called Wukong are pretrained using different backbones including ViT and Swin-Transformer. Compared with other previous works, e.g., CLIP and FILIP, extensive evaluation experiments including classification and retrieval are conducted for benchmarking.
- Contributions
1. A large-scale Chinese VLP dataset is constructed and released, and the dataset scale is larger than the previous ones (for Chinese) by an order of magnitude. This dataset is useful for future research in the community.
2. Various VLP models, including CLIP, FILIP, and the proposed Wukong using several special training techniques and different backbones, are pretrained on the proposed dataset and evaluated on multiple downstream tasks.
3. The proposed model called Wukong shows good performances in multiple downstream tasks.
strengths: * Strengths
1. V-L Pretraining is a hot topic and the community is lack Chinese benchmarking. The released image-text dataset is quite large-scale and beneficial to future research specifically for Chinese VLP in the community.
2. The variants of VLP models including CLIP, FILIP, and Wukong are well evaluated and released online. This is useful for Chinese VLP research and even applications.
3. Some findings are instructive, for example: using word-grained tokenization for Chinese is not a good choice in comparison with the plain character-grained one.
weaknesses: 
- Is there any statistical information for the Wukong-Test, e.g., average and maximum lengths of tokens, like Table 3 for the Wukong dataset for pretraining? What is the maintenance plan of this dataset specifically for testing? In my opinion, it is critical if this testing dataset will be a leaderboard for image-text retrieval of Chinese VLP models.
- There are no descriptions of the number of parameters of the evaluated models, e.g., CLIP_ViT-L, FILIP_ViT-L, Wukong_ViT-L. The comparison of the model scale is needed because model size is related to performance in most cases.
- In line 132, "Thus, the current community lacks a large-scale Vision-Language dataset in Chinese. " The introduction is not solid enough. 
- In line 199, you mentioned, "This token reduction layer is used in all the Wukong-style models."  But It seems all models(Clip-Style, FILIP-Style, WuKong-Style) used this component. Is there any ablation study to prove its effectiveness?
correctness: - The original English prompt templates, which are translated into the Chinese version in the appendix, need to be shown for comparison.
clarity: Yes
relation to prior work: Yes.
additional feedback: Please see the questions above.
review_title: A pioneering Large-scale Chinese VLP Dataset and extensive Benchmarks.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work releases a large-scale Chinese cross-modal dataset, which aims to benchmark different multi-modal pre-training methods. It also releases a group of models pre-trained with various image encoders and applies advanced pre-training techniques into VLP. Extensive experiments have shown that this dataset can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods.
strengths: 1.	It releases a large-scale Chinese VLP dataset with 100 million image-text pairs, covering a wide range of concepts.
2.	It releases a group of large-scale VLP models pre-trained with various popular architectures methods. An extensive study and benchmarking are also provided.
3.	The pre-trained model of this work shows state-of-the-art performance on Chinese benchmarks such as zero-shot, image classification and image-text retrieval tasks.

weaknesses: Nothing. 
correctness: 
The dataset in this paper is constructed in a reasonable way. The evaluation methods and experimental design are appropriate and correct.
clarity: This paper is well written.
relation to prior work: It's clearly discussed how this work differs from previous contributions.
additional feedback: Very nice.
review_title: A excellent job in benchmarking

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: A large-scale Chinese cross-modal dataset containing 100 million image-text pairs is released. Extensive experiments and a benchmarking of different downstream tasks including a new largest human-verified image-text test dataset are also provided. Experiments show that Wukong can serve as a promising Chinese pre-training dataset and benchmark for different cross-modal learning methods.
strengths:  •Large-scale: containing 100 million image-text pairs.    
 •Released a group of models that can be used for downstream tasks which can not be trained limited by compute resources. 
•Experiments surrounding zero-shot text-image retrieval seem to show good generalization of the latent representations.

weaknesses:  •The Experiment part is relatively thin which just includes zero-shot Image Classification and image-text retrieval.  \
 •We can see from Table 4 and Table 5  that the results of Wukong sometimes can not get a better result compared with CLIP and FILIP. We 
   may get the conclusion that may be the diversity of the dataset is insufficient.

correctness: The dataset is constructed soundly, from collecting <image, text> pairs from Baidu, limiting variance in image aspect-ratio, to text-based filtering, everything seemed logical to do to arrive at the final dataset.
clarity: Yes
relation to prior work: It is clearly discussed how this work differs from previous contributions.
additional feedback: S1. the diversity of the dataset should be an improvement.   \
S2.  more pre-train models should be public to the community.
review_title: The paper is good whild it could be imporved 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Wukong, a Chinese Vision-language pretraining dataset of ~100 million image-text pairs, is introduced so that the research community has a large enough dataset to pretrain VLP models. Translating existing English vision-language datasets and using that for pretraining has been shown to perform suboptimally in comparison to the source language. Additionally, several large-scale VLP models were pretrained on Wukong and shown to generalize well in the zero-shot setting on image classification and image-text retrieval.
strengths: - First large-scale Chinese VL dataset enables pretraining of large VLP models, saving computation for others when fine-tuning on downstream tasks
- Released pre-trained VLP models using their methodology, inspired by the methods of CLIP, ALIGN, and FILIP
- Experiments surrounding zero-shot text-image retrieval seem to show good generalization of the latent representations
weaknesses: - Not very well motivated why such a dataset is necessary. Might want to point more towards downstream tasks that could motivate it better (e.g. image search engines, visual question answering, embodied AI, etc.)
- Zero-shot image classification is used as a way to justify that VLP on Wukong is good for VL models. I don't quite understand why this justifies the dataset creation. Was it that the labels were converted to Chinese and so VLP on Wukong allowed models to ground visual concepts to Chinese vocabulary?
correctness: The dataset is constructed soundly, from collecting <image, text> pairs from Baidu, limiting variance in image aspect-ratio, to text-based filtering, everything seemed logical to do to arrive at the final dataset. However, for the benchmarking portion/evaluation methods, I believe the submission was lacking. The portion on performing better than WenLan 2.0 doesn't seem like an apples-to-apples comparison. Was the same model used but the pretraining dataset differed? Like I mentioned previously, the focus on zero-shot image classification for evaluation seems misguided.
clarity: The paper is somewhat clear. I think the introduction does not have a clear enough motivation for the need of Wukong. Like I mentioned before, mentioning downstream applications would bode well for strengthening this. The related work is well done in that it places your work in context. Construction is fine, but I find it a bit strange to focus so much on the modeling (i.e section 4) in this paper. Maybe an exploration of the dataset is more suitable for the main body of the paper rather than an appendix. The experimental setup should be changed to more reflect how pretraining on Wukong helps in downstream performance (which text-image retrieval is already fine).
relation to prior work: It is clearly discussed how this work differs from previous contributions.
additional feedback: It might help to actually finetune on downstream vision and language tasks so that you could perform an ablation study of models trained from scratch on the task and the finetuned version (pretrained on Wukong). That way, others can see the significance of this pretraining. Another suggestion I have is to try and give a high level breakdown of topics/types of images because determining that by researchers who didn't collect the dataset could be as good as the dataset being non-public (due to not understanding what topics are in the dataset).

How is the list of Chinese sensitive words constructed (pg 4 line 163)? You might want to have a sentence on how that was made.

Grammatically speaking, some suggestions:
- pg 1 line 24, change "have" to "has"
- pg 4 line 148, "is no more than 3" is really awkward wording in that context. Maybe consider "of at most 3"?
- pg 5 line 205, "while apart otherwise" is pretty awkward as well. Maybe "and far otherwise"?

Update: I raised the score to a 6, but I'm still not entirely convinced. Please look at my response for details.
Update: I raised the score to a 7.
review_title: A large scale Chinese vision-language dataset that could be motivated and evaluated better

Meta-review: metareview: This paper has collected and released a large-scale Chinese multimodal dataset with 100M image-text pairs. The authors have benchmarked VLP with various image encoders and different pre-training techniques. Evaluation is conducted on image classification and image-text retrieval. All reviewers are positive about the paper. 

Pros:
- largest Chinese image-text dataset. 
- benchmarked different image encoder architectures. 
- different training schemes are explored and experimented. 

Cons:
Several reviewers have pointed out that the downstream tasks are limited to only image classification and image-text retrieval. Authors have replied and identified that expanding more downstream tasks as future work. 

Overall, the AC believe this dataset can be valuable to the research community. The benchmark of VLP pretraining architecture/method can also bring insights. 
confidence: 5: The area chair is absolutely certain
Decision: Accept