Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes TaiSu, a large-scale multi-modal dataset that contains over 166M images and 219M Chinese captions, along with a new framework for acquiring and cleaning image-text data. The experiments on downstream tasks (i.e., zero-shot image classification and zero-shot image-text retrieval) show the effectiveness and high quality of TaiSu for vision-language pre-training.
strengths: 1. TaiSu is currently the largest Chinese image-text dataset. 
2. Compared with previous works (e.g., Wukong), this dataset leverages an image-text retrieval model to clean unpaired data and further improve the correlations between image-text pairs. In addition, some of the images on the dataset have multiple captions generated by an image captioning model, which expands this dataset. This may provide an idea for future work to build and filter a multi-modal dataset. 
3. The experiment shows the dataset can help pre-training models can get better performance on many datasets than previous works with the same pre-training method.
weaknesses: 1. Although the number of the images in TaiSu has 50% larger than Wukong, the pre-trained model by TaiSu fails to outperform the one got from Wukong on some downstream datasets.



correctness: It is constructed in a sound way.
clarity: It is fine to me.
relation to prior work: The proposed dataset is much larger than previous works, such as Wukong and WudaoMM.
As I state in the Strengths, the authors utilize an image-text retrieval model to remove the image-text pairs which are weakly aligned, and an image captioning model to further augment the dataset.

additional feedback: 1. Documentation for the dataset should be provided.
2. Details of the paper can be improved. For example, there are some duplicate references, e.g., [9] and [21], [12] and [23]. 
3. Could you please explain why the pre-trained model from TaiSu fails to outperform the one from Wukong on sone datasets, such as MUGE and Flowers? I am interested in this phenomenon.
review_title: Review to Paper155

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper contributes the largest chinese cross-modal benchmark, named as Taisu. The whole dataset acquisition framework is clearly described and reasonable. The experiments are sufficient.
strengths: The presented Taisu is the largest chinese cross-modal dataset so far and publicly available. It supports many downstream cross-modal tasks. The performance by proposed method is reasonable good compared to SOTA methods. 
weaknesses: 1. Product1M in ICCV 2021 and M5product in CVPR 2022 also proposed the Chinese cross-modal dataset. However, the authors miss the comparison with them in Table 1.
2. In sec. 5.4, Taisu presents the Text-to-Image generation results. It misses the comparison with the latest Cogview and Cogview2, which are SOTA image generation method. Could the authors add such comparison?
3. CLIP, BLIP and FILIP are publicly open-sourced. Even though they do not have results on chinese cross-modal dataset in their original paper, the authors are highly encouraged to compare with one of them as they are most widely used.

Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining, ICCV 2021.
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining, CVPR 2022.
correctness: Yes
clarity: Reasonable well
relation to prior work: Yes
additional feedback: N/A
review_title: Valuable benchmark

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors propose a new framework for automatic dataset acquisition and cleaning to construct TaiSu, a new large-scale and high-quality image-text pair dataset containing 166 million images and 219 million Chinese captions, to address the lack of high-quality public Chinese multi-modal datasets for VLP tasks. In the framework, they perform text-based filtering, cross-model semantic based filtering, and image caption text augmentation to produce a clean and high-quality dataset. They perform extensive experiments using dual-stream VLP models, in particular the RN101 and ViT-B/32, as locked image encoders, and use three types of data to verify the effects of filtering and generated texts. TaiSu shows great performance on a number of vision-language downstream tasks including zero-shot image-text retrieval, zero-shot image classification, and text-to-image generation. The authors plan on making the dataset, pretrained models, and codes public. 
strengths: 1. As the current largest and publicly accessible Chinese image-text pair dataset, TaiSu serves as a promising benchmark for VLP tasks. This will be another useful resource for researchers to use when conducting their own research.

2. The authors provide a framework that automatically filters texts that include sensitive words and words that have no semantic connection with the image, filters image-text pairs with low correlation using cosine similarity, and augments image captioning by combining texts from web and texts generated by machine. TaiSu is the first to augment texts using both sources. 

3. The authors perform extensive experiments using the RN101 and ViT-B/32 models pre-trained for CLIP as locked image encoders for a wide variety of VLP tasks. They use more than 10 datasets to do a comparative analysis with TaiSu. The results in the tables are succinctly shown and easy to understand.

4. The paper is organized very well and easy to follow. 
weaknesses: 1. In Table 1, the authors list datasets that are primarily in English or Chinese. Are there VLP datasets in other languages? The multilingual LAION-5B which was released recently may be another useful dataset to include in this list. 

2. It would be nice to provide an English translation for the Chinese characters in section 3.2 and the 50 most common nouns in Figure 7 of the Appendix.

3. Can the authors please explain why they used RN101 and ViT-B/32 specifically as the models for their experiments? 

4. In Figure 2, it is difficult to tell apart between watercolor painting and oil painting. Using a different artistic style can help show clear differences between the four categories. 


correctness: The dataset is constructed in a sound way with filtration and augmentation steps that make sense. The experiments are very thorough and the results support their claim that TaiSu is a promising benchmark for VLP. 
clarity: The paper is structured very well and easy to follow. Except for a few grammatical errors, the paper is written well. I would suggest to expand all the abbreviated words such as "hasn't" and "it's" to sound more formal. 
relation to prior work: In Section 2, the authors describe related VLP datasets and models and highlight again the dearth of accessible Chinese multimodal datasets. 
additional feedback: There are several typos that can be fixed:

1. line 6: more than 100M image-text pairs are in English, --> change comma to semicolon (;)
2. lines 8-9: we propose a new framework for automatic dataset acquisition and cleaning and construct a new... --> combine into one sentence
3. line 17-22: Further, we test our dataset on several vision-language downstream tasks. On zero-shot image-text retrieval tasks, TaiSu outperforms Wukong by a considerable margin when the model has a similar scale of parameters. Results demonstrate that TaiSu can serve as a promising benchmark for VLP, both for understanding and generation (generating?) tasks. 
5. line 24: foundation model does not make sense
6. line 26-28: Its success is not only attributed to the appearance of advanced models like BERT and ViT, but also to the use of large-scale multi-modal datasets. 
7. line 30: in the VLP domain
8. line 31: the English corpus 
9. line 34: M6-corpus hasn't been --> has not been
10. line 37: Thus, compared with the English VLP community, there is a lack of large-scale and high-quality public Chinese multi-modal datasets for VLP tasks.
11. lines 39-46: Change Firstly, Secondly, and Thirdly to First, Second, and Third
12. lines 45-46: while it is even mismatched (please fix)
review_title: TaiSu is the largest publicly accessible Chinese image-text pair dataset that can serve as a promising benchmark for VLP tasks. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: 1. Construct a new large-scale and high-quality image-text pair dataset named as TaiSu, containing 166 million images with captions.
2. Add more restricted constraints on filtering the dataset.
3, Add some model-generated captions in the dataset and show some improvements.
strengths: 1, V-L Pretraining is a hot topic and the community is lack of Chinese benchmarking.
2. Another Chinese large-scale benchmarking dataset using more restricted constraints.
3. Generated some samples of different artistic styles.
weaknesses: 1. Need more comparisons with more VL-pretraining methods other than just CLIP. Methods in Table 4 and Table 5 are all CLIP pretraining methods with different backbones and different dataset constructions. In order to have a better benchmarking, the Authors may compare with more VL-pretraining methods such as FILIP, DeCLIP, DeFILIP, and so on. 
2. The generation task is very interesting. Can you provide a more quantitive analysis on generation tasks with some comparing methods such as DallE and GLIDE? For example, provide evaluation metrics like  FID, IS, and Clip-score on MS-COCO CN?
3. By the way, what is Mental Search in Fig.1?
correctness: The performance of Wukong in Table 3 seems to be out of date. For example, the Flickr30K-CN of Wukong is mean recall(MR) 93.4 other than the reported 75.1. The ImageNet zero-shot top-1 acc performance for Wukong is 58% (not 35%) You may update the numbers from wukong latest arxiv version. 
clarity: Yes.
relation to prior work: Yes. The comparison with Wukong may need to be updated.
additional feedback: None.
review_title: A High-Quality Dataset for Chinese V-L Pretraining

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes the largest available Chinese Vision-Language Pre-training dataset -- TaiSu. The contributions of this paper can be summarized threefold:

1. This paper proposes a new framework for automatic dataset cleaning, which includes text based filtering, cross-model semantic based filtering, and image captioning. The proposed framework can continuously improve the quality of the dataset in a self-supervised manner.

2. The authors construct the currently largest (166M) public Chinese cross-modal image-text dataset, TaiSu. They also release the pre-extracted image feature embeddings using RN101 and ViT to facilitate future researchers in the image retrieval community.

3. The experimental results demonstrate the effective of the proposed framework, and bringing improvements for zero-shot image-text retrieval by combining the raw Web and the generated image-text pairs.
strengths: 1. The dataset will be useful towards better evaluating cross-modal image-text tasks, especially for the ones who based on Chinese corpus.

2. The dataset is quite large and diverse in comparison with existing Chinese cross-modal datasets, such as Wudao, M6, and WuKong. The comparison with existing datasets is clear.

3. The authors have done a good job implementing several classes of baselines and the results look interesting.
weaknesses: 1. There could be more discussion regarding the annotation process since there may be possible biases resulting from the annotation process given that different annotators follow different distributions. More discussion regarding checks for the accuracy of annotations would also help the paper.

2. There are some typos in this paper as follows:
    - L63 cross-model → cross-modal
    - L67 cross-model → cross-modal
    - L80 result → results
correctness: Yes. The dataset collection process sounds reasonable to me.
clarity: Overall, I feel it is easy to follow the paper.
relation to prior work: Yes. The comparison with existing datasets is clear.
additional feedback: none.
review_title: review

Meta-review: metareview: The reviewers are positive regarding the high level of the contribution of the work for the NeurIPS 2022 Track Datasets and Benchmarks. The authors properly addressed all reviewers comments and concerns during the rebuttal period.
confidence: 5: The area chair is absolutely certain
Decision: Accept