Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper proposed the first publically available large-scale dataset of image-text pairs. It consists of 5.8 billion pairs in total and the authors successfully reproduced the CLIP models of different sizes using the proposed dataset. The toxic, offensive and pornographic content are tagged rather than removed to encourage research in ﬁelds such as dataset curation. The data bias and limitation is well discussed and notified in the paper. Extensive experiments validate the effectiveness of the proposed paper. 
strengths: A good paper. 

It is very happy to see such a big dataset (5 billion image-text pairs) open-sourced since this can significantly boost the development of language-vision pretraining techniques.

The image-text pairs in the dataset are divided according to the languages, and each part is on a large scale. This provides new opportunities for multilingual and potentially low-resource language researchers.

The pairs are well-tagged with toxic, offensive, and pornographic. This encourages research in ﬁelds such as dataset curation.

Several research works have been conducted based on a pre-released subset LAION-400M, which proves the value of the proposed dataset.

The potential bias caused by such a large dataset is well discussed in the paper.
weaknesses: 1) Figure 3 is missing.

2) The data is filtered by ViT-B/32 in the post-processing part. Is there any chance that the data would be filtered by a much larger model like ViT-L? Besides, as the author claimed that the CLIP model in itself introduces a bias towards LAION, are there any other techniques in the future to avoid this bias and even measure the effect brought by the bias? It would be appreciated if the authors can discuss it.
correctness: Yes. It is constructed in a sound way.
clarity: Yes. It is easy to follow.
relation to prior work: Yes
additional feedback: A good paper. I think it is far above the acceptance bar thus I would give a rating of 8 for this paper.
review_title: The review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors propose a massive 5+ billion instance dataset comprised of image-text pairs. This functions as a novel exceptionally large dataset for the research community, and is the only open-sourced one of its size. On top of its large size, LAION 5B demonstrates novel data aggregation techniques, sophisticated curation pipelines, and extends several key analyses. By providing both a reproducible process and pipeline, as well as open sourcing all their data and code, LAION 5B exhibits a high quality standard of reproducibility and visibility. Furthermore, LAION 5B provides access to a large open source implementation of CLIP. By collecting images using CLIP as a filter, and adequately screening for ethical and moral hazard (while providing important disclosures about this dataset's usage), LAION continues the work of other papers demonstrating that important data quality analysis can be performed at large scales.
strengths: 1. The dataset describes and utilizes a very well defined aggregation methodology. 
2. The paper and dataset employ a useful approach of cosine similarity filtering, employing CLIP to compute these scores.
3. The authors apply several safeguards to prevent unsavory and biased information from affecting the dataset. The application of these safeguards does not, in itself, resolve all issues, but this is a step in the correct direction. 
4. The authors evaluate the models trained on their dataset using a variety of tasks and provide detailed information in the supplementary material regarding these experiments.
5. Ethical concerns have discussion that is equal to that of its predecessor papers. 
6. Allowing access to all data enables independent probing and review of the dataset itself.
7. Technical limitations are well outlined and clearly explained.
8. The supplementary materials (webpage, blog, services, and figures) are a true highlight. This meets the high-water-mark for excellent documentation and supplemental information. 
weaknesses: This paper is certainly significant within the field, however there are some important weaknesses to mention.
1. Investigation of other model architectures and their scaling to a dataset of this size would be useful to include.
2. The inclusion of few-shot learning or a linear probe for downstream evaluation tasks would allow for more comparisons to other datasets.
3. GLIDE and generative works could have been attached in the supplementary or explained in greater detail.
4. While there is a reasonable amount of ethical review, it would be nice to see a greater probe into the nature of the content throughout LAION 5B. There could have been more manual analyses performed. Furthermore, while the authors attempt to mitigate the risk of potential harms in their dataset, the certainty with which they assert their mitigations could lead to confusion. This point is continued in the ethics section. 

correctness: The dataset creation, testing, and evaluation methodology are all sound. This paper appears correct about its claims. The supplementary section also answers questions correctly. 
clarity: The paper is very well written and clearly explained. Particular kudos to the authors as the paper seems easily understandable to even a non-technical audience. This is increasingly important as this paper will likely be cited, paraphrased, and consumed by a different audience than this conference. There are minor typos in the paper (eg. the trailing hyphen in this line"one prominent example is data efficient zero- and few-shot c"), but overall the paper is extremely well written. The supplementary sections as well are very clear. 
relation to prior work: This paper does discuss its relation to prior work quite extensively. Both throughout the paper and in the related work section, the paper continually references related work. 
additional feedback: Great paper, I think supplementary investigations into the dataset as a whole as well as a deeper look into potential ethical hazards would make a great follow up.
review_title: Exceptional large scale dataset with high quality analysis, performance, and review.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper created a new open dataset LAION-5B that contains over 5.85 billion image-text pairs, of which 2.32 billion contain English language. Besides, this work provided a replicable pipeline for the data curation process, based on Common Crawl, including data acquisition, distributed processing of Common Crawl, distributed downloading of images, and post-processing.  To validate the usefulness of LAION-5B in training and analyzing the large-scale vision-and-language models, this work trained the CLIP models of various scales on a subset of the dataset and showed that they match the strong zero-shot and robustness performance of the original CLIP counterparts trained on closed curated data. Finally, the safety, ethical concerns, and data biases were thoroughly discussed.

strengths: (1) The proposed LAION-5B is the largest openly available dataset for training vision-and-language models. 

(2) The data collection and curation process has been comprehensively discussed to support community investigation and further improvement.

(3) Experiments support that CLIP models trained on LAION-400M match the performance of original CLIP models trained on the private dataset.

(4) The dataset will add much value to the multi-modal learning community, regarding 1) being able to train large-scale models, 2) publishing pre-trained models for many downstream tasks, and 3) being able to audit and refine a dataset of this magnitude. 

weaknesses: (1) Why does the CLIP model with ViT-L/14 trained on LAION-400M perform largely worse than the original CLIP model with ViT-L/14, compared to the cases with ViT-B/32 and ViT-B16? 

(2) In the abstract, the authors claimed that “we show successful replication and fine-tuning of foundational models like CLIP and GLIDE using the dataset”. However, I didn’t see the experiments of replication or fine-tuning on GLIDE models.

(3) Figure 2 is difficult to interpret. Maybe add more detailed caption descriptions and also refine the whole diagram to make it more accessible.

(4) What are the WAT files in Common Crawl?

(5) In the Related Work section, I suggest putting the reference numbers before the full stop at the end of a sentence.

correctness: The benchmark is constructed in a sound way and the evaluation methods and experiment design well support the claims about replicating CLIP. However, I didn’t see experiments to support the claims about replicating GLIDE.

clarity: Yes.
relation to prior work: Yes.
additional feedback: N/A
review_title: An interesting large-scale dataset for multi-modal research

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces the LAION-5B dataset, an ultra-large-scale collection of image-text pairs designed for vision-language research. The dataset contains additional metadata about each image-text pair, including language information, resolution, and multiple content flags. The metadata enables researchers to filter the dataset into smaller sub-datasets, for example by language (LAION-2B-en) or resolution (LAION-High-Resolution). 

The contribution of the paper is clear: the dataset is significantly larger than any previous image-text dataset openly available to the public. From the perspective of this reviewer, this is a significant and valuable contribution. There is little doubt that over the past five years, the increase in dataset scale has been an important factor in the large improvements of vision-language models. It seems likely that the trend of increasing model size, dataset size, and training time will continue, as it has in NLP. 

Regarding explicit/questionable content, the authors leaned on the side of tagging and including data rather than removing it from the dataset. This decision enables researchers and practitioners to decide upon their own dataset filters, giving them the greatest amount of flexibility when constructing their data subset. On the other hand, the inclusion of a very large amount of explicit content could lead to some users training models on such content without intending to do so. Many organizations (especially those subject to European privacy laws) will inevitably have to re-filter that data in order to ensure that it complies with relevant policies and regulations. Further ethical/privacy issues are discussed in the ethics section below.
strengths: The two primary strengths of this dataset are its size and its openness. With regard to size, there are no other datasets of similar (or even close to similar) size that are available to the community. Collecting a dataset of this scale requires a significant engineering effort. 

As emphasized throughout the paper, all aspects of the data downloading, processing, and tagging pipeline are open-source. These could be used to reproduce/update the dataset and to collect specialized datasets in the future. In general, the openness of the project is a refreshing change from other large-scale vision-language accessible only by API. 

This paper also makes a few minor contributions which should not go unmentioned:
 - The img2dataset tool is already widely used
 - The watermark detection model is useful to the community
 - The VTAB+ dataset is a slightly larger version of VTAB
weaknesses: The primary weakness of the paper is that there should be a more in-depth analysis of the dataset contents and filtering methodology. 

It is not reasonable to expect the authors to train a huge number of models on the dataset, but it would be appreciated if they provided more dataset-level statistics, especially with regard to the NSFW content. 

For example, one piece of analysis that could improve the paper would be a human review of a small random subset of images (perhaps on the order of 10,000 images). This human review would help us assess the accuracy of the CLIP-based NSFW tagging (as well as the other models such as the watermark tagger). It may also identify some forms of harmful content which are not caught by the current filtering methodology. Relative to the high cost of data processing and model training, this would probably not be a large cost for the project overall. 

A second example would be a more in-depth exploration of the different types of images contained in the dataset. For example, how many of these images are stock photos, how many are memes, how many are advertisements, etc. I would not expect exact numbers, but a broad estimate would still be valuable. 

A third example would be a linguistic/NLP analysis of the captions. For example, how many of the captions are full sentences, how many contain proper nouns, how many contain numbers, how many contain locations, etc. 

Some aspects of the data pipeline could also be enhanced to make the data more accessible and more widely-used. For example, the paper emphasizes in multiple places that it is designed for use primarily in an academic setting. However, as the paper also mentions, training models on data of this scale requires extremely large computational resources which are generally not available to academia (with the exception of a few supercomputers, such as JUWELS). Based on this, it seems likely that in practice, the dataset will be primarily used in industry and industrial research settings. 

To facilitate use of the data in academic settings, it may be valuable to provide some tools for researchers with fewer computational/storage resources. For example, perhaps one could provide a utility for dataset streaming or a small but very high-quality subset of the data for academic use-cases. 
correctness: To the best of my knowledge, all claims made in the dataset are correct. The dataset pipeline is open-source and the quantitative results are reproducible using Open-CLIP. 
clarity: The paper is well written. 
relation to prior work: The dataset clearly discusses how it relates to prior work. 
additional feedback: At least from this reviewer's perspective, the extent to which this extreme scale (i.e. 5B+ images) is necessary for training large-scale vision-language understanding is not yet clear. In other words, it is not clear from the paper that it is possible to achieve anything with 5B+ images that was impossible with 400M images. Additionally, if I read the paper correctly, I do not believe that the authors actually trained a model on the full LAION-5B dataset. Such a model would necessarily have to be multilingual (and may also be valuable to the multilingual NLP community), and there are no discussions of the performance of multilingual vision-language models. 

For English-only tasks, Table 2 shows results for LAION-2B-en compared to LAION-400M and shows significant improvements across the board. However, it is not clear whether the improvements should be attributed to the dataset size or the additional training time/compute used to train the 5B-en model. Table 4 in the Appendix shows that the (ViT-B/32) 2B model was trained for 210 hours whereas the (ViT-B/32) 400M model was trained for 36 hours. If I am interpreting this correctly, the results are not comparable and they should not be presented in the same table as they are currently presented. 

Indeed, Figure 4 shows that at the same level of compute, a (CLIP-ViT-B/16) model trained on 400M and (CLIP-ViT-B/32) model trained on LAION-2B-en deliver approximately the same downstream performance. It would be very helpful to have an apples-to-apples comparison of two models trained on the 2B and 400M subsets using the same set of hyperparameters and the same amount of total compute.

Additional small questions
 * The caption and description of Figure 4 mention a "log-log plot". Am I confused or is this figure a log-linear plot?
review_title: A strong contribution to the vision-language community which could benefit from additional analysis

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The main contribution is a new large-scale open-source 5.85 billion image-text pair dataset named LAION-5B. The dataset is a folloup to LAION-400M. Both datasets were crawled from the web and filtered with CLIP with the goal to keep only pairs where images and text are related to each other.

The authors describe the collection process and properties of the resulting dataset. Next, they explain and show usage examples for LAION-5B and LAION-400M, both new contributions by the authors (subdataset generation) and contributions from other works (CLIP reproduction, generative vision-language tasks).

The main experiments in this work are reproductions of various CLIP models on the LAION-400M and LAION-2B-en datasets and their comparison to the original CLIP model trained on the private WebImageText dataset (CLIP WIT).

The authors evaluate the models on zero-shot top-1 classification accuracy (%) on ImageNet and 4 datasets with distribution shifts compared to ImageNet (e.g. ObjectNet was "collected to intentionally show objects from new viewpoints on new backgrounds", quote from https://objectnet.dev/ ).

Additionally, the authors scale up to VTAB benchmark with 19 tasks to VTAB+ with 35 tasks and evaluate on this benchmark. VTAB and VTAB+ contain natural tasks (classical vision like CIFAR), specialized tasks (remote sensing like areal images or medical images) and structured tasks which require counting objects or predicting their orientation.

Zero-shot classification evaluation is done using predefined text prompts for each class and selecting the class where the prompt has the highest predicted similarity to the image.

Experimental results show: 1) It is possible to match the performance of the original CLIP VIT model when training on LAION datasets. 2) The model trained on 2B images improves over 400M images. 3) Zero-shot performance of models does not seem to saturate yet when increasing compute, so further upscaling may improve results further.

Finally there is an extensive discussion on technical limitations, safety, ethics and biases. The authors choose to flag images that are predicted to be NSFW instead of removing them and remove only illegal images.
strengths: Open-sourcing a dataset of this scale will enable many researchers to work on vision-text problems. This is especially important since the latest SOTA models in many vision-text fields are all models pretrained on huge amounts of image-text or video-text pairs. In this context, another strength is to not only open-source the dataset but also the collection process and trained models.

The creation of the VTAB+ benchmark is not discussed in detail in the paper, but as a superset of VTAB it can be assumed that VTAB+ is a better estimate of zero-shot classification performance in various tasks than VTAB.

A single model was pretrained on LAION-2B-en and shows the usefulness of the dataset over LAION-400M. This is an achievement given the scale of model, datasets and compute required (400 A100 GPUs for 88 hours). Downstream experiments and their analysis are detailed.

The ethical implications are well discussed and the authors explain their choices on releasing a dataset containing potentially harmful content in a convincing manner. The benefits of the work outweigh the risks.
weaknesses: In table 2, the original CLIP VIT model is compared with the reproduction on LAION. The authors state in the introduction "numerous groups have further increased CLIP’s generalization ability". My question would be why there is no comparison to these improved CLIP models in this table 2, but only a comparison to the original model. The reasons for choosing only this model and no others should be stated explicitly, so that the reader can assert that the comparison and claims are fair and valid.

In chapter 5.2, all models except one are trained on LAION-400M and only one model is trained on the new LAION-2B-en. The reasons on why the models are not trained on LAION-2B-en (e.g. too high compute cost) should be stated explicitly. In table 2, especially the experiment of model L/14 pretrained on LAION-2B-en is missing, as there the original CLIP WIT model significantly outperforms the CLIP LAION-400M model and the best performance of the original L/14 CLIP WIT is never reached.

The new VTAB+ benchmark is mentioned briefly but lacking details. What are the reasons for creating it, why is it more useful than VTAB? How are the text prompts exactly created, how does the downstream evaluation pipeline look exactly? Unless I missed it, the benchmark and evaluation code is not public yet. At least the supplemental should contain these details in a separate chapter as well as an outlook on when/how the benchmark and evaluation code will be made public, since the "open-sourced-ness" of this work is its main strength.
correctness: The experimental claims are correct and can be reproduced with the OpenCLIP code. 

The dataset construction process is sound and also documented in detail.

The experiments are appropriate and correct. As stated in the weaknesses, the claims on the VTAB+ benchmark are currently hard to understand and verify due to missing details and code but seem believable and sound.
clarity: Yes
relation to prior work: Yes
additional feedback: Typo on page 6: "Additionnaly"
review_title: Open-source image-text pair dataset of unprecented scale

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: - Creation and distribution of a dataset composed of 5.8B pairs of images and texts – in several languages - in order to propose to the research community an artifact that has not been shared by scientific publications that have shown the usefulness of such a dataset. 
- Sharing of different features related to this dataset facilitating its use: web interface allowing to search in it, sharing of several nearest neighbor indices, detection scores for watermark, NSFW, and toxic content detection 
- Experiments showing that the constituted dataset allows to reach similar performances as the ones reported by [34] 
strengths: - The authors have built and made available to the research community the largest dataset of image and text pairs. Moreover this dataset is not limited to 2.32 billion pairs of image and english descriptions but also contains 2 other subsets : 2.26 billion multilingual descriptions, and 1.27 billion salient but non-concrete language samples. 
- The release of such an artifact is particularly important for the research community. As the authors indicated in their paper, many recent scientific advances have demonstrated the usefulness of datasets of this nature of several hundred thousand image and text pairs. Nevertheless in these previous research publications these datasets have not been released and the information about their content is limited. This dataset, whose constitution is technically complicated and costly, will allow more research groups to work on 1) models requiring text and image pairs as input and 2) audit and refinement of this dataset.
- The very large size of the dataset allows future searches to look for the best possible cleaning filters or filters.
- A series of experiments showing that the content of the dataset does indeed allow access to similar performances of CLIP [34].
- The distribution of all the codes used to constitute such a dataset. This represents huge engineering work.
- The addition of metadata for each example of the dataset (LANGUAGE, NSFW, similarity, watermark, ...).
weaknesses:  I very much appreciate the presence of section 5: "Experiments Validating the Utility of LAION-5B". Nevertheless, after reading Figure 4 and Table 2, three questions come to my mind:
1. why not have carried out the training of B/16, B/16+ and L/14 on LAION-2B-en? 
2. what would have been the performance of B/32, B/16, B/16+ and L/14 if the training with LAION-400m had been continued during several epochs in order to reach the same amount of compute as the training on LAION-2B-en?
3. Why the Lit method has been used for training CLIP models on Laoin? It makes the comparison of their performance with the original CLIP model less impactful than if the training method had been identical.
correctness: The authors' claims seem to me to be correct. The limitations that could be seen with this method - notably the use of the CLIP score and the biases that could thus be reproduced or the use of data from Common Crawl - are well discussed by the authors.
clarity: The paper is overall well written. 

However, there are a few points that I think could use some clarification:
-  Figure 2 could in my opinion be improved. As it stands, the reader may find it difficult to associate a text with the corresponding arrow/symbol. In this figure there is a mention of "Bloom filters" which are not mentioned (under this term) in the rest of the paper and the step of calculating the CLIP scores is not clearly indicated (even if one could guess that it is the "GPU Inference" step). One way to improve the reading of this scheme might be to make the temporal reading of the pipeline from left to right or from top to bottom.
- LAION-400M is mentioned both as a reference [42] and as a subset of LAION-5B. Indeed, [42] is mentioned in the section "3. Collection Methodology" as a source of inspiration to fix some pipeline filters but also used in the section "5. Experiments Validating the Utility of LAION-5B", notably in section 5.2.1 where the authors state that "CLIP models trained on LAION match in their robustness original CLIP". In order to fully understand the results shown in Table 2, I think the reader would benefit a lot from knowing the relationship between LAION-400M and LAION-5b: is it a uniform random subset? If not, what are the specific criteria that distinguish this subset?
relation to prior work: - The authors have well discussed the relationship of their work with the previous contributions in section 2. 
- As mentioned in the review section "Clarity", the relation of this paper with reference [42] would certainly benefit from being clarified.
- In section 2, under "Image-Text Datasets", it seems to me that the set of datasets mentioned is not made available to the scientific community. It would seem to me that this should be specified in this section to justify the impact of the contribution made by LAION-5b.
additional feedback: Thanks to the authors for creating and sharing these artifacts which I think will be very useful for future research. 

I have distilled several comments, suggestions for improvement and my questions in the dedicated sections above.

You indicated that "We release the first open-source reproduction of CLIP models up to ViT L/14 scale achieving competitive zero-shot classification and retrieval", I may have missed something but I didn't find the points of this trained model, do you have/plan to open source it?
review_title: A good paper releasing 5.8B image-text pairs dataset potentially very impactful for research on multimodal models

Meta-review: metareview: The reviewers agree that this dataset represents a huge undertaking that offers potential opportunities for various applications. The paper also provides insightful experiments across a series of use cases, models, and tasks. Several reviewers also noted the potential risks of the dataset and the potential for misuse -- although there is also an acknowledgment of these issues from the authors' side, and active steps to mitigate these risks. Authors are encouraged to incorporate the reviewer comments, especially some potentially factually incorrect statements raised by ethical reviewers surrounding IRB practices. 

As it stands, this dataset should be of wide interest across various communities, including computer vision and natural language understanding. Given the careful considerations of the reviewers and the AC, this paper is recommended for acceptance in the program. The AC also recommends a more thorough review of similar resources before COCO and VisualGenome. There is the Pascal-1k, Flickr-8k, Flickr-30k, SBU-1M, IAPR-TC12 (ImageClef), which preceded most other works listed here. 


confidence: 5: The area chair is absolutely certain
Decision: Accept