Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: null
review_content: I commend the authors for thinking through the ethical considerations, making their various choices explicit and documenting them in a data sheet. While there are still concerns, in particular the potential presence of stereotypes and derogatory images, the presence of children's faces, I have not seen this level of diligence in other large scale datasets and see this due diligence as a positive step forward. I think reviewer ugYV has very thoroughly listed out the concerns and their gravity, and I agree with all of their analyses. Unfortunately, I believe the computer vision community is extremely behind in understanding this, and compared to prior datasets in the community, this one does a much better job of thinking through the concerns, attempting to mitigate them and making explicit their choices. 

I have some additional comments below

“We only collect 100 posts with images from three image hosting domains: Reddit (i.redd.it), Imgur (i.imgur.com), and Flickr (staticflickr.com). "

Be aware that IBM, Microsoft etc are in a lawsuit for violating Illinois biometric laws by creating the diverse faces dataset using Flickr and applying it to applications of automatic facial analysis. Even though people uploaded their photos to Flickr they did not consent to having it used in large scale automated facial analysis tools. As mentioned by Reviewer ugYV below, even though the cost of curation can seem high, the cost/potential harm of not curating, and even 1 child's face being present in the dataset can be really high. While all the steps the authors have taken are much more than prior computer vision datasets, and probably current datasets that may be published in computer vision venues, I suggest that the authors do as much as they can to ensure that there are zero, rather than potentially a few, children's faces (e.g.)

“Subreddit sizes highly correlate with their 151 popularity on Reddit, which depends on what humans find interesting to view and share on social media.”

I suggest that the authors change this to qualify to Reddit users rather than all humans. They mention in their paper that size doesn't guarantee diversity. Similarly, Reddit users' interests can't be generalized to "what humans find interesting."

I suggest some way of limiting the dataset usage by either having a form where people request to use the dataset for specific purposes, a lisence etc. 
review_title: Dataset with much more due diligence compared to prior computer vision datasets

final_score: rating: null
review_content: This work includes user-generated content that can include privacy and consent issues, including for vulnerable populations (faces, and children).  The concerns mentioned also include NSFW images and harmful stereotypes. 

The authors do a fantastic job at highlighting some of the key ethical concerns in this work (consent, privacy, stereotypes, demographics) and do due diligence on addressing them.  Another ethical concern that the authors speak to, but don't name directly, has to do with "proliferation" and amplification of images that were not intended to be shared outside of reddit.  (This concern is implicit in the other listed concerns).  The authors address this by utilizing urls, rather than the images themselves, which allows reddit users to delete their content and have that content correspondingly deleted in the dataset.

The authors even provide a datasheet!  And additional metadata about the dataset beyond that.  Would be great to make sure these are easily accessible at the host url (https://vhosts.eecs.umich.edu/kdexd/redcaps/).

Regarding NSFW images, I wouldn't agree that these bring up serious ethical concerns as long as the other related concerns around consent and privacy are reasonably addressed, as they seem to be.

Regarding demographic skews and stereotypes, as long as these issues are made clear, they are reasonable aspects of this dataset. If for no other reason, these provide insight into the skews within Reddit, and are useful for describing what Reddit is like.  In my opinion, it is the work of people who use this dataset to train models, rather than the dataset creators, to address skews and stereotypes according to their models' intended use.

Since the initial reviews, the authors have done even more work on addressing the ethically problematic issues. 

I think this dataset is reasonable to release, as long as the other reviews agree the work as a whole is sound.
review_title: An Official Ethics Review Comment

final_score: rating: null
review_content: This is a comment from one of the official ethics reviewers.

The authors should be commended for their original submission's focus on the issues raised by Prabhu and Birhane. Reviewer ugYV should be equally commended for being so fastidious in their review and back-and-forth with the authors, who were thoughtful and responsive throughout. All the actions that the authors have taken (not just promised), including removing the 2.3% of images with detected people, have continued to strengthen the ethical aspects of the work. 

The remaining issue is the calculus of the NSFW filtering cost ($50K) and ethical benefit. Note that this will not be a one-time cost, but an ongoing one as long as the dataset is maintained. Although we may not like to assign monetary amounts to ethical benefits, there is no way around it. I do sympathize with the thought that we should have exactly zero NSFW examples in the dataset, but I also believe that nothing in real life is ever perfect. I am satisfied by the current efforts of the authors and given the strong technical reviews, am not opposed to this work being published.
review_title: An official ethics comment

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper introduces a large-scale image-text pair datasets called RedCaps, sourced from Reddits. The paper argues for advantages of Reddits as a source of data compared with previous ones such as search engines. Based on the image-text dataset, the authors conducts experiments and show that the learned representation is transferrable to many downstream tasks. The dataset can also be used for image captioning .
strengths: + RedCaps is built upon Reddits where the image-text pair is generated by users and thus potentially simplifies data collection procedure and increases data quality. 

+ The paper conducts extensive experiments on visual-textual representation learning. In particular, It applies the learned representation to ten different downstream visual recognition tasks.

+ The data collection procedure is clearly described. The pipeline including image post filtering and caption sanitization is discussed in details. 

+ The paper discusses ethical concerns, including user contents, privacy (face/children), and harmful stereotypes.  It also provides solutions to alleviate such ethical concerns. 
weaknesses: - In order to align with user content,  posts deleted from Reddit will be removed from RedCaps. This may be problematic when comparing a new method with a previous one (since data used to training may be less).


====after rebuttal====

The authors have well addressed my concerns. It is a good idea to only include the posts that exist at least 6 months in the dataset since those are unlikely to be deleted. 
correctness: The data collection procedure is clearly described. The pipeline including image post filtering and caption sanitization is discussed in details. 
clarity: Yes. The paper is well written.
relation to prior work: Yes, the paper clearly discusses the relation to prior works.
additional feedback: No.
review_title: Review for RedCaps

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a dataset called RedCaps which consists of 11.7 Million image-text pairs collected from 256 subreddits. The data includes popular day-to-day items in social media that are shared by Reddit users regularly. The authors describe their data collection pipeline, which includes selecting subreddits based on the volume of image posts, filtering out posts based on upvotes, and cleaning captions for better image-text mapping. In the paper, the authors have reported different corpus statistics and have used the dataset for different vision-language tasks. The evaluations show that the RedCaps dataset is useful for pre-training feature extractors that can be used for multiple downstream tasks like image captioning, zero-shot image classification, etc.
strengths: 1. The dataset collection is well described and motivated. Reddit has traditionally been a source of high-quality language content and is relatively well moderated. 

2. The authors clearly mention the dataset collection procedure. The subreddit selection ensures that the content does not contain memes, sketches, and, most importantly, too many celebrity pictures. The caption sanitization step also ensures that the text is processed to remove unwanted characters, emojis, non-Latin characters, insta-id, self-promoting text, etc.

3. The dataset statistics are adequately reported.

4. The ethical concerns are discussed reasonably well. The authors have removed children's faces using automatic age detection methods (some examples will probably remain) and have also removed harmful text content using several techniques. They also will have a protocol to remove content uploaded without consent manually once brought to their notice. 

5. The experiments conducted help establish that the dataset can be used and meaningful signals can be learned from it for several tasks.

6. This will be one of the largest publicly available corpus for vision-language and can benefit the community.
weaknesses: 1. While pretraining on RedCaps is hugely beneficial for Zero-shot classification, and low shot classification, the scores in other tasks are very similar to CC-3M while CC-3M having only 1/4th data. Is there any specific reason for this? Does CC-3M has data that is better suited for these tasks?

2. Qualitative results for some tasks like Instance segmentation are missing from both the main paper and the supplementary.
correctness: The paper is technically sound.
clarity: The paper is well written
relation to prior work: Prior work is mentioned adequately
additional feedback: Currently, the dataset does not have any comparison with CC-12M. I believe CC-12M was released just before the submission, and it might have been impossible to add comparisons with it during the first-round submission. Since CC-12M has more comparable size to RedCaps, adding comparison results for it will in the main paper or supplementary materials will be insightful.
review_title: A large scale vision-language corpus collected from subreddits.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors present a new large-scale image-text dataset collected from subreddits devoted to images. This dataset provides a set of images and captions more actively and realistically curated than typical image datasets, without complex automated filtering. The authors propose several methods to mitigate bias concerns and show evidence that models trained on this dataset produce more attractive captions.

By way of framing, my expertise is in ethical computer vision - hence the focus on ethics in this review. I am very confident in the ethics-related critiques, less so in the critiques related to data quality for image captioning and transfer learning tasks.

---
EDIT (07/16/2021): I'm upgrading my review to 5 (marginally below acceptance) thanks to the authors' responses. The leftover margin is delineated in the Ethics section - if these concerns were fully resolved, my rating would change to >7.
strengths: 1. **[Major] Language diversity & curation**. I find the author's claim that the captions and images are more "conversational, humorous, emotional, and generally more diverse" than typical HTML alt-text - I agree that the authors have identified a uniquely rich source for images and image captions compared to prior datasets. Linguistic statistics in 3.1 back this claim.
2. **[Minor] Human evaluation**. The authors conduct a helpful study to compare the readability/attractiveness of a captioning model trained on their dataset and find an improvement over previous models. The use of Reddit scoring is another unique component of the dataset that many developers & researchers might find useful.
3. **[Minor] Documentation and disclaimers**. The authors cite and follow relevant data ethics literature (especially Prabhu & Birhane) in presenting the risks of their data to potential data users (though I discuss later how those concerns should be better addressed).
weaknesses: 1. **[Major] As yet unaddressed ethical concerns.** The authors correctly and clearly identify many ethical concerns and propose several solutions (mostly automated filtering) that they *plan* to implement before public release at the end of 2021, but have only forecasted the results of those mitigation techniques. The paper lacks a thorough test of the efficacy of those solutions with respect to both dataset composition (does the dataset still contain a large number of NSFW, stereotypical, offensive or otherwise problematic images?) and model performance (do models trained on this dataset exhibit problematic behavior?). The Ethics section of this review raises further concerns about the sufficiency of the proposed techniques and the ways the authors choose to balance ethical trade-offs.
2. **[Minor] Usefulness of stylistic captions for other tasks.** While this dataset has the distinct advantage of relying on stylized, conversational captioning, this also poses a weakness that the authors don't clearly state: captions produced by a model trained on this dataset may be less useful for less playful image captioning applications, or applications where phrases like "itap" are out of context - though reading the paper, there is some indication that the dataset is rich enough to enable a model to discern between contexts given enough subreddits (e.g. Fig. 8). Asking the human evaluators to mark which caption they prefer doesn't capture which caption is more useful or informative - this is a limitation. An increase in stylistic speech also make things more difficult for non-English speakers.

EDIT (07/16/2021): the authors have helped clarify concern (2) somewhat. See the ethics section for an update on (1).
correctness: As far as I can tell, the collection and sanitization methods presented are correct and reproducible. The collection method is laudably simple and easy to follow. The authors take care to include failsafe measures in case of bad data - for example, the authors use both moderator-created NSFW ratings and NSFW ratings from their own automated classification model.
clarity: The paper is clearly written and easy to follow. The same applies to the documentation. The figures are useful, well annotated, and clearly relevant.
relation to prior work: In 5, the authors briefly summarize other image-text datasets, but only hint at the differences in their own work. The introduction (1) is more clear: this dataset is unique in its size, continuity, and conversationality.

Given the large portion of the work spent on ethical considerations (3.2), the authors may also consider citing work around the construction of ethical datasets, or other image datasets constructed with ethics in mind (e.g. [FairFace](https://github.com/joojs/fairface), or the revised version of ImageNet). (The authors did a good job of applying a relevant method in 3.2, but more discussion might be warranted.)
additional feedback: The dataset website looks very nice! Well done.
review_title: Promising contribution with major ethical concerns

Decision: Accept