Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces the Reddit Photo Critique Dataset  (RPCD), a new dataset of 74K high-resolution images and 220 associated comments from photography groups on Reddit. An aesthetic score / judgment is assigned to each image based on the sentiment polarity score of the critique of the image contained in the comments. The sentiment polarity score is assigned automatically calculated using TwitterRoberta. The paper also includes the results of a series of experiments that the authors conducted to illustrate the potential uses of the dataset. 
strengths: Compared to already existing datasets, RPCD offers some advantages, including 1) higher image resolution and 2) higher average word per comment (albeit only slightly higher than PCCD). The paper (including the appendices) also presents the methodology and the results of the experiments in a clear way.

weaknesses: The authors already identified some of the limitations of the paper (p. 9), such as the limited number of comments per image and the risks associated with the automated labelling of aesthetic judgment in the dataset. Another weakness of this paper is that the use of the sentiment polarity of the comments as a proxy for the image aesthetic score is not justified in a clear way. In particular, compared to the other datasets presented in Table 1, RPCD appears to be limited by the use of a rating scale on a 0-1 range (based on the sentiment polarity) rather than a 0-10 scale. Some analysis of the comparison of the different scales would have been useful.
correctness: The dataset is constructed in a sound way and the experiments are designed and reported in an appropriate way. However, compared to other benchmark dataset for aesthetic image captioning, this dataset is characterised by the fact that the aesthetic score was automatically assigned (machine-based annotation). A more detailed discussion of the limitations of this approach (which are acknowledged by the authors) would have been useful
clarity: The paper is written in a clear way.
relation to prior work: The paper clearly discusses how this work relates to previous contributions and it includes experiments comparing RPCD with other banchmark datasets.
additional feedback: No additional feedback.
review_title: A sound contribution to Automated Image Aesthetics Assessment

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces the RPCD which consists of 74k high-quality photos paired with critique comments from the r/photocritique subreddit. The authors introduce an algorithm for extracting quantitative aesthetic ratings from these comments and evaluate models’ ability to predict these scores
strengths: - The provided dataset has the potential to be quite useful to the computer vision and NLP communities. Including both scalar values as well as full comments allows for a wide range of future work to be pursued.
- The dataset appears to be thoughtfully constructed and the development process is well documented in the text.
- The paper is very well written: It establishes sufficient motivation, carefully outlines the thought process of the authors, and answers many potential questions a reader may have. The text includes a nice analysis of limitations and future work.
- The figures, graphs, and examples are well chosen.
weaknesses: - It would be interesting to compare the authors’ approach to assigning a scalar value measuring the aesthetics of an image to the scores a human would choose to assign to the image when prompted, given that the latter seems to be the primary method for labeling among most aesthetics datasets.
- As noted by the authors, given the high subjectivity of an aesthetic evaluation of a photograph it is unlikely that the average of 3 comments per image fully captures the distribution of human sentiment for a given data point.

correctness: Yes.
clarity: Yes.
relation to prior work: Yes, although it would be potentially helpful to include prior work from a broader range of research areas. For example, citing relevant work in sentiment prediction outside image aesthetic assessment.
additional feedback: N/A
review_title: Paper introduces a high-impact dataset that addresses existing issues in aesthetic analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents the Reddit Photo Critique Dataset (RPCD) consisting of image and photo critiques
tuples. The dataset was collected by crawling posts from a community where people posted positive and negative comments on image aesthetics. Detailed data collection techniques/processes and the aesthetics evaluation are provided. Overall, the paper is well-written and easy to follow. Aesthetics is always an important topic in design studies and computer vision studies. Therefore, the dataset may have great application potential.  
strengths: 1. The work looks technically sound.
2. Technical details of data collection, implementation and evlauation are provided. 
3. The paper is well-written and easy to follow
weaknesses: 1. Contributions of the work can be further elaborated
2. Ethics issues have been discussed but some issues are not clear to me. 
3. Data access can be arranged in a more user friendly way. 
correctness: The submissison looks to me technically sound. 
clarity: Yes
relation to prior work: Yes. Table 1 also provides a good comparison of the proposed dataset with the benchmarked datasets.  
additional feedback: NA
review_title: A very good dataset paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper introduces a dataset for the purposes of image aesthetic assessment. Specifically, the dataset called Reditt photo Critique Dataset (RPID) consists of 74k images with accompanying 220 k (user) comments /critiques which serve as a richer source of aesthetic characteristics than mere ratings which can be subjective and biased.  The main distinction of the proposed photo critique dataset when compared to existing photo critique dataset is that the images are mostly Full HD. Apart from this, it is bigger and richer in terms of comments and number of images.  The paper also proposes an image ranking algorithm based on the aesthetic assessment data available. 
strengths: 1. A new photo critique dataset that is considerably larger and richer ( in terms of comments) than existing ones

2. A simple ranking algorithm for image aesthetic assessment

3. Reasonable experimental validation and comparison with other datasets. 
weaknesses: 1. There is not sufficient details pertaining to dataset documentation - as outlined in datasheets/artsheets paper. Please also refer to comments in the sections below.  Please see detailed comments under ethics and documentation sections. 

2. Presentation clarity can be improved- fixing typos, elaborating certain sections, etc. - please see comments under clarity. 
correctness: seems ok. 
clarity: - There is reference to Appendix A and B, but these are not included with the submission. 

- some typos can be found in the paper- e.g., In Section 4.1, "containsts"  should be contains

- Certain sections could be elaborated to provide more clarity and details. For e.g., sec 3.3 can be elaborated to emphasize how the proposed method is better than existing approaches - how the method allows weak supervision, which two aesthetic tasks does it combine and why that is useful,  and on what grounds it is a strong indicator of human judgement. 
relation to prior work: The paper needs to be contextualized with respect to prior dataset documentation papers such as datasheets for datasets and art sheets for art datasets. 
additional feedback: Please see responses to above questions.
review_title: A new photo critique dataset 

Meta-review: metareview: This paper introduces a new dataset for critiquing photos mining ~74K from the r/photocritique subreddit and an algorithm for transforming the reddit comments into a judgement score between 0-1. They then introduce the task of predicting these judgement scores from the images. All reviewers find this to be a useful contribution to the community, and they appreciate the thoughtful process behind the dataset creation process as well as the thorough documentation of this process in the paper. All reviewers also find the manuscript well written and easy to understand. The authors have also addressed all points brought up by the ethics review.

confidence: 5: The area chair is absolutely certain
Decision: Accept