Venue: NeurIPS 2022 Workshop SVRHM

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper tries to show that visual aesthetic experience is a multi-modal phenomenon, shaped by both visual perception and language. For this, the authors use computational modelling to disentangle the role of visual and linguistic features in explaining human aesthetic ratings of natural images. The primary findings are:

1.Visual features alone can explain around 75 % of the variance in average human aesthetic ratings but adding language features can explain up to 87% of the variance in human ratings.
2.To further understand how visual and language features interact to improve the prediction power of aesthetic ratings, the authors used a technique where they use the embedding learned by a CLIP model to disentangle the contributions of visual and language features. For this, they extract a prefix embedding that captures the features from both vision and language. Further, they use this embedding to generate captions of images using GPT2. Now they compare the predictive power of layer activations in GPT 2 model, when it is fed (a) the prefix embedding along with the caption, (b) the prefix embedding without the caption, and (c) the caption alone. They found that conditions (a) and (b) resulted in equal predictive power as CLIP (~85%). However, (c) only had a predictive power of 38.6%. The main take away is that language features can capture a non-trivial portion of variance in aesthetic ratings.

Pros:

1.This paper provides a novel way to approach a very complex topic and demonstrates how one can harness the power of deep learning models to infer mechanisms of how humans perceive aesthetics. As such the topic is very relevant to the theme of the workshop and is an important line of work. 
2.The paper is well-written, and the results are easy to understand.
3.The design, implementation and analysis were expertly conducted.  

Comment:

1.Lines 122-123: This finding appears to be an artifact of how the analysis was set up. The prefix embedding, which comes from CLIP, presumably contains information that correlates with both visual and language content in the images (since that is how CLIP is trained). The fact that the predictive power of “prefix embedding only” matches that of CLIP suggests that the CLIP-Cap method can harness the full expressive power of CLIP in terms of explaining variation in aesthetic ratings.  Now, when GPT 2 is fed the “prefix embedding only”, it will generate a caption that is based on this information. Thus, the information captured by the caption needs to be equal to or a subset of the information present in the prefix embedding. The “caption only” condition only accounted for 38.6% of the variance in aesthetics ratings. This result shows that converting the preferred embedding to a caption by GPT 2 results in partial loss of information reducing the predictive power from 84.8% to 38.6%. Since the caption only condition captures a subset of the variance explained by the “prefix embedding” condition, the “caption+prefix embedding” condition as well as the “prefix embedding only” conditions are expected to have the same predictive power (as was shown in the paper). Based on this argument, one might draw the same conclusions as in the paper without having to use the “caption+prefix embedding” condition.  While this does not change any findings of the paper, it may require re-writing the section from line 122-126. 

Minor error:

1. Figure 2 caption: The facet references in the text seem to be reversed. The left facet consists of the red trace, while the right facet consists of the green, blue and purple traces. 

review_title: Novel way to disambiguate the role of visual and language features in aesthetic judgments

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper aims to predict human ratings of image aesthetics, using features extracted from several different deep neural network models pre-trained with different objectives, to dissociate the contributions of vision and language in aesthetic judgment. The paper is extremely well written and the model comparisons have been very thoughtfully chosen to control for as many factors as possible.  

Below, I offer a few suggestions / comments, which I hope will be helpful for strengthening this very interesting paper.

  - The authors allude to the "inherent subjectivity of affective self-report" but avoid this subjectivity by relying on group-averaged aesthetic judgments. While it is certainly fascinating and unexpected that these DNN models can explain so much of the variance in these group averaged aesthetic judgments, this begs the question of how much interesting variance is being excluded by the group-averaging process. That is, would these vision-trained or vision&language-trained models do as well at predicting individual subjects' aesthetic judgments? 
  - I appreciate the authors' using the SLIP family of models to fairly compare training objectives, while holding constant the effect of dataset size and architecture. 
  - I would be curious how the relative performance of these models at the aesthetic prediction relates to the relative performance of these models at a more standard computer vision task (e.g. Imagenet top-1 performance). This might help to determine whether the aesthetic judgment behavioral task is actually capturing anything more than a standard visual discrimination/categorization task.
  - The experiment using the CLIP-GPT2-caption only embedding is quite interesting and helps to make the argument about the "ineffability" of beauty judgments, though I'm not entirely certain it is a fair comparison given the number of additional steps required to get that representation. Perhaps asking humans to generate a verbal caption and then using that as input to a GPT2 embedding might more directly capture the effect of language alone. It would also be interesting to see how humans rate the aesthetics of an image caption, without being able to see the image it is associated with, and how consistent these ratings would be with human ratings of the image itself.
review_title: Fascinating approach, many questions remain

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper takes a dataset of human ratings of aesthetics and tries to partial out the difference between visual and linguistic representations by using neural networks trained with self-supervision and those trained with language alignment. 

Quality: 
* One methodological concern is the use of only a single $\lambda$ parameter across all of the models/layers, chosen based on another model. Depending on the distribution of model activations this choice could dramatically change the predictions, and I was left wondering if that alone could explain the results in Figure 2 and Figure 3.   
* The statement in lines 50-51 seems unfounded -- it seems like an open question of whether this type of decoding "mimics" how information is decoded from neurophysiological signals (and even if true, it is likely a simplification)?

Clarity: 
* The initial results on Unimodel and Multimodel visual models are fairly clear (and the Schematic in Figure 1S helps) however the section on “Language Models via Captions” is very hard to follow. In fact, the authors refer to their own approach as “somewhat oblique” in the appendix (which still did not help the clarity or motivation of this section). Including a schematic which has the inputs/outputs of the different models that are chained together would improve the paper. 
* Why are there multiple lines for each model on Figure 1B Left? This is not explained in the caption. 

Originality:
* Using different training of these models to capture aspects of language vs. perception and relate this to aesthetics seems original.  
* On line 69 the authors state that their unimodal vision model is “recapitulating previous work” but do not provide a citation. What previous work are they referring to and how does their work extend on it? 

Significance: 
* Building image-computable models of human judgements of complex features such as “aesthetics” which are hard to directly define is an interesting direction, and using the features of neural networks for this purpose seems promising. 
* Fully separating language from vision is in these models seems tricky to nail down, because it has been shown that the readouts of the final layers of visual trained models are able to support tasks such as classification, and in fact the models that are released publicly are chosen to have this feature. Because of this, the statement that their results show “a general methodology for disambiguating perceptual and linguistic abstractions” seems too strong. 
review_title: An attempt at disentangling vision and language with differences in training

final_score: rating: 7: Good paper, accept
review_content: This paper investigates the extent to which neural network representations can be used to predict human aesthetic ratings for images. It finds that a SEER-pretrained RegNet64 model can explain 75% of the explainable variance in this dataset, but OpenAI's CLIP models perform even better, explaining 87% of the variance. A set of experiments with models trained on the same dataset using either SimCLR, CLIP, or SLIP training shows that language pretraining generally provides a small but significant boost in accuracy. However, using CLIP-Cap to try to generate new captions from the images does not lead to any improvement.

Overall, this is an interesting study and the experimental setup is appropriate. I appreciate that the authors have controlled for the effect of training data by investigating SLIP models. A longer version of this work might attempt to provide more interpretable insights into what aspects of aesthetics can and cannot be captured by neural network representations or through text, but I recognize that there are space constraints given the venue.

My only major concern is that, from the appendix, it seems like, when fitting ridge regression models, the authors consider only a single value of the regularization hyperparameter, which was selected using a different network from the networks studied here. However, representations from different networks could require different regularization strengths to achieve optimal predictive power. It would be better to use some sort of nested cross-validation to pick the optimal regularization strength for each representation. It would also suffice to validate that the selected regularization strength is close to optimal for all models, if that is indeed the case.

Minor nits:

- According to L44, the dataset investigated has valence, arousal, and aesthetic ratings. Why not repeat this analysis for the valence and arousal ratings as well?
- In Figure 1B (left), it seems like the final layer is best for many models, but I can't tell if it's best for all models. Since which layer is optimal seems like the most important piece of information in the plot, it would be helpful to clarify it, either visually or in the text.
- The significance testing on L100 resamples the human subject pool but doesn't account for variance due to initialization used for pretraining, so it's not useful if the goal is to compare SimCLR pretraining to SLIP pretraining (vs. comparing these specific SimCLR models to these specific SLIP models). However, eyeballing Figure 1B, the difference between SimCLR and SLIP is probably significant for each of the model sizes individually, which probably means it is robust to initialization.
- L102 refers to "Figure 1B-C" but there is no Figure 1C.
- I am not entirely convinced that the sparse random projection doesn't affect the results. Clearly, with enough dimensions, $R^2$ converges to the correct value, but the current analysis bounds only the distortion of the embeddings and not of $R^2$ as a function of the number of dimensions, and some additional analysis is necessary to bound the error in $R^2$. It would be good to derive a bound for $R^2$, to empirically validate that the number of features used for the sparse projection has no effect on the results, or simply to solve the problem exactly. For $X \in \mathbb{R}^{n \times D}$ with $n = 900, D = 24\times 24\times 1024$ (the size of a CLIP L/14@336 representation), it should be tractable to compute $XX^\top$ and then either use that as a kernel matrix for kernel ridge regression or take its (matrix) square root and use that as a $900 \times 900$ feature matrix.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
**Major**:

- The statistical testing and idea behind the pipeline (feature extraction, dimension reduction, training) seems sound. For the dimensionality reduction, the authors used sparse random projects. This seems well-motivated theoretically in the appendix. However, for some tasks, it has been shown [1-3] that just selecting a random subset is also a viable strategy which can outperform the sparse and gaussian random projections. Would it be possible for the authors to try it or do they consider this to not be the performance bottleneck of the pipeline?
- The error analysis for more intuitive understanding of the problem seems to be missing. Would it be possible for authors to include the following images (and associated discussion) in the appendix?
  1. Cases where the prediction is aligned with the human judgement?
  2. Cases where the prediction differs from the human judgement?
- Baseline or any other meaningful comparison seems to be missing. Although it is not the goal of the paper to show the possibly high quality of the probes and rather compare them among the models of different modalities, having a baseline to compare would be very beneficial to a reader. E.g. 60% explainable variance explained may be both very low (if close to the baseline) or surprisingly high. How high would the baseline of just the image class go? I imagine that the class of _cat_ has on average much higher ranking than _house_.
- Explanation of the showed results seems to be missing. Do the probes work because they essentially detect classes (see previous point)? Why do the probes get better the deeper you go to the model?

**Minor**:
- The general approach appears to be supervised probing, which should be more explicitly acknowledged.
- The large figure captions are hard to parse but this may be justified given the page requirements.
- I enjoyed the writing which seems to be without any typos and seems clear and well motivated.
- The research question of multimodality seems interesting.

**References (for dimensionality reduction)**:
- [1] Padmaja, D. Lakshmi, and B. Vishnuvardhan. "Comparative study of feature subset selection methods for dimensionality reduction on scientific data." 2016 IEEE 6th International Conference on Advanced Computing (IACC). IEEE, 2016. [link](https://www.researchgate.net/profile/Lakshmi-Padmaja-Dhyaram/publication/306304009_Comparative_Study_of_Feature_Subset_Selection_Methods_for_Dimensionality_Reduction_on_Scientific_Data/links/61e5a2468d338833e37690af/Comparative-Study-of-Feature-Subset-Selection-Methods-for-Dimensionality-Reduction-on-Scientific-Data.pdf)
- [2] Huertas-García, Álvaro, et al. "Exploring Dimensionality Reduction Techniques in Multilingual Transformers." arXiv preprint arXiv:2204.08415 (2022). [link](https://arxiv.org/pdf/2204.08415.pdf) 
- [3] Zouhar, Vilém, et al. "Knowledge Base Index Compression via Dimensionality and Precision Reduction." Proceedings of the 1st Workshop on Semiparametric Methods in NLP: Decoupling Logic from Knowledge. 2022. [link](https://arxiv.org/pdf/2204.02906.pdf)
review_title: Interesting, sound, standard parts & sections possibly missing

Decision: Accept(Poster)