Venue: NeurIPS 2022 Workshop SVRHM

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
Strengths
- This work proposes a novel extension of Cultural Consensus Theory (CCT).
- The new CCT framework was applied to a dataset from Peterson et al. 2016, addressing an important limitation of this prior work, considering responses made by different groups of people rather than in aggregate.

Weaknesses/improvements
- An ethical implications section should be added to this work. A good model for this would be the ethical implications section of Peterson et al. 2016 as this work builds on their data and has similar potentials for miss-use. There is extra space on page 5 to fit this in.
- Further analysis of the size and spread of the clusters found could be an interesting addition to this work as it could aid in collecting more diverse datasets and understanding current dataset biases.

Misc
- line 132 extra word, "model that were"
- line 175 extra word, "for from"
review_title: Cultural alignment of machine-vision representations

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary

This work uses the Cultural Consensus Theory (CCT), a Psychometric model that estimates the consensus of a group of participants as a latent variable that shapes their judgements, to finetune DNN visual representations and show that it can improve performance in predicting human subjective judgements on face stimuli (e.g. smug, memorable, familiar, etc). They show that, for some of these judgements, being able to account for multiple cultures can boost predictions of human judgements over assuming only one consensus. 

## Strengths
* The work is well motivated, because being able to finetune these models to be able to account for multiple possible consensuses is an important direction to aligning human-machine visual representations. 

* The authors do a really cool extension using Bayesian nonparametrics to make CCT compatible with an unbounded number of cultures/consensuses. 

## Suggestions

* I think the results can be shown with a little more clarity. For instance, in Fig 1. right, RMSE is not defined. I assume it could mean root mean squared errors, but then all the black bars are considerably larger than the red bars, suggesting that the multiple truths models have higher error than the single truth models, which goes against what is said in the discussion?

* For at least Fig 1. left, I was surprised by how many traits for which the single-truth models still do better than multiple-truths. For example, for cute, red bar is still considerably higher than grey bar, though maybe I might be reading these wrong (see first suggestion). I'm wondering if the dataset's participant bias (primarily white and in the U.S) has something to do with there being a lower number of consensuses. I think collecting/using a dataset that draws from a much more diverse participant pool would allow this method to shine the most. 
review_title: An Interesting Way to Align Machine Vision Representations using a Psychometric Model

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses an interesting problem of how latent space features used for image generation using deep generative models can also be informative of the sentiment that humans attribute to those images.

The upsides to this this paper are that instead of simply regressing to human judgements, which has been done by a quite a few cited papers such as Peterson et al. (2022), this paper considers how different cultures may align by taking CCT into account and evaluating both a single-truth and multi-truth model for each attribute. This is particularly interesting since people of different background may give different ratings, and this taken their respective cultures into account. Also, the metrics by which the authors evaluate their ridge regression models (R-squared and RMSE) are quite appropriate to assess how well the model fits the data.

The downsides here are that the authors claim they are aligning machine representations to human judgments, but the input to the regression model is simply the conditional input to StyleGAN2. What is the significance of regressing samples drawn from a multivariate normal distribution to human judgments? Instead, it would have made more sense for an intermediate representation to be used in the regression, as those would be machine representations, and not just samples from a random probability distribution.

On aesthetics, I strongly felt that the paper could have been written in a more clear an informative way, and could use more figures.
review_title: Review of "Cultural alignment of machine-vision representations"

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper is clear and easy to follow. The work would be of interest to the audience of SVRHM. Studying cultural consensus via intermediate latent representations is quite an original and fascinating direction. 

A few details that I would have liked to have gotten in the paper are: 
1. What is the pretrained deep network that was fine-tuned for this work? 
2. What is the dimensionality of the Linear layer used for the fine-tuning? 
3. Why did authors use Ridge regression as opposed to e.g. train with plain SGD etc.? 

It would be nice to include this information in the main paper to facilitate reproducing results by other researchers. Would also like to hear what follow up directions this work opens for the field.
review_title: Interesting work, would love to see it it at SVRHM 2022. 

Decision: Accept(Poster)