Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a method of fine-tuning a language model to analyze a set of opinions and produce a candidate consensus statement.  This is accomplished by various human evaluations, re-ranking, and fine-tuning algorithms. The consensus statements produced outperform various baselines, while retaining important information. 
strengths_and_weaknesses: Wow. I really, really liked this paper. I found it intellectually stimulating, technically complex but understandable, interdisciplinary, and potentially impactful in a way that I rarely see. 

+ The paper addresses an important, timely, societally relevant problem
+ The paper makes substantial methodological and technical contributions
+ The paper is well written
+ The results are powerful
+ I believe the paper will serve as a foundational piece of work for subsequent research
+ The paper draws on a wealth of ideas from computer science and social science
+ The paper has carefully considered limitations and ethical concerns
+ The paper has potential for strong, immediate impact if used properly

Weaknesses:

- While I appreciate the technical goal of using reranking to be able to change SWFs at test time, it's unclear if that's actually a useful feature. And while the paper claims that reranking is on par with or exceeds an RL-based approach, it would have been nice to include such a direct, RL based consensus generator as a baseline.
- While the model does outperform all baselines (usually), the difference is not as pronounced as I would like
- The paper does devolve into excessive technicalities at times

questions: None.
limitations: The paper does a good job of assessing both limitations and ethical issues.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use a large language model (LLM) to find consensus between user statements; the main difference with previous summarization or opinion aggregation work is that the summary is evaluated by the users (people who generated the statements) themselves; thus the goal is to find common aspects in the statements that most users agree with.
The paper conducts a large-scale data collection with thousands of questions (political issues); human participants write statements about the questions and also evaluate the quality and agreement of candidate statements. Method-wise, the paper adopts a reranking model to assign rewards to consensus and uses it to rerank 16 consensus statements generated by another model. The consensus model is trained to generate high-quality summaries. The rewards model is trained to prefer statements with a higher agreement for a particular user (given the user's opinion). Results show that the model can beat zero-shot and few-shot baselines, as well as the model without the reranking step, for both quality and agreement ratings. 
strengths_and_weaknesses: The paper presents an interesting idea of generating consensus, and collected high-quality data about political issues. The results clearly show gains over the baselines as well as human statements. The paper is well written and easy to follow.

The weakness is that the paper ignores many nuances about consensus generation and focuses solely on consensus generation. For example, one important question is the [detection] of consensus; is a consensus possible given the current statements or not? It is natural that for some questions and users, one cannot find much consensus for them to agree on. It is also interesting to verify the coverage of generated consensus (i.e., whether it covers all the points that the users agree on).
questions: 1. The paper uses a quite large language model (70B parameters) for finetuning. Is this necessary and can we use smaller models? Usually, small models can also handle summarization and quality evaluation well if there is enough data, and smaller models are easier to deploy.

2. There is one step I do not quite understand about model training: in the first round, how is the top-1 candidate chosen from the 16 candidates? In the first round, we do not have the reward model yet, so it seems we do not have the utilities $u_1, …, u_n$ for ranking.
limitations: The authors state the limitation of the current work well and I agree with it.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present an experiment for adapting a pre-trained language model (“Chinchilla”) to generate a consensus opinion given a set of individual opinions on a target question. Their method for generating consensus statements relies on 3 core components:
a) Use Chinchilla to generate candidate consensus statements. This model is (optionally) conditioned on the opinions of all/some individuals.
b) Use a reward model (RM; also based on Chinchilla) to predict the agreement score of each user (using the opinion of that user for conditioning) with each candidate statement.
c) Aggregate agreement scores using different social welfare functions (SWFs) and generate one final ranking over all candidate statements. Then they select the statement with the highest ranking. For training, they sample different functions from a parametric family of SWFs.

To train the models, the authors collect a novel dataset in two rounds of human-in-the-loop training iterations, where they use study participants give their opinions on a set of curated political questions (targeted to a UK audience), generate consensus statements online, and have participants rate those consensus statement based on their agreement with them and quality of writing. This protocol creates a number of training instances (comprising questions, individual opinions, consensus statements, agreement/quality ratings) that they then use to fine-tune their models before the next round.

A big part of their work is centered on evaluating their proposed approach.. They first look at average agreement and quality scores obtained for their proposed method and the baselines (proposed method is better), as well as the “winning ratio” (percentage of times where proposed method was preferred over each of the baselines; generally, higher than 50%). Additionally, they investigate the gains of finetuning, as well as the additive impact of the RM, and conclude that both are useful (with finetuning yielding the largest gains). Furthermore, they look into the impact of user consensus (undivisive questions, where users unanimously agree, and divisive ones, where there’s various degrees of disagreement) but find no major effect. They test whether the consensus opinions generated by their model are preferred by opinions (not consensus ones - see weaknesses) given by human annotators, and find that their model is preferred in the majority of times. Their last evaluation concerns the impact of excluding some opinions when generating the consensus (as a sanity check that their model is indeed utilizing all opinions).

Finally, they discuss limitations and opportunities of their proposed approach.
strengths_and_weaknesses: # Strengths

The paper’s main strength lies in its motivation (fine-tuning a language model to generate consensus opinions is a novel and interesting task) and the sheer computational and financial cost expended for this experiment (dozens of GPUs; 43k pounds spent for compensating human participants) making it an interesting contribution to the field of NLP as it investigates a very interesting application. The authors also deserve credits for highlighting a lot of the ethical issues that come with this application in Section 5 (What is consensus anyway? What are potential biases introduced by their recruiting protocol? Etc.). This casts this contribution as a work of significance in its field in terms of novelty of applications, though not in terms of theoretical advancement (which is fine, as it was not the goal here).

In addition, there are several strengths in how the authors conducted their experiments:

1. They made efforts to pay human participants a living wage (approx. 15 pounds an hour), which is appreciated in an era where similar works choose a pay-per-click model. This may potentially increase the quality of their data.
2. They held out an out-of-domain validation set for testing the generalization of their models to new topics.
3. They utilize a well-validated concept from economics for parametrizing consensus (the SWFs, which allow for different interpretations of consensus depending on the value of parameter alpha). This makes their method suitable for different formulations of consensus that might be needed for different applications.
4. They include an ablation analysis to see how different components improve performance (few-shot vs zero-shot, fine-tuning, reward model) and additionally test the differences with mixed-effects models.
5. They test various aspects of their models (see summary)

# Weaknesses

There is one main weaknesses that stand out in this work: The experiments are not reproducible, as a) the authors do not release their data/models/code and b) the cost of the experiment is prohibitive for most groups that would want to replicate this study. This does not diminish the importance of the contribution (as it is good that someone did it) but makes it harder to verify the main findings of this work in the future (Are language models capable of performing this task? What can we do to improve consensus? How else can we formulate consensus? What bias did this model introduce? Etc.).

Other weak points are:

1. Most of their analyses are done by comparing means, except Sec. 4.3, where the use of medians pops up all of a sudden (l. 279). In general, the use of the median as basis for comparisons should be preferred throughout as the distributions are not expected to be Guassian (e.g. in the case of divisive questions where the model optimizes for the Rawlsian SWF there should be a sort of bimodal distributions, with some raters agreeing and some disagreeing with the consensus)
2.. The comparison in 4.2 seems unfair. Authors compare model-generated consensus statements (which are learned to be more balanced) with participant opinions which often are, by the authors’ own motivation, biased. Moreover, in the case of undivisive questions, where participants should all more-or-less agree with one another, we should not expect to see a big difference (certainly not ~80% preference for model candidates).

## Presentation issues

Some minor presentation issues that can be easily improved:

1. Results section (l. 217-229) contains a lot of new information on the data collection process that should be moved further up (potentially in Sec. 3.2). In particular, the number of raters for each question should be mentioned before discussing how many opinions were presented to the model (l. 168)

questions: 1. The authors state (l. 122-123 in main text; Appendix B.2.1) that they had each participant rate each consensus prediction twice to measure intra-rater reliability (and subsequently discard raters with a low reliability). What is the time interval between those two ratings, and how many other instances did they rate in-between? In short, is it reasonable to assume that the two ratings were somewhat independent (i.e. the raters had “forgotten” their first rating by the time they did the
second?)?
2. l. 171 “each of which is ranked top-1 by a different $\alpha$”: does this ranking include the RM model to estimate the agreement of each rater?
3. Did you obtain new opinions and ratings from new participants for 4.2? This means you had an initial set of participants which generated the initial 4/5 opinions, from which the model created a consensus, and then 4 more participants who gave a second set of opinions (for comparison) and preference ratings? I think the authors are referring to the original participants but it’s not 100% clear from their formulation.
4. Can the authors provide estimates of agreement for divisive vs undivisive questions? Decisive questions should have, in general, much higher agreement scores, as it is trivial to produce a consensus (i.e. a simple yes/no without any further justification should be enough to get the participants to agree, since they all agreed/disagreed with the positive statement anyway).
limitations: While the authors do a very good job at covering potential limitations/biases of their work, there is one potential risk of misuse that they do not cover: It is possible to (mis)use their model to wrongly present a summary as the consensus opinion of a particular demographics. For example, one could aggregate several social media posts of that demographic, and generate a statement “Demographic X thinks the government should do Y”. This would be something other than “Persuasiona” and “Factuality”. A potential title for this type of abuse would be “Misrepresentation”. This is particularly pertinent as the authors removed questions that could potentially illicit offensive answers (for good reason), which means they are unable to test how their model functions in those cases.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Motivated by the fact that people could express diverse opinions on a topic, this work explores whether a LLM would be capable of generating consensus statements from such differing opinions. As part of their extensive data collection initiative, the authors collect opinions from participants on moral and political issues specific to the UK. Consensus statements are first generated using a Chinchilla model, with opinions as prompts. The generated statements are rated by the participants for agreement and quality, and high-quality statements are used to fine-tune a pretrained Chinchilla model. A reward model is then trained to rank consensus statements in order of their perceived appeal to the overall group of participants. The work demonstrates that the model-produced consensus statements significantly outperform those obtained by their baseline models. 
strengths_and_weaknesses: Strengths:
- The paper is well-written and easy to comprehend.  The underlying problem is well-explained and the methodology has been clearly presented.
- It is great to observe that ethical considerations were taken into account during the data collection efforts (lines 128-133).
- The authors are aware of the potential biases that may arise and have detailed them in the Limitations section. For instance, lines 314-315 mention the likelihood of users with authoritative tones commanding more influence over the consensus. Further, the limitations w.r.t. the specifics of the data study cohort are well etched out in lines 325-333.
- Potential risks of this work have also been touched upon in lines 342-348, where the authors mention the potential for misusing such LMs for persuasion.

Weaknesses:
Although well-written, the work unfortunately does not showcase any significant advancements in ML/NLP. It amounts to fine-tuning (a very specific) LLM for a specific task to show improvements over baselines. Questions arise on the choice of the LM made and the modeling strategy, along with the evaluation criteria used. 
- Instead of using machine-generated consensus statements and further obtaining human annotations on them for finetuning the LM (lines 164-183), a more obvious alternative would have been to obtain ground-truth consensus statements from human participants and fine-tuning the LM on them. This set of participants could come from a different pool to the cohort who provided the differing opinions (i.e. the input). This would allow for the measurement of usual metrics for generation tasks, such as ROUGE, BLEU, METEOR etc. It is unclear why this was not selected as the modeling strategy for the paper.
- The motivation for selecting Chinchilla as the only LLM to experiment on needs some explanation. Have the authors explored other generative models such as BART, T5, GPT-2 etc for this task? Performance on these models should at least be reported as baselines.
- The evaluation criteria currently involves participants rating their preference of the consensus statements generated by the model in comparison with the baselines. This enforces the assumption that the statements generated by the model are satisfactory, both in terms of factuality and coherence. A stronger quality check on the generated text would be to compare these against human-generated consensus statements, and report ROUGE, BLEU/METEOR scores.
questions: - It would be helpful if the authors could motivate and provide a short overview about the Social Welfare Functions used, e.g. Rawlsian, Utilitarian (lines 141-152). These would potentially be new concepts to ML practitioners and it would be helpful to briefly understand the motivation for using these specific functions.
- Lines 168-170: The authors mention omitting a few opinions to ensure data diversity. Could they please elaborate further on why they adopted this strategy, since this would reduce the number of diverse opinions the model would be allowed to see? 
- The core of the study is based upon the fact that participants provide diverse opinions to the questions in the corpus. Could the authors please elaborate on how diversity was ensured? Is there a quantitative measure they could report for this?
limitations: The Limitations section has been well-written.
review_title: null

Meta-review: metareview: This paper tackles the interesting task of training a language model to generate a consensus statement that maximize the expected  approval of a group of people with diverse opinions. The proposed approach that is based on human evaluations of LM-generated statements and training of a reward model, is compared with a set of baselines. Based on reviewers' suggestions, authors performed additional experiments with smaller models and included analysis of results.

SACs, please note that reviewer bGmJ changed their rating from 3: Reject to 6: Weak Accept verbally, but this change isn't reflected in the average rating.
confidence: Certain
Decision: Accept