Venue: NAACL 2025

# LLMs as Meta-Reviewers' Assistants: A Case Study

 Eftekhar Hossain\({}^{\clubsuit}\), Sanjeev Kumar Sinha\({}^{\spadesuit}\), Naman Bansal\({}^{\spadesuit}\), Alex Knipper\({}^{\clubsuit}\), Souvika Sarkar\({}^{\diamondsuit}\), John Salvador\({}^{\clubsuit}\), Yash Mahajan\({}^{\spadesuit}\), Sri Guttikonda\({}^{\spadesuit}\), Mousumi Akter\({}^{\spadesuit}\), Md. Mahadi Hassan\({}^{\clubsuit}\), Matthew Freestone\({}^{\spadesuit}\), Matthew C. Williams Jr.\({}^{\spadesuit}\), Dongji Feng\({}^{\spadesuit}\), Santu Karmaker\({}^{\clubsuit}\)

###### Abstract

One of the most important yet onerous tasks in the academic peer-reviewing process is composing meta-reviews, which involves assimilating diverse opinions from multiple expert peers, formulating one's self-judgment as a senior expert, and then summarizing all these perspectives into a concise holistic overview to make an overall recommendation. This process is time-consuming and can be compromised by human factors like fatigue, inconsistency, missing tiny details, etc. Given the latest major developments in Large Language Models (LLMs), it is very compelling to rigorously study whether LLMs can help meta-reviewers perform this important task better. In this paper, we perform a case study with three popular LLMs, i.e., GPT-3.5, LLaMA2, and PaLM2, to assist meta-reviewers in better comprehending multiple experts' perspectives by generating a controlled multi-perspective-summary (MPS) of their opinions. To achieve this, we prompt three LLMs with different types/levels of prompts based on the recently proposed TELeR taxonomy. Finally, we perform a detailed qualitative study of the MPSs generated by the LLMs and report our findings1.

Footnote 1: Our dataset is available at [https://github.com/BridgeAI-Lab/LLM-as-Meta-Reviewer](https://github.com/BridgeAI-Lab/LLM-as-Meta-Reviewer)

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7763–7803

April 29 - May 4, 2025 \(\copyright\)2025 Association for Computational Linguistics

## 1 Introduction

Meta-reviewing is a critical step in the overall scientific peer-review process, which focuses on understanding the consensus of expert opinions on a scholarly work and making informed judgments on its scientific merit Fernandes and Vaz-de Melo (2022). Meta-reviews are also helpful as they concisely summarize the strengths and weaknesses of the manuscript and highlight the scope for further improvement. Composing a meta-review from peer reviewers' comments includes the following steps:

1. Collect expert opinions on the paper from peer researchers in the same/similar domain.
2. Analyze the comments and opinions from different reviewers and comprehend the common strengths, weaknesses, and suggestions for improvement emphasized by the peer reviewers and summarize them into a holistic overview.
3. Present meta-reviewer's own judgment about the paper (resolve or arbitrate conflicts among reviewers) and make a final recommendation.

The process of meta-reviewing is arduous and time-consuming Sun et al. (2024). In addition, over the years, the number of research manuscript submissions has been rising exponentially Fire Guestrin (2019). As such, the volume of meta-reviewing tasks is also rising exponentially, making the job of meta-reviewers even more challenging and time-consuming. Since human beings are prone to fatigue, bias, and distraction Goldberg et al. (2023), it is possible that this large workload may make meta-reviewers susceptible to inconsistency and missing details. To address this issue, we performed a case study using LLMs to generate multi-perspective summaries (MPSs) Kammaker Santu et al. (2018); Bansal et al. (2022a) of reviewers' comments to see whether LLM-generated summaries can assist meta-reviewers in being more consistent and efficient in this important role. Although several works have proposed automated meta-review generation techniques Zeng et al. (2024); Bhatia et al. (2020); Pradhan et al. (2021); Shen et al. (2022), our work is different in three aspects: 1) Instead of meta-review generation, we focus on the MPS task with a goal to assist the human meta-reviewer, 2) We study LLMs in combination with a taxonomy-based prompting technique (e.g., TELeR Santu and Feng (2023)), which is currently understudied, 3) We examine the correlations between human and automatic (GPT-4) evaluations to explore the reliability of LLM-based assessments.

**Case Study:** For our case study, we selected 40 research papers submitted to the ICLR Conference in

recent years, along with peer-reviewer comments and a handcrafted meta-review written by a highly experienced researcher for each article. These manuscripts and the associated peer-review comments and meta-reviews are all publicly available on OpenReview.net. Using this data set in combination with the three LLMs (GPT-3.5, PaLM2, LLaMA2) and the TELeR taxonomy, we performed an extensive qualitative analysis by engaging human evaluators to go through the meticulous process of reading and analyzing \(40\) individual paper abstracts, \(120~{}(40\times 3)\) associated peer-review narratives, \(40\) corresponding human-written meta-reviews, and a total of \(480\) distinct LLM-generated MPSs2 and then collected comprehensive survey data, which includes \(4800\) granular manuscript level judgments on five specific criteria and \(90\) LLM level judgments3 from humans. Although there are several popular text generation evaluation metrics such as ROUGE, SEM-F1, BERTScore, etc. [7], we exclusively focus on meticulous qualitative human evaluation because previous studies have reported several limitations of automated evaluation metrics for the text generation task [6, 1]. In addition to human judgments, we also conducted a GPT-4-based evaluation of MPSs and examined the correlation between human and automatic evaluations to explore the reliability of LLM-based assessments of MPSs along five specific criteria.

Footnote 2: Total comes from 3 LLMs \(\times\) 4 prompt levels \(\times\) 40 papers \(\times\) 5 evaluation criteria \(\times\) 2 types (precision/recall)

**Findings:** Our qualitative analysis reveals that regarding the quality of the generated MPSs, GPT-3.5 and PaLM2 performed comparably and were rated higher by humans than LLaMA2 in terms of manuscript-level judgments. Interestingly, PaLM2 generally yielded better recall scores, while GPT-3.5 yielded better precision scores. More surprisingly, GPT-3.5 was rated poorly by humans at LLM-level judgments, which demands further investigation. On the other hand, the automatic evaluation shows that the GPT-4 evaluation correlates poorly with human judgments when tasked with evaluating the meta-reviews generated by its predecessors (i.e., GPT-3.5). Our findings also highlight that LLM-based evaluations are unreliable in tasks that require aspect-aware summary evaluation, especially when dealing with narratives from multiple diverse perspectives.

## 2 Related Work

Over the past few years, several studies have been accomplished on meta-review generation. Some researchers have employed rule-based techniques in conjunction with deep neural network architectures [1, 2], while others proposed encoder-decoder-based methods [13, 2] to tackle the meta-review generation task. Recent developments with Large Language Models (LLMs), including but not limited to GPT-3.5 and GPT-4 [15, 16], LaMA [17], LLaMA [18], Bloom [14], GLaM [19], have shown remarkable performance in traditional summarization tasks. Yet, their performance on complex, i.e., multi-constraint multi-document summarization tasks [1] like meta-review composition is still understudied.

Recently, Du et al. du2020learning examined the effectiveness of large language models (LLMs) in both reviewing and meta-reviewing research papers. Similarly, Tyser et al. tysser2020learning evaluated LLMs' ability to review papers and compared their quality with human-written peer reviews. In another line of work, Zeng et al. zeng2020learning introduced a Checklist-guided Iterative Introspection (CGI2) approach using LLMs, which breaks down scientific opinion summarization into several stages. However, LLMs are quite sensitive to prompts that are fed to them [16, 15], and therefore, designing appropriate prompts with the right amount of detail has become more important than ever [10, 11]. This issue worsens in practice as there is no standard taxonomy of prompts to follow for benchmarking LLMs on complex tasks that the community has a general consensus on yet. As such, the main motivation of this work, which makes our case study more compelling, stems from the following two gaps in the literature: 1) LLM's performance on multi-constraint multi-perspective summarization tasks (e.g., meta-review composition) is still understudied, 2) no study has yet followed a standard prompting taxonomy to systematically compare LLMs on such complex tasks.

Fortunately, very recently, Santu and Feng santu2020learning have proposed a general prompting taxonomy called TELeR, which can serve as a unified standard for comparing and benchmarking LLMs' performances on complex generation tasks (de 

tails in appendix). In this work, we leveraged this TELeR taxonomy for designing prompts in the context of a meta-review composition task. We reported the specific categories of prompts we experimented with across multiple LLMs. This enabled more meaningful comparisons among the three LLMs we studied and, thereby, helped to derive more accurate conclusions about their relative performances.

## 3 Case Study

This section describes the details of our case study, including the data set, LLMs, evaluation criteria, and prompting details.

### Data-set Details

For evaluation, we carefully selected 40 research papers and their associated peer-review comments and meta-reviews from the collection of more than \(13,800\) research manuscripts submitted to the ICLR conference during a span of four years, i.e., the year 2020 through 2023. While selecting papers for our evaluation, we followed some general principles, as mentioned below.

1. Meta-review is detailed and substantially encompasses the core contributions, strengths, weaknesses, and scopes for improvements.
2. Manuscript received at least 3 detailed reviews that comment on different aspects, e.g., core contributions, strengths, weaknesses, suggestions for improvement, and literature review quality.
3. Preference was given to include more recent papers, especially those submitted during 2021-2023 dealing with diverse topics.
4. The corpus also includes 10 rejected papers (25%) to increase meta-review diversity.

### Large Language Models

As mentioned previously, our case study evaluates 3 recent popular LLMs - GPT-3.5 Brown et al. (2020), PaLM2 Chowdhery et al. (2023), LLaMA2 Touvron et al. (2023b). Prompting of GPT-3.5 and LLaMA2 (13B) was performed through their API, but interaction with Google's PaLM2 was done manually through a web browser because PaLM2 does not provide an API. As Zero-shot learning techniques have become very popular in recent years Sarkar et al. (2023, 2022), we use all LLMs in the Zero-shot setting for prompting without further fine-tuning.

### Prompt Design using TELeR Taxonomy

We prompted LLMs with different levels of prompts designed based on the recently proposed TELeR taxonomy (described in section A.1). To be more specific, let us look at what the TELeR levels meant for our MPS task. In all the following examples, we assume that a prompt is a concatenation of directive text (instruction) followed by the peer review texts.

* Level 1: This high-level directive asks the LLMs to simply generate a meta-review without providing any further specific requirements.
* Level 2: At level 2, directives include a paragraph with multiple sentences describing instructions/questions articulating multiple sub-tasks.
* Level 3: Directives are a bulleted list in this level, where each bullet describes a particular sub-task.
* Level 4: This level is similar to level 3. The only difference is that the prompts at this level also ask LLMs to explain their own outputs.

As part of the meta-review generation task, LLMs were asked to perform the following subtasks for prompts at level 2 and above.

1. What is the summary of core contributions? Provide an answer with supporting evidence.
2. Which common strengths are referred to in the reviews? A common strength is a strength that is mentioned in at least two reviews.
3. What common weaknesses are described in the reviews? A common weakness is a weakness that is mentioned in at least two reviews.
4. What suggestions for improvement have been provided by three reviews? A common suggestion for improvement is a suggestion that is mentioned in at least two reviews.
5. Do the reviews mention about missing references? A list of missing references is optional.

For further details, the exact prompts used in our case study are shown in the appendix (see Table 4).

### Evaluation Criteria and Process

For qualitative evaluation, \(10\) human annotators, all researchers in NLP and co-authors of this paper, were involved as volunteers. Each annotator was assigned to assess LLM-generated MPSs for \(4\) research papers. For each paper, an annotator was provided with MPSs generated by \(3\) LLMs using \(4\) prompts at \(4\) different TELeR levels. Each annotator evaluated the LLM-generated MPS in light of publicly available reviews posted by the peer 

reviewers corresponding to a particular manuscript. It was not disclosed to them which LLM they were evaluating to preclude any psychological bias in the evaluation. All of them were briefed in detail about the evaluation task. Two separate evaluation forms were designed to collect their judgments:

1. **Micro-Evaluation:** The first set of human evaluations captured their judgment on the quality of the MPSs generated by each LLM for each prompt. For a given prompt and a corresponding output from a particular LLM \(x\), each human evaluator was asked to judge the generated MPS based on five different criteria as described below. For each criterion, the human evaluators were presented with an assertive statement about that criterion and then asked to provide their judgments by choosing their level of agreement with the given statement in terms of five different labels: 1) Strongly agree, 2) Agree, 3) Neutral, 4) Disagree, and 5) Strongly disagree. The five criteria and their associated assertive statements (against which humans rated their level of agreement) are presented below. 1. Core Contribution: _LLMx generated MPS captured core contributions very well._ 2. Common Strengths: _LLMx generated MPS captured common strengths very well._ 3. Common Weaknesses: _LLMx created MPS captured common weaknesses very well._ 4. Common Tips for Improvement: _MPS generated by LLM x captured common suggestions for improvement very well._ 5. Literature Review: _MPS generated by LLM x included the missing references mentioned in the reviews very well._
2. **Macro-Evaluation:** The second set of evaluations was designed to capture human judgments regarding the overall performance of a particular LLM on the MPS task. We call it the "Macro-Evaluation". This form requires two types of responses - the first type has three assertive statements concerning LLM's performance in terms of adherence to instructions, ability to create useful MPSs, and matching against actual expert-written meta-reviews. Again, evaluators were asked to rate their level of agreement on a five-point scale - 1) Strongly agree, 2) Agree, 3) Neutral, 4) Disagree, and 5) Strongly disagree, with respect to the following 3 Macro-Evaluation statements. 1. _"While generating MPSs, LLMx strictly followed prompt instructions"._ 2. _"LLMx generated MPS is useful for preparing a meta-review"._ 3. _"MPS generated by LLMx matches with the actual meta-review to a great extent"._ The second type of feedback is an essay question - "Kindly provide comments about LLMx's performance in generating MPSs." where \(x\) represents a particular LLM.

## 4 Results

### Results from Micro-Evaluation

For the micro-evaluation experiments, we focused on answering the following Research Questions (RQs) related to different components of the LLM-generated MPS:

"_While generating a multi-perspective summary (MPS) based on the consensus among multiple reviewers' comments on a scholarly work, can LLMs properly capture-_"

1. [leftmargin=*]
2. _RQ__Mic-1:_ _the core contributions of the work?_
3. _RQ__Mic-2:_ _the major strengths of the work?_
4. _RQ__Mic-3:_ _the major weaknesses of the work?_
5. _RQ__Mic-4:_ _the avenues for improving the work?_
6. _RQ__Mic-5:_ _how well the manuscript conducted review of relevant literature?_ Below, we answer each Micro-Evaluation research question based on \(4800~{}(40\times 3\times 4\times 2\times 5)\) fine-grained human ratings related to \(480~{}(40\times 3\times 4)\) different LLM-generated meta-reviews.
7. _RQ__Mic-1 (Core Contributions):_ We answer this question by analyzing Figure 1 and Figure 8 (in appendix), which depicts the comparative performances of three LLMs in terms of their (human-perceived) precision/recall distributions, where human-perceived precision and recall distributions are derived from their qualitative rating counts, i.e., {SA=Strongly Agree, A=Agree, N=Neutral, D=Disagree, SD=Strongly Disagree}, followed by a normalization term to convert them into a valid distribution (individual values ranging between [0-1]). To be more specific, while computing the precision/recall distributions corresponding to RQ-Mic-1, human annotators independently rated their agreement with the following two statements. _RQ__Mic-1-P: Core Contributions (Precision)_: _"While generating meta-review, LLM x was pre 

cise in capturing the core contributions as highlighted by at least two reviewers."_
* 4 (more details) for both precision and recall. This suggests that GPT-3.5 could understand complex prompts better than PaLM2 and LLaMA2. Next, we looked at precision and recall distributions for each LLM by aggregating over all 4 Prompt Levels (Figure 0(a)), which revealed that GPT-3.5 and PaLM2 were comparable, while LLaMA2 was often rated as inferior (more Disagree and Strong Disagree ratings). Finally, Figure 0(b) shows the precision/recall distributions for each prompt level while aggregating human ratings over all three LLMs, indicating that Level 3 and Level 4 prompts were more effective for the meta-review generation task than Level 1 and Level 2, which is intuitive as meta-review generation is a complex task. Finally, Figure 2 shows the precision/recall distributions aggregated over three LLMs and four prompt levels. This result suggests that humans generally voted in favor of LLMs more often for the meta-review generation task than otherwise.
* _RQ Mic-2 (Common Strengths):_ In the case of Common strengths, Figure 2(a) reveals that GPT-3.5 and PaLM2 were comparable, while LLaMA2 was often rated as slightly inferior. On the other hand, Figure 2(b) demonstrates that the performance of LLMs improves substantially from level 1 to level 2 prompts. There is further improvement in the precision of LLMs as we go to higher levels of prompts. But this is not true in the case of recall, where the performance keeps improving till level 3 and then goes down.
* _RQ Mic-3 (Common Weaknesses):_ We notice a similar comparative performance of LLMs (Figure 2(c)) and Prompt Levels (Figure 2(d)) in the case of Common weaknesses as we observed for Common Strengths.
* _RQ Mic-4 (Common Suggestions):_ We again notice a similar comparative performance of LLMs (Figure 2(e)) and Prompt Levels (Figure 2(f)) in the case of Common suggestions as we observed for Common Strengths and Weaknesses.
* _RQ Mic-5 (Literature Review):_ We notice GPT-3.5 and PaLM2 were rated more favorably than LLaMA2 (Figure 2(g)) by the human annotators for the Literature Review criterion. Further, as Figure 2(h) demonstrates, the performance of LLMs improves substantially from level 1 to level 2 prompts. However, we did not find appreciable improvement in performance by upgrading from level 2 to levels 3 and/or 4.

**Summary of Micro-Evaluation**: Figure 4 shows the precision/recall distributions aggregated over three LLMs and four prompt levels for the follow

Fig. 1: Core Contributions Ratings - rated separately across different Prompt Levels and different LLMs. Here, SA: Strongly Agree, A: Agree, N: Neutral, D: Disagree, SD: Strongly Disagree, P: Precision, and R: Recall.

 

ing aspects of LLM-generated meta-reviews: Common Strengths, Common Weaknesses, Common Suggestions and Literature Review. This result (in combination with Figure 2) suggests that humans generally voted in favor of LLMs more often for the Core Contributions, Common Strengths, and Literature Review aspects. Common weaknesses and common suggestions were evaluated with mixed ratings, suggesting LLMs struggled more along these aspects. Finally, an overall precision and recall score was computed by assigning the following scores to ratings, i.e., Strongly Agree = 4, Agree = 3, Neutral = 2, Disagree = 1, and Strongly Disagree = 0, and, thereafter, adding the scores of 4 prompt styles and dividing the sum by the maximum possible score to get a normalized score in the range [0,1]. Table 1 summarizes these scores, where we observe the following:

Fig. 4: Overall Rating aggregated over three LLMs and four Prompt Levels.

Fig. 3: Ratings of Four Criteria (Common Strengths, Common Weaknesses, Common Suggestions, and Literature Review Quality) - Aggregated separately across different Prompt Levels and different LLMs.

 



### Results from Macro-Evaluation

For the macro-evaluation experiments, we focused on the following broad research questions related to the overall performance of LLMs.

1. _RQ Mac-1: Can LLMs properly understand the complex requirements and sub-tasks of MPS?_
2. _RQ Mac-2: How useful were the meta-review generated by the LLMs?_
3. _RQ Mac-3: Do LLM-generated meta-reviews match human-expert-written meta-review?_

Human evaluators again rated their agreements with the Macro evaluation statements on the same 5-point scale, i.e., {SA=Strongly Agree, A=Agree, N=Neutral, D=Disagree, SD=Strongly Disagree}. Next, a unified quality score was computed by assigning the following scores to ratings, i.e., SA = 4, A = 3, N = 2, D = 1, and SD = 0, and, thereafter, adding the scores of 4 prompt styles and dividing the sum by the maximum possible score to get a normalized score in the range [0,1] (Table 2). Surprisingly, Table 2 shows that GPT-3.5 is the worst performer, whereas PaLM2 is the best in all three aspects at the Macro Level. We were baffled by this outcome and decided to interview the annotators further. Based on the follow-up discussions, annotators generally suggested that micro-evaluation should be more reliable as micro-evaluation judgments are much more fine-grained and focused than macro ones. However, it remains an open question as to why GPT-3.5 was generally rated high in micro-evaluation but not in macro-evaluation.

### Automatic Evaluation with GPT-4o

In addition to qualitative human evaluation, we also automatically evaluated the LLM-generated MPSs. While prior work [76] employed GPT-based evaluators [98] to judge meta-reviews overall quality, we employ them for more fine-grained evaluation. To be more specific, we designed a structured, detailed instruction (Appendix SSA.3) and prompted the GPT-4o model to asses the five aspects of MPSs using the Likert scale for each LLM.

**Correlation in Micro-Evaluation:** Figure 5 illustrates the Pearson correlation between human scores and GPT-4o scores for micro-evaluation. We report the mean of the precision and recall correlation scores for each of the five criteria. From Figure 5d, we can see that the PaLM2 evaluation exhibits a better correlation between humans and GPT-4o across several aspects (i.e., core contributions, common strengths, common weaknesses), while GPT-3.5 demonstrates the weakest correlation when aggregating all the prompt levels. Figure 5b further reveals that at higher prompt levels (P3 and P4) for GPT-3.5 outputs, GPT-4o evaluations show near-zero correlation with human evaluations. This lack of correlation might be attributed to the phenomenon that GPT-4o almost always rated its predecessors' (i.e., GPT-3.5) generated summaries more favorably than humans. To validate whether this is indeed true, we plot the distribution of GPT-4o and Human evaluation scores on GPT-3.5 generated MPS for prompt levels 3 and 4 (shown in Figure 6). As shown in Figure 5a and Figure 5b, GPT-4 unduly assigned a higher score for more than 50% samples (more than 20 out of 40 papers) across most criteria. There were even instances where GPT scores exceeded human scores

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline
* Case Study Micro Evaluation.

\begin{table}
\begin{tabular}{c|c|c|c} \hline
**Aspect** & **PaLM2** & **GPT-3.5** & **LLaMA2** \\ \hline Adherence to instructions & **0.65** & 0.53 & 0.55 \\ \hline Meta-review usefulness & **0.65** & 0.53 & 0.63 \\ \hline Meta-review matching & **0.58** & 0.50 & 0.50 \\ \hline \end{tabular}
\end{table}
Table 2: Macro Evaluation Results

by \(3\) (G=\(3\)+H) and \(4\) (G=\(4\)+H) points, which led to a poor correlation with human evaluations.

**Correlation in Macro-Evaluation:** Table 3 shows Pearson correlation scores for macro-evaluation. Interestingly, the evaluation by GPT-4o shows a better correlation with human evaluations for the LLaMA2-generated MPSs, particularly in adherence and usefulness criteria. In contrast, no significant correlation is observed for the GPT-3.5-generated MPSs. Specifically for the adherence aspect, the _NaN_ value is attributed to GPT-4 assigning the same Likert score across all the MPSs. Similarly, looking at the other aspects implies that GPT

Fig. 6: Count Distribution of Human (H) vs. GPT-4 (G) evaluation scores on GPT 3.5 generated MPS for higher prompt level (level 3 and 4.). Here, ‘_HG=’_ indicates the human and GPT-4 give the same scores, ’_G>H’_ indicates GPT provides a higher score than humans, and ‘_G=x+H’_ indicates GPT provides x points more score than human.

\begin{table}
\begin{tabular}{c|c|c|c} \hline
**Aspect** & **PaLM2** & **GPT-3.5** & **LLaMA2** \\ \hline \hline Adherence to instructions & 0.102 & NaN & **0.534** \\ \hline Meta-review usefulness & 0.356 & -0.212 & **0.442** \\ \hline Meta-review matching & -0.128 & 0.00 & 0.263 \\ \hline \end{tabular}
\end{table} TABLE III: Correlation for Macro Evaluation 

4 evaluation over its predecessors did not accurately align with human judgments. In summary, automatic evaluation results reveal that GPT-4o is not a reliable judge when assessing multi-perspective summaries generated from different expert opinions on a scholarly work.

## 5 Conclusion

In this paper, we conducted a case study with three LLMs and four different prompt levels by following the recently proposed TELeR taxonomy to study whether LLMs can be useful in mitigating these issues. To achieve this goal, first, we collected 40 different ICLR research manuscripts, and then, for each manuscript, we used three LLMs separately to compose holistic multi-perspective summaries (MPSs) from available multiple expert reviewers' opinions. Next, we performed rigorous human evaluations to assess the MPSs' reliability. More specifically, we engaged \(10\) human evaluators to judge the LLM-generated MPSs on five important criteria, including the quality of capturing Core contributions, Common strengths, Common weaknesses, Common suggestions, and Literature Review. Further, we assess the reliability of GPT-4o as an automatic evaluator for this complex task.

Our case study showed that the reliability of LLMs as Meta-Reviewers' assistants widely varies across different criteria (e.g., core contributions, common strengths, etc.), along with the LLM and prompt type used. Our study also revealed that while summarizing multiple reviewers' comments, GPT-3.5 and PaLM2s were rated higher by humans than LLaMA2 in all five aspects. The performances of PaLM2 and GPT-3.5 were comparable on average, but PaLM2 yielded better Recall ratings in general, while GPT-3.5 yielded better Precision ratings. Although our micro-evaluation indicated PaLM2 and GPT-3.5 to be comparable, macro-evaluation results indicate that humans like PaLM2 generated MPSs more than the GPT-3.5 generated ones. Meanwhile, the automatic evaluation with GPT-4 shows a poor correlation with human evaluations, suggesting that GPT-4 is unreliable for evaluating this facet-aware complex MPS tasks.

## 6 Limitations

This work only analyses three LLMs and 40 research manuscripts due to time and resource constraints. More work will need to be done to increase the number of models tested and draw a more general conclusion on the performance of LLMs on the meta-review composition task. However, it should be noted that the estimated time required to annotate each paper is about 4 hours. Each annotator was assigned four papers for annotation. With ten annotators, a total of 160 hours of annotation effort was spent apart from initial exploration, preliminary studies, and data analysis.

Due to a lack of space, we could not discuss our study on the sensitivity of LLMs regarding prompt variations in detail in the main paper. However, all details have been provided in the appendix for the interested reviewer. To summarize, we found that in terms of the TELeR level of prompts, the performance of LLMs improves substantially from level 1 to level 2 prompts. However, we did not always find appreciable improvement in performance by going from level 2 to level 3 and level 4 prompts.

During annotation, human volunteers did not read the actual research papers; they only read the abstract and the associated reviews, comments, and meta-reviews for that paper. Therefore, the annotators assumed the reviewer's comments were correct and justified. Also, although the human meta-reviewers have access to the original paper in practice, our experiments with LLMs did not have that access.

The NeurIPS experiment of 2022 found various biases and inconsistencies when humans evaluate reviews4Goldberg et al. (2023). Although our annotators are all NLP researchers, it is still possible that those biases and inconsistencies also applies when evaluating meta-reviews.

Footnote 4: [https://blog.ml.cmu.edu/2023/12/01/peer-reviews-of-peer-reviews-a-randomized-controlled-trial-and-other-experiments/](https://blog.ml.cmu.edu/2023/12/01/peer-reviews-of-peer-reviews-a-randomized-controlled-trial-and-other-experiments/) 

Ethics Statement

Regarding ethical concerns for the potential implications of this line of research, we would like to highlight the following points.

The objective of the paper is NOT to have LLMs write the meta-review by itself; rather, the goal is to help meta-reviewers prepare better quality meta-reviews by providing them with a holistic summary of the expert peer reviewers' consensus on the intellectual merits of the paper. The human meta-reviewer should never blindly trust LLMs; rather, they should verify the summary claims first and then write the meta-review manually only after verification of the comments against the paper contents.

The paper clarifies in multiple places that the role of the meta-reviewer is not just limited to the summarization of the individual reviews, but in cases, it also involves resolving conflicting viewpoints, which may also include adding their own expert opinion. Another task is to identify low-quality reviews and down-weight them or invite new reviewers. This is why LLM-generated multi-perspective summaries should never be treated as a substitute for meta-review; rather, they should be viewed as additional assistance in better comprehending expert opinions.

## 8 Acknowledgments

This work has been partially supported by the National Science Foundation (NSF) Standard Grant Award #2452028 and Air Force Office of Scientific Research Grant/Cooperative Agreement Award #FA9550-23-1-0426. We would also like to thank the University of Central Florida CS Department and AI initiative for their continuous support through Student Fellowships and Graduate Assistantships.

## References

* M. Akter, N. Bansal, and S. K. Karmaker (2022)Revisiting automatic evaluation of extractive summarization task: can we do better than ROUGE? In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, pp. 1547-1560. External Links: Link Cited by: SS1.
* N. Bansal, M. Akert, and S. Karrmaker Santu (2022a)Semantic overlap summarization among multiple alternative narratives: an exploratory study. In Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic, pp. 6195-6207. External Links: Link Cited by: SS1.
* N. Bansal, M. Akter, and S. Karrmaker Santu (2022b)SEM-F1: an automatic way for semantic evaluation of multi-narrative overlap summaries at scale. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 780-792. External Links: Link Cited by: SS1.
* C. Bhatia, T. Pradhan, and S. Pal (2020)Metagen: An academic meta-review generation system. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1653-1656. External Links: ISBN 978-1-4503-387X, Document, ISSN 1053-6905 Cited by: SS1.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, Benjamin Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. PreprintarXiv:2005.14165. External Links: Link Cited by: SS1.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. (2023)Palm: scaling language modeling with pathways. Journal of Machine Learning Research24 (240), pp. 1-113. External Links: Link Cited by: SS1.
* J. Du, Y. Wang, W. Zhao, Z. Deng, S. Liu, R. Lou, H. Peng Zou, P. Narayanan Venkit, N. Zhang, M. Srinath, et al. (2024)LIMS assist nlp researchers: critique paper (meta-) reviewing. arXiv preprint arXiv:2406.16253. External Links: Link Cited by: SS1.
* N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. Wei Yu, O. Firat, B. Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, Y. E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier-Hellstern, T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and C. Cui (2022)GLAPL: efficient scaling of language models with mixture-of-experts. PreprintarXiv:2112.06905. External Links: Link Cited by: SS1.
* G. Leraus, F. Fernandes, and P. O. Vaz-de Melo (2022)Between acceptance and rejection: challenges for an automatic peer review process. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries, Vol. 1-12.

 

Michael Fire and Carlos Guestrin. 2019. Over-optimization of academic publishing metrics: Observing goodhart's law in action. _GigaScience_, 8.
* Goldberg et al. (2023) Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave, and Nihar B Shah. 2023. Peer reviews of peer reviews: A randomized controlled trial and other experiments. _arXiv preprint arXiv:2311.09497_.
* Han et al. (2022) Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. Ptr: Prompt tuning with rules for text classification. _AI Open_, 3:182-192.
* Santu et al. (2018) Shubhra Kanti Karmaker Santtu, Chase Geigle, Duncan Ferguson, William Cope, Mary Kalantzis, Duane Searsmith, and Chengxiang Zhai. 2018. Sofsat: Towards a setlike operator based framework for semantic analysis of text. _SIGKDD Explor. Newsl._, 20(2):21-30.
* Kumar et al. (2021) Asheesh Kumar, Tirthankar Ghosal, and Asif Ekbal. 2021. A deep neural architecture for decision-aware meta-review generation. In _2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_, pages 222-225. IEEE.
* Li et al. (2023) Miao Li, Eduard Hovy, and Jey Lau. 2023. Summarizing multiple documents with conversational structure for metric-review generation. _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 7089-7112.
* Liu et al. (2023a) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023a. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. _ACM Computing Surveys_, 55(9):1-35.
* Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: Nlg evaluation using gpt-4 with better human alignment. _arXiv preprint arXiv:2303.16634_.
* Loya et al. (2023) Manikanta Loya, Divya Anand Sinha, and Richard Futrell. 2023. Exploring the sensitivity of llms' decision-making capabilities: Insights from prompt variation and hyperparameters. _arXiv preprint arXiv:2312.17476_.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Pradhan et al. (2021) Tribikram Pradhan, Chaitanya Bhatia, Prashant Kumar, and Sukomal Pal. 2021. A deep neural architecture based meta-review generation and final decision prediction of a scholarly article. _Neurocomputing_, 428:218-238.
* Bai et al. (2022) Ananya Bai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation metrics used for nlg systems. _ACM Computing Surveys (CSUR)_, 55(2):1-39.
* Samutu et al. (2023) Shubhra Kanti Karmaker Santu and Dongji Feng. 2023. Teler: A general taxonomy of llm prompts for benchmarking complex tasks. _Preprint_, arXiv:2305.11430.
* Volume 2: Short Papers, Online only, November 20-23, 2022_, pages 135-147. Association for Computational Linguistics.
* Sarkar et al. (2023) Souvika Sarkar, Dongji Feng, and Shubhra Kanti Karmaker Santu. 2023. Zero-shot multi-label topic inference with sentence encoders and llms. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMLNP 2023, Singapore, December 6-10, 2023_, pages 16218-16233. Association for Computational Linguistics.
* Le Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Elie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galli, et al. 2022. Bloom: A 17Gbparameter open-access multilingual language model. _arXiv preprint arXiv:2211.05100_.
* Selcar et al. (2023) Melanie Selcar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. _arXiv preprint arXiv:2310.11324_.
* Shen et al. (2022) Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022. Mred: A meta-review dataset for structure-controllable text generation. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 2521-2535.
* Sun et al. (2024) Lu Sun, Stone Tao, Junjie Hu, and Steven P. Dow. 2024. Metawriter: Exploring the potential and perils of ai writing support in scientific peer review. _Proc. ACM Hum.-Comput. Interact._, 8(CCW1).
* Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegal, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhone Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zeewenberg, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erri Hoffman-John, Josh Lee, Lora Arroyo, Ravi Rajakumar, Alena Butyrna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Mariau Croak, Ed Chi, and Quoc 

Le. 2022. Lamda: Language models for dialog applications. _Preprint_, arXiv:2201.08239.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. _Preprint_, arXiv:2302.13971.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Tyser et al. (2024) Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, et al. 2024. Ai-driven review systems: evaluating llms in scalable and bias-aware academic reviews. _arXiv preprint arXiv:2408.10365_.
* Zeng et al. (2024) Qi Zeng, Mankeerat Sidhu, Ansel Blume, Hou Pong Chan, Lu Wang, and Heng Ji. 2024. Scientific opinion summarization: Paper meta-review generation dataset, methods, and evaluation. _Preprint_, arXiv:2305.14647.
* Zeng et al. (2023) Qi Zeng, Mankeerat Sidhu, Hou Pong Chan, Lu Wang, and Heng Ji. 2023. Scientific opinion summarization: Meta-review generation with checklist-guided iterative introspection. _arXiv preprint arXiv:2305.14647_.

## Appendix A Appendix

### Background on TELeR Taxonomy

As shown in Figure 7, the taxonomy introduced by Santu and Feng (2023) categorizes complex task prompts based on the following four criteria.

1. **Turn**: This refers to the number of turns or shots used while prompting an LLM to accomplish a complex task. In general, prompts can be classified as either single or multi-turn.
2. **Expression**: This refers to the style of expression for interacting with the LLM, such as questioning or instructing.
3. **Level of Details**: This dimension of prompt style deals with the granularity or depth of question or instruction. Prompts with higher levels of detail provide more granular instructions. For example, Level 0 has no directive, i.e., questions or instructions. On the other hand, Level 4 prompts have to specify the particular subtasks, few-shot examples, and/or the basis for evaluating the output generated by the LLM.
4. **Role**: Some LLMs provide users with the option of specifying the system's role. The response of LLM can vary due to changes in role definitions, although the prompt content remains unchanged.

### Detailed Plots for Five Criteria

These figures (Figures 9, 10, 11 and 12) depict the comparative performances of three LLMs in terms of their (human-perceived) precision/recall distributions along four criteria: Common Strengths, Common Weaknesses, Common Suggestions, and Literature Review. Here, human-perceived precision and recall distributions are derived from their qualitative rating counts, i.e., {SA=Strongly Agree, A=Agree, N=Neutral, D=Disagree, SD=Strongly Disagree}, followed by a normalization term to convert them into a valid distribution (individual values ranging between [0-1]). Refer to Figures 9, 10, 11 and 12 for these detailed results.

### Prompt Design Details
Prompt designs during our preliminary study and final case study are shown in Table 4. Also, the prompt design for automatic evaluation is presented in Table 5 and Table 6.

### Responsible NLP Checklist

* **Potential Risks:** This work has no potential risk. It is a purely curiosity-driven academic study.
* **Human Annotator Recruitment:** We recruited eight graduate students and two undergraduates for the annotation task. Undergrad human annotators were paid at the hourly rate of $15/hour, while graduate student annotators were paid monthly stipends at the university-prescribed rate.
* **Annotation Guideline:** Detail instructions were given to human annotators for Microevaluation and Macro-Evaluation. Instructions in these evaluation forms are presented in Figures 13, 14, and 15.
* **Data Privacy and Anonymity Measures:** Any data relating to human subjects (e.g., evaluation survey/interview results) was reviewed to ensure confidentiality and privacy before being preparing the manuscript. We did not collect any user behavior data when annotators performed qualitative evaluations. Finally, we conducted a user survey and reported the summary data from the survey. The survey did 

not collect any user demographic and personal background information; no identifiable information was stored or reported during/after the lifecycle of this work.
* one of the Creative Commons licenses. This license lets others distribute, remix, tweak, and build upon our project as long as users credit the project funded by NSF for the original creation. We decided to use this license because Attribution CC BY is the most accommodating oflicenses offered for maximum dissemination.

Fig. 8: Core Contributions Ratings - Prompt TELeR Level 1-4.

Fig. 7: TELeR Taxonomy for prompting LLMs to perform complex tasks. For details, see Santu and Feng (2023).

 



### An Example Case

Three reviews used for the Preliminary Study are as below: The full text of these reviews follows -

#### a.5.1 Review 1

Review No.: 1 This is full review. Review Date: 10/24/2020 Summary of the paper : The authors propose to improve the sample quality of autoregressive models. The authors propose to (1) - smooth the input data distribution leveraging methods that have shown success in adversarial defense, (2) recover input distribution by learning to reverse the smoothing process. The authors first demonstrate the efficiency of their method on ld toy-problem, and extend the demonstration to more complex datasets such as MNIST, CIFAR-10 and CelebA with application such as image generation, inpainting and denoising. Pros :

Fig. 9: Common Strengths Ratings - Prompt TELeR Level 1-4.

Fig. 10: Core Weaknesses Ratings - Prompt TELeR Level 1-4.



Figure 12: Literature Review Ratings - Prompt TELeR Level 1-4.

Figure 11: Core Suggestions Ratings - Prompt TELeR Level 1-4.

 

The idea to leverage a method

previously used for adversarial defense to density estimation is interesting and novel. The paper

previously used for adversarial defense to density estimation is

\begin{table}
\begin{tabular}{|c|p{142.3pt}|} \hline Level & Prompt (Preliminary Study) & Prompt (Case Study) \\ \hline
1 & messages=[“role”: “system”, “content”: “As a meta-reviewer, answer the following: What would be a reasonable meta-review considering the given reviews?”; “user”, “content”: 1.\textless{}R1>2.<R2>3.<R3>] & Using 3 reviews given below draft a meta review: 1.Review#1\n\(<R1>\backslash\)n 2.Review#2\n\(<R2>\backslash\)n 3.Review#3\(\backslash\)n\(<R3>\) \\ \hline
2 & messages=[“role”: “system”, “content”: “You are a meta-review assistant. You should create a meta review by answering the following questions: ” “(a) According to reviews, what are the core contributions?” “(b) What are the reasons mentioned in the reviews?” “(c) What are the weaknesses as mentioned in the reviews?” “(d) What suggestions would you provide for improvement?”” (e) What are the missing references as described in the reviews?”, “role”: “user”, “content”: 1.\textless{}R1>2.<R2>3.<R3>] & Using 3 reviews given below draft a meta review: 1.Review#1\n\(<R1>\backslash\)n 2.Review#2\(\backslash\)n\(<R2>\backslash\)n 3.Review#3\(\backslash\)n\(<R3>\) \\ \hline
3 & messages=[“role”: “system”, “content”: “As a meta-reviewer, draft a meta review as per the following directions: ”"(a) State core contributions.” “(b) Huntington strengths.” “(c) Mention weeknesses.” “(d) Write suggestions for improvement.” “(e) State missing references. “role”: “user”, “content”: 1.\textless{}R1>2.<R2>3.<R3>] & You are a meta-review assistant. Using three reviews given as Review#1, Review#2, and Review#3, give me a meta-review by answering: * Maintion core contributions with common contributions first. * A common strength is a strength that is mentioned in at least 2 reviews as a strength. Mention common strengths. * A common weakness is a weakness that is mentioned in at least 2 reviews as a weakness. Mention common weaknesses. * A common suggestion for improvement is a suggestion that is mentioned in at least 2 reviews as a suggestion for improvement. Mention common improvements suggested. * State whether reviews refer to missing references or not. A listing of missing references is not required. Three reviews are as follows:’\n 1.Review#1\n\(<R1>\backslash\)n 2.Review#2\n\(<R2>\backslash\)n 3.Review#3\(\backslash\)n\(<R3>\) \\ \hline
4 & messages=l “role”: “system”, “content”: “As a meta-reviewer, draft a meta review by answering the following bullet points: ” – What is the summary of core contributions? Provide answer with supporting evidences.” & Using three reviews given as Review#1, Review#2, and Review#3, as a meta-reviewer, your task is to draft a meta review by answering the following bulleted questions:” & Using three reviews given as Review#1, Review#2, and Review#3, as a meta-reviewer, your task is to draft a meta review by answering the following bulleted questions:” & What is the summary of core contributions? Provide answer with supporting evidence. \\  & \multicolumn{1}{p{142.3pt}}{– Which common strengths are referred to in the reviews? Support your answer with explanations.”””What common weaknesses are described in the reviews? Give evidences in support of the reply.” “What suggestions for improvement have been provided by three reviews? Explain the basis for the answer.” “Which missing references are mentioned in the reviews? Answer with explanations will be desirable.”, “role”: “user”, “content”: 1.\textless{}R1>2.<R2>3.<R3>] & - Which common strengths are referred to in the reviews? An answer with explanations will be desirable.”, “role”: “user”, “content”: 1.\textless{}R1>2.<R2>3.<R3>] & - Which common strengths are referred to in the reviews? A common strength is a strength that is mentioned in at least 2 reviews as a strength. Support your answer with explanation. \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 4: Prompt Design for different levels 

(which results in densities with high Lipschitz constants) and compounding errors. The theory is strong Cons :

The experiments on denoising and inpainting are only qualitative and suffer from a lack of quantitative evaluation. Recommendation :

The article is clear, well motivated, and have a strong theoretical grounding. Therefore I would tend to accept the article. Detailed comments :

The experiment on 2d synthetic datasets (especially the olympic dataset) should be discussed more thoroughly. First, it is not clear that the proposed model is generating better sample than the MADE baseline on this specific dataset. Second, the intersection between rings, in the olympic dataset, seems to be much poorly modeled with the proposed approach compared to the MADE baseline. What is the reason ?

In the section 3.2 the authors are introducing 2 different

\begin{table}
\begin{tabular}{|p{140.9pt}|} \hline Evaluation Prompt \\ \hline Imagine you are a human annotator evaluating the quality of meta-reviews written for a conference. You will evaluate five aspects of the meta-review by giving a value from 1 to 5, with 1 being the worst and \(5\) being the best, without providing any additional explanation. Please follow these steps: \\ 1. Carefully read the reviews, noting the key information they contain. \\ 2. Read the meta-review thoroughly. \\ 3. Rate the meta-review on the following five aspects using Precision (P) and Recall (R): \\  * Core Contributions (CC) Common Strengths (CS) Common Weaknesses (CW) Suggestions for Improvement (SI) Missing References (MR) \\ 4. Provide your ratings on a scale from 1 (worst) to 5 (best) for each aspect in both Precision and Recall. Definitions are as follows: * Core Contributions (CC): Does the meta-review contain the contributions of the paper mentioned in multiple reviews? \\ * Common Strengths (CS): Does the meta-review include all the strengths that are common across reviews? \\ * Common Weaknesses (CW): Does the meta-review include all the weaknesses that are common in multiple reviews? \\ * Suggessions for Improvement (SI): Does the meta-review include all the common suggestions for improvement mentioned in multiple reviews? \\ * Missing References (MR): Does the meta-review include all the missing references mentioned in multiple reviews? Specific Questions for Rating: \\ * CC-P: Is the meta-review precise in capturing the contributions of the paper highlighted by at least two reviews? \\ * CC-R: Has the meta-review covered all the contributions of the paper, which are highlighted by at least two reviews? \\ * CS-P: Is the meta-review precise in capturing the common strengths highlighted by at least two reviews? \\ * CS-R: Has the meta-review covered all the common strengths highlighted by at least two reviews? \\ * CW-P: Is the meta-review precise in capturing the common weaknesses highlighted by at least two reviews? \\ * CW-R: Has the meta-review covered all the common weaknesses highlighted by at least two reviews? \\ * SI-P: Is the meta-review precise in capturing the suggestions highlighted by at least two reviews? \\ * SI-R: Has the meta-review covered all the suggestions highlighted by at least two reviews? \\ * MR-P: Is the meta-review precise in capturing the missing references highlighted by at least two reviews? \\ * MR-R: Has the meta-review covered all the missing references highlighted by at least two reviews? Now complete the task for the given reviews and meta-review. Input: \\ 1.Review#1\n< _RD_ >=n 2.Review#2\n< _RD_ >=n 3.Review#3\n< _RD_ > Output: Provide your evaluated scores in a list: \\ \end{tabular}
\end{table}
Table 5: Prompt Design for automatic Micro evaluation 

debiasing methods (either a denoising step or another autoregressive model). In the rest of the article it is not clear which of the two methods the authors are using. In addition, in the 2d toy-problem (i.e. ring and olympic) as the authors are choosing a gaussian smoothing both debiasing methods are usable. Therefore it would be interesting to show both methods and to describe thoroughly the differences (in addition, it might provide an answer to my previous point). The authors should not mention denoising and inpainting applications if there is no quantitative assessment (at least in appendix)... For the inpainting part, the corrupted input are not even shown (which part of the image has been predicted). The denoising and inpainting experiments sounds like it's been rushed... Typos and suggestions to improve the paper : Minor : Both theorems are provided with nice demonstrations, then the authors should refer to the demonstration in the coretext of the article (e.g. see Appendix A). Minor : Add small arrows in Table 2 to indicate that Inception score is better when lower, and opposite for FID
* Type : page 5, section 3.3, paragraph 2 : relative |> relatively Figure3 : Right panel : What are the 3 shaded curves ? This should be shown in the legend or at least in the caption Figure3 : Right panel: In the x-axis it should be specified 'Variance of

\begin{table}
\begin{tabular}{l l} \hline Evaluation Prompt & \\ \hline Imagine you are a human annotator tasked with evaluating a model’s performance in generating a meta-review based on three individual reviews. You are provided with the following: & \\  & The **prompt** that guided the generation of the meta-review. & \\  & The **three individual reviewers**, which the model used to generate the meta-review. & \\  & The +=generated meta-review** created by the model. & \\  & The **actual expert-written meta-review** that serves as a reference for what the meta-review should ideally look like. & \\ Please carefully examine this information and evaluate the model’s performance based on three criteria. Provide a score on a Likert scale from 1 (poor) to 5 (excellent) for each aspect: & \\  & 1. **Adherence to instructions**: How well did the model follow the specific instructions given in the prompt? Consider how accurately the model addressed the tasks or questions posed in the prompt when generating the meta-review. - Score: [1-5] & \\  & 2. **Ability to create useful Meta-Reviews **: Evaluate the usefulness of the generated meta-review in terms of its practicality and effectiveness for someone preparing a comprehensive meta-review. Consider whether the generated content helps synthesize the individual reviews into a coherent and insightful summary. - Score: [1-5] & \\  & 3. **Matching against actual expert-written meta-reviews**: Assess the extent to which the model-generated meta-review aligns with the expert-written meta-review. This involves comparing the content, tone, insights, and overall quality. - Score: [1-5] \\ Input: & **Prompt** \\  & ProMPT\(\backslash n<P>\backslash n\) & \\  & **Reviews**:** & \\  & 1.Review#1\(\backslash n<R1>\backslash n\) & \\  & 2.Review#2\(\backslash n<R2>\backslash n\) & \\  & 3.Review@3\(\backslash n<R3>\backslash n\) & \\  & 4*Generated Meta-review:** & \\  & Generated Meta Review\(\backslash n<GMR>\backslash n\) & \\  & **Actual Meta-review:** & \\  & Actual Meta Review\(\backslash n<AMR>\) & \\ Output: & \\  & Please provide your evaluation in numerical format in a list. & \\ \end{tabular}
\end{table}
Table 6: Prompt Design for automatic Macro evaluation 

Performance Ratings on 5 aspects - LLM!P1* This question seeks your input on five aspects on 5 scales - Strongly agree, Agree, Neutral, Disagree, and Strongly disagree. These scales have been abbreviated as SA, A, N, D, and SD respectively in order to render the form in a compact form. The aspects are:

* A meta-review needs to capture core contributions in the reviews. On the basis of instructions contained in the prompt and the response generated by LLM, you need to provide your rating about the performance of the LLM against the item **Core Contrib.**
- P and **Core Contrib.**
- R where subscript P and R correspond to precision and recall respectively. Please record your rating about the following statement:

"_While generating meta-review, LLM1 performed very well in highlighting the core contributions._"

* The process of meta-review generation involves capturing common strengths mentioned by reviewers. You are requested to provide your rating about the performance of the LLM against the item **Strengths**
- P and **Strengths**
- R where subscripts P and R correspond to precision and recall respectively. To what extent do you agree with the following statement?:

"_LLM1 included all the strengths that are common in the reviews._"

* The process of meta-review generation involves capturing common weaknesses in the reviews by reviewers. You are requested to provide your rating about the performance of the LLM against the item **Weaknesses**
- P and **Weaknesses**
- R where subscripts P and R correspond to precision and recall respectively. To what extent do you agree with the following statement?:

"_LLM1 included all the weaknesses that are common in the reviews._"

* Reviewers provide suggestions for improvement. These suggestions may be quite varied from one reviewer to another. The process of meta-review generation involves incorporating common suggestions for improvement. You are requested to provide your rating about the performance of the LLM against the item **Suggestions**
- **P** and **Suggestions**
- **R** where subscripts P and R correspond to precision and recall respectively. To what extent do you agree with the following statement?:

"_LLM1 included all the common suggestions for improvement recorded in the reviews._"

* Reviewers inform about missing references, links etc. Meta-review is expected to encompass all such instances highlighted by the reviewers. You are requested to provide your rating about the performance of the LLM against the item **References**
- **P** and **References**
- **R** where subscripts P and R correspond to precision and recall respectively. To what extent do you agree with the following statement?:

"_LLM1 included all the missing references mentioned in the reviews._"

**q**(\(\tilde{x}\mid x\))

Page 7 : paragraph 1 : 'Thus, it is hard to conclusively determine what is the best way of choosing q(x|\(\tilde{x}\)).' |> I think the authors actually give the key to properly choose the noise level (i.e. variance). It seems to depend on the task : if one wants to generate good samples, then the variance has to be set by heuristic. If one needs

Figure 13: Micro Evaluation Instructions



a good likelihood (e.g. for subsequent downstream tasks) then the variance could be optimized.

Figure 6: On my understanding, the part 'denoising' is redundant with the section image generation. It is interesting to mention the denoising application, but I am not convinced of the utility of the figure 6.

Figure 7: What is the corrupted input ? Which part of the input has been masked ??

#### a.5.2 Review 2

Review No.. 2 This is full review. Review Date: 10/25/2020 Summary. Autoregressive models have demonstrate their potential utility for modeling images and other types of complex data with high flexibility (particularly in density estimation). However, its sampling ability is not that good as explained in the paper.

Fig. 14: Micro Evaluation Form 

Based on your evaluation in part 1, to what extent do you agree with these statements:

Overall assessment on 5 point scale - LLM1 *

* On the basis of instructions contained in the prompts and the responses generated by LLM, you need to provide your rating about the performance of the LLM against the item **Statement**. Please record your rating about the following statement Statement1: Statement 1: "_While generating meta-review, LLM1 strictly followed all the instructions provided in the prompts._"

Other Statements, you need to rate are:

Statement 2: "_LLM1 is able to create useful meta review._"

Statement 3: "Meta review generated by _LLM1 matches with the actual meta review to a great extent._"

Pleas keep in mind the following abbreviations:

SA: Strongly Agree A: Agree N: Neutral D: Disagree SD: Strongly Disagree

Kindly provide comments about _LLM1_'s performance in generating Meta-review. *

Authors show that one of the main weaknesses of autoregressive models comes from the propagation of mistakes due to the mismatch of conditionals. Inspired in the promising results of randomized smoothing in adversarial models (Cohen et al. 2019), authors propose a similar strategy. The addition of Gaussian noise and posterior modeling of the smoother data makes easier to the autoregressive density to capture the true data distribution. The benefits of this strategy are empirically proved and shown in the experiments. Strengths. The quality

Figure 15: Macro Evaluation Form



of writing is high and the presentation of the paper facilitates the process of reading. I have to say that I enjoyed while reviewing it. The analysis and description of problems for sampling from autoregressive models is completely understandable to me and I agree with the manifold hypothesis held. Results with the "sharp" multimodal data looks reliable to me and I believe that the smoother process can also reduce the lipschitz constant as stated in Theorem 1. Until pp. 5, nothing is said about the data denoising process, so one could initially think that there is no way to recover the target density without noise, but authors also did an effort on this. Good point. It is important to remark that the randomized smoothing process can be reverted once learning finishes. Additionally, I particularly like how authors first present the idea on l-d examples, later in the experiments, the method is validated with 2-d rings and finally, as stated in the introduction, with different image datasets. Finally, I did not find any similar work that mixes the idea of smoothing for improving autoregressive modelling. Weaknesses, Questions & Recommendations. To me, there are 3 main points of weakness:
[W1]. A lack of analysis about the optimal noise for randomized smoothing. [W2]. Why just Gaussian noise, what if data is discrete, could we do this with another type of noise?
[W3]. Comments about denoising are included a bit late in the manuscript. I think that authors should remark that this is a reversible process. My main questions are: [Q1]. In section 2.2, I don't see why data closer to the manifold should have larger first order derivatives or even infinity. Is this a bit counter intuitive, or not? Like, better positioned, worse gradient values? [Q2]. Is the l/N term in the global likelihood expression of lst paragraph of section 2 correct?
[Q3]. If I do not appropriately choose the \(\sigma\) parameter for smoothing, do I have the risk of not capturing some modes of the original data? I have the opinion that adding too much or too less noise to data could "mask" modes and something could be lost. Am I correct? Did authors empirically analyzed this in the experiments? [Q4]. How could we assess that conditionals are now better fitted than before? A few recommendations for improvement: [Recl]. I would explain a bit more the manifold hypothesis of section 2.2, maybe a diagram or figure would help for quicker comprehension of the problem. [Rec2]. Some acronym for \randomized smoothing" would help in the lst paragraph of section 3.1. To avoid repetitive expressions. Reasons for score. I liked the idea, think that the paper is well written and I trust the results presented by the authors. Despite the randomized smoothing strategy is rather simple, it seems to work particularly well. For this reason I tend to vote for accept. If I not set a higher score, it is because a bit more of analysis on the optimal sigma, distribution for smoothing and lipschitz constant could have been included. Post-rebuttal update. Thanks 

to the authors for their response to all my questions and comments. I also read the updated version of the manuscript, which is clearly improved and the rest of reviews and comments by the AC. Looking to that, I agree with the rest of reviewers about the quality of the paper, so I raised my score and I recommend to accept it.

#### a.5.3 Review 3

Review No.: 3 This is full review. Review Date: 10/27/2020 SHORT DESCRIPTION This paper proposes a two-stage generative modeling approach, first learning a distribution over noised data, then learning the original data distribution conditioned on this noised data. The paper demonstrates that this leads to improved sample quality compared to fitting the data distribution directly. DISCUSSION Overall, I like this paper: it's straightforward idea, decently motivated and fairly well described, and has good supporting empirical evaluation. I didn't expect the sampling performance to improve substantially by adding just a single denoising step, and I think demonstrating this is a good contribution. However, I think the paper could be improved by some more careful discussion, and a better placement in the literature. "Theorem 2 shows that our smoothing process provides a regularization effect on the original objective... This regularization effect can, intuitively, increase the generalization capability of the model." How does the extra term in the theorem lead to a regularization effect? Why does this ''intuitively'' increase the generalization capability of the model? Unless I'm mistaken, the added terms (up to a constant) the Laplacian of the log-likelihood w.r.t. data. The objective maximizes this on average across observed data, which intuitively minimizes the 'curvature' or 'steepness' of the log-likelihood at observed data, thus presumably smoothing the maximum likelihood solution. This Laplacian term also appears in the score matching objective presented in Theorem 1 of 'Estimation of Non-Normalized Statistical Models by Score Matching, Hyvarinen 2005', where it is minimized instead of maximized. There are also known connections between score matching and denoising methods e.g. 'Optimal Approximation of Signal Priors, Hyvarinen 2006', and 'A Connection Between Score Matching and Denoising Autoencoders, Vincent 2011', which you've cited in passing later. Much of this material and how it relates to the objective in Theorem 2 might be discussed in more depth rather than passing over it as simply a 'regularization term'. "Our approach is related to two-stage VAE (Dai & Wipf, 2019) which introduces a second VAE to correct the errors made by the first VAE." I'm not sure I agree with this. The idea of the two-stage VAE in that paper is to clean up mismatch between the aggregate posterior q(z) and the prior p(z). On the other hand, your variational model is identical to the canonical VAE setup: xis data, zis noised data, the 'posterior' q(z | x) is fixed and adds Gaussian noise, the 'prior' p(z) is a powerful autoregressive model, and the observation 

likelihood p(x | z) is another powerful autoregressive model (the canonical VAE would have learned q(z | x), fixed simple p(z), and learned but simple p(x | z)). This is one of the reasons 'VAE' can a confusing term when used to describe latent variable generative models in general: assuming something should be 'encoded' and 'decoded' can sometimes obfuscate the actual probabilistic model. What you propose in this paper might generically be called a 'denoising VAE', but again that's maybe not the most accurate description. I think the most closely related work is probably the denoising diffusion and denoising score-matching approaches which have received attention recently and which you've mentioned, but you could also think of it as turning a denoising autoencoder into a generative model. In any case, I think a more careful discussion of these points would be beneficial for the paper. Finally, the approach isn't really tied to autoregressive models, apart from the motivation given in terms of smoothing 1D distributions. It's fairly likely that the same idea could readily be applied to e.g. normalizing flows and that it would work well there also, so it would have been nice to see experiments featuring flows included here. This would be especially useful since your best-performing two-stage method takes autoregressive models, which are already slow samplers, and effectively doubles their sampling time. EXTRA NOTES Maybe careful with the word 'spurious' in the intro - I know what you mean, but samples from a model are by definition typical samples from that model, and there's nothing spurious about them. They're only questionable when compared to data. Similarly: "The "erroneous" sample x", in some sense, resembles an adversarial example, i.e., an input that causes the model to make mistakes." This seems to be implying that samples generated by a model are somehow pathological. By virtue of the fact that they are generated by the model, they are by definition typical samples from the model. There is nothing pathological whatsoever about them. Why the model specification and fitting procedure have resulted in such samples, and whether the samples resemble training data or not, is another issue entirely. 'However, this approach bounds the capacity of the model by limiting the number of modes for each conditional.' All models have limited capacity -- what is particularly bad about the capacity of an autoregressive model being limited in this way? Do we have reason to believe inability to cover multiple nodes is a common bottleneck? Figure 2 & Figure 3: Axis ticks are too small (and there probably too many), whole figure could be made bigger (this would also help with legends cutting off a lot of the plots). 'Proofs' for theorems 1 and 2 should be referred to in the main text. Theorem 2 should also have log p(x) on the LHS? Figure 5 caption: "All the samples are not conditioned on class labels." -> None of the samples are conditioned on class labels. What exactly is being inpainted in Figures 7 (a) and (b)? 

Since a central claim of the paper is that the method results in improved sample quality, it might be good to add the Kernel Inception Distance ("Demystifying MMD GANs" Binkowski et al 2018) which has many favourable properties over FID, and is really no more difficult to compute. CONCLUS Overall, I think this paper is anice submission, and would like to see it accepted given a few tweaks. UPDATE: I've upped my scoreto a 7, and would like to see the paper accepted.

 



### Preliminary Study with GPT-3.5's and TELeR Taxonomy Based Prompting

These suggest "role": "system," content": 1) n \(<Ri>\backslash\)n2,n \(<R2>\backslash\backslash\)n3,n \(<R3>\).]

Three ways of using GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 1) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a small number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 4) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 6) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 7) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 8) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 9) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 11) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 12) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 13) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 14) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 15) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 16) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 17) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 18) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 19) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 19) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 3) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 10) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.5's in a large number of cases are: 2) the number of events that have been generated by GPT-3.5's to generate the most popular GPT-3.



messages=[

"role": "system", "content": "As a meta-review assistant,"

"write a meta review mentioning core contributions, strengths, "

"weaknesses, suggestions for improvement, and "

"missing references."

"role": "user", "content": '1.\n< \(R1>\backslash\)n2.\n< \(R2>\backslash\)n3.\n< \(R3>\)'.]

The three meta reviews provide a comprehensive evaluation of the paper, discussing its contributions, strengths, weaknesses, and suggestions for improvement. The first review praises the novelty of the proposed method, its strong theoretical grounding, and the clear writing style. However, the reviewer suggests that the authors should provide a more in-depth analysis of the results on the 2D synthetic datasets, clarify the debiasing methods used, and provide quantitative evaluations for the denoising and inpainting applications. The reviewer also suggests minor improvements in the text and figures.

The second review prussise the simplicity and effectiveness of the proposed method, the quality of the writing, and the strong empirical validation. However, the reviewer notes some weaknesses related to the lack of analysis about the optimal noise, the use of Gaussian noise, the late inclusion of denoising comments, and the absence of a discussion of the literature on score matching, denoising methods, and two-stage VAEs. The reviewer also suggests some recommendations for improvement, such as a clearer explanation of the manifold hypothesis, acronyms for the proposed method, and experiments with normalizing flows.

The third review similarly prasies the straightforwardness of the idea, the well-described motivation, and the sound empirical evaluation. However, the reviewer suggests a more careful discussion of the Theorem 2 regularization effect, a better placement in the literature, and the inclusion of experiments with normalizing flows. The reviewer also provides extra notes on the use of language, the capacity of the models, the figures, the proofs, and the evaluation metrics.

Overall, the three meta reviews agree that the paper presents a promising and well-executed idea, but could benefit from some improvements in the analysis, discussion, and experiments. The reviewers suggest that the authors should address the weaknesses and recommendations in a revision and potentially add more experiments with normalizing flows. Based on the reviews, the paper seems suitable for acceptance if the authors address the suggested improvements in the revision.

Srief Analysis and Systems

Adherence to instructions: GPT-3.5 has not mentioned anything about the missing references. Other instructions have been adhered to.

Common Strengths: GPT-3.5's output mentions strengths for each review separately. The overall strength is captured very briefly in the last paragraph. The response cannot be termed as good in terms of meta-review where the expectation is to get summary information.

Common Weaknesses: GPT-3.5 mentions weaknesses for each review separately. The overall weakness is not mentioned. The quality of weakness description is good. The response can be termed as good for meta-review although the summary information is missing.

Common Suggestons for improvement: GPT-3.5 has provided suggestions for improvement for each review separately. The overall suggestion for improvement is very brief in the last paragraph. The quality of is good. The response can be termed as good for meta-review.

Coverage: GPT-3.5 has covered all the three reviews. GPT-3.5 has not mentioned about missing references at all. GPT-3.5 has answered only the first 4 questions out of total 5 by taking into account three reviews.

Scores:

1. Adherence to instructions: 8/10

2. Common strengths: 6/10

3. Common weaknesses: 6/10

4. Common suggestions for improvement: 6/10 

messages("role": "system," content); "You are a meta-review assistant.",

"role": "user," content/": 1\n<\(R1>\backslash\)n2\({}_{x}\rangle\n<R2>\backslash\)n3\({}_{v}\rangle\n<R3>\);

"role": "user," content?: "Write a meta review mentioning core contributions, strengths,"

"weaknesses, suggestions for improvement, and"

"missing references."][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnote=!][footnoteootnoteootnote=!][footnoteootnoteootnote=!][footnoteootnoteootnote=!][footnoteootnoteootnote=!][footnoteootnoteootnote=!][footnoteootnoteootnote=!][footnoteootnoteootnote=!][footnoteootnoteootnote=!][footnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnoteootnoteootnote=!][footnoteootnoteootnoteootnoteootnoteootnoteootnote=!][footnote 

messages("role": "system", "content": "You are a meta-review assistant. ",

"role": "user", "content": 1)\({}_{n}<R1>\backslash\)2)\({}_{n}\backslash\)\(<R2>\backslash\)u3)\({}_{n}\backslash\)=\(R3>^{*}\)]

"role": "user", "content": "What would be a reasonable meta-review considering the given reviews?".]

###### Abstract

The proposed approach is to identify and analyze the effects of the underlying process on the development of the process. The main focus is to develop an analysis of the effect of the process on the development of the process. In this paper we propose a novel model for the influence of the process on the development of the process on the development of the process. We present a new model for the influence 

messages="role: "system", content=" You are a meta-review assistant.",

"role", "user", "content": '1,n\(<R1>\backslash\)m2\(\backslash\)u\(<R2>\backslash\)n3\(\backslash\)n\(<R3>\)',

"role:", "user", "content": "Create a meta review by answering the following questions."

"(a) According to reviews, what are the core contributions?"

"(b) What are the strengths as mentioned in the reviews?"

"(c) What are the weaknesses as mentioned in the reviews?"

"(d) What suggestions would you provide for improvement?"

"(e) What are the missing references as described in the reviews?". ]
 

messages["role": "system", "content": "As a meta-reviewer,"

"draft a meta review by answering the following questions in separate paragraphs:"

"What would be a summary of core contributions?"

"Which strengths are referred to in the reviews?"

"What would be the overall weakness description?"

"Could you please provide suggestions for improvement using three reviews?"

"Which missing references are mentioned in the reviews?",

"role": "user", "content": '1.\n< _R1_>=**02**,_>=**03**,_>=_< _R3_>^*.]

(Hulse Response)

Core Contributions: The paper proposes a novel approach to improve the sample quality of autoregressive models by smoothing the input data distribution using methods previously used for adversarial defense and then recovering the input distribution by learning to reverse the smoothing process. The paper demonstrates the efficiency of this method on 1D toy problems and then extends the demonstration to more complex datasets such as MNIST, CIFAR-10, and CelebA with applications such as image generation, inpainting, and denoising.

Strengths: The paper is well written and clearly presented, and the motivation behind the proposed approach is adequately explained. The empirical results are convincing, and the experimental evaluation is thorough. The paper is novel, and the proposed approach shows significant improvement in sample quality.

Weaknesses: The paper could be improved by providing a more detailed analysis of the optimal noise for randomized smoothing and its effect on the Lipschitz constant. The paper could also benefit from a more thorough discussion of related work and the limitations and assumptions of the proposed approach.

Improvement Suggessions:

1. The authors should provide a more detailed analysis of the optimal noise for randomized smoothing and its effect on the Lipschitz constant.

2. The authors should discuss related work more thoroughly and explain the limitations and assumptions of the proposed approach.

3. The authors could consider adding experiments featuring normalizing flows to demonstrate the applicability of the proposed approach to other generative models.

Missing References: None mentioned in the reviews.

Fixed Analysis and Issues

Adherence to instructions: All the instructions have been followed by GPT-3.5.

Common Strengths: The strengths in GPT-3.5's response are correct but the content is inadequate.

Common weaknesses: The weaknesses mentioned by GPT-3.5 are better in response to strength both in terms of quality and quantity.

Common Suggestations for improvement: The suggestions for improvement by GPT-3.5 correct and better than response to strengths.

Coverage: It seems that GPT-3.5 has considered all the reviews for responding to the above instructions.

Sceores:

1. Adherence to instructions: 10/10

2. Common strengths: 6/10

3. Common weaknesses: 7/10

4. Common suggestions for improvement: 7/10

5. Coverage: 10/10 

messages[" role] : "system", "content". "You are a meta-review assistant.",

"role". "user". "content". '1.|n< \(R1>\)=n2.|n< \(R2>\backslash\)n3.|n< \(R3>^{*}\),

"role". "user". "content".

"Draft a meta review by answering the following questions in separate paragraphs."

"What would be a summary of core contributions?"

"Which strengths are referred to in the reviews?"

"What would be the overall weakness description?"

"Could you please provide suggestions for improvement using three reviews?"

"Which missing references are mentioned in the reviews?". JJ

CIFARAS Redpaper

Summary of Core Contributions:

The paper proposes a novel approach to improve the sample quality of autoregressive models by leveraging randomized smoothing. The authors introduce a two-stage generative modeling approach that first learns a distribution over noised data and then learns the original data distribution conditioned on this noised data. The paper demonstrates through experiments that this approach leads to improved sample quality compared to fitting the data distribution directly. Additionally, the paper provides theoretical analysis and strong motivation for their method.

S Strengths:

The reviews mention several strengths of the paper. Firstly, the paper is well-written and easy to understand. Secondly, the idea proposed by the authors is novel and has not been explored before. Thirdly, the experiments presented in the paper are thorough and demonstrate the effectiveness of the proposed approach on various datasets. Lastly, the paper provides strong theoretical analysis and motivation for their method.

Overall Weakness Description:

The reviews point out a few weaknesses in the paper. Firstly, the paper lacks a detailed analysis of the optimal noise for randomized smoothing. Secondly, the paper does not provide a clear explanation of why only Gaussian noise was used and whether the method could be extended to discrete data. Thirdly, the paper could benefit from a more careful discussion of related work and its relation to the proposed method. Fourthly, some of the experiments lack quantitative evaluation, and the authors should provide more detailed discussions of their results.

Suggents for Improvement:

The reviews provide several suggestions for improvement. Firstly, the authors should provide a more detailed analysis of the optimal noise for randomized smoothing. Secondly, the authors should explain why only Gaussian noise was used and whether the method could be extended to discrete data. Thirdly, the authors should provide a more detailed discussion of related work and its relation to the proposed method. Fourthly, the authors should provide more quantitative evaluation of their results and provide more detailed discussions of their experiments. Additionally, the reviews suggest adding the Kernel Inception Distance as a metric to compare different generative models.

Missing References:

The reviews do not mention any missing references in the paper.

Brief Analysis and Sources 

messages["role": "system", "content": "As a meta-reviewer, "

"draft a meta review as per the following directions:"

"a) State core contributions."

"b) Mention strengths."

"c) Mention weaknesses."

"d) Water suggestions for improvement."

"e) State missing references."

"role": "user", "content": '1.\n< \(R1>\backslash{}\)n2\(\backslash{}\)n< \(R2>\backslash{}\)u3\(\backslash{}\)n< \(R3>^{*}\).]

Contributions:

The paper proposes a two-stage generative modeling approach, which uses a denoising step to improve the sample quality of autoregressive models. The authors demonstrate that this method leads to better sample quality compared to fitting the data distribution directly. The paper is well-written and presented, and the empirical evaluation supports the proposed method's effectiveness.

Strengths:

The paper presents a straightforward and novel idea, which is well-motivated and supported by empirical results. The presentation and writing style are clear and easy to understand, and the paper's structure facilitates easy reading. The results show substantial improvements in sample quality, which is a significant contribution to the field.

Weaknesses:

The authors could provide more detailed discussion and analysis of the method's underlying theory, particularly the relationship between the added term in Theorem 2 and regularization. The paper's description of related work using similar methods (e.g., denoising diffusion and denoising score-matching) could be more detailed. Finally, while the paper's empirical evaluation is strong, the authors could add Kernel Inception Distance for more rigorous comparison to other methods.

Suggations for Improvement:

The authors should provide more detailed discussion and analysis of the method's underlying theory, particularly the relationship between the added term in Theorem 2 and regularization. The paper's description of related work using similar methods (e.g., denoising diffusion and denoising score-matching) could be more detailed. The authors could add Kernel Inception Distance for more rigorous comparison to other methods.

Missing References:

"Estimation of Non-Normalized Statistical Models by Score Matching, Hyvarinen 2005" and "Optimal Approximation of Signal Priors, Hyvarinen 2006" are mentioned in passing but could be discussed in more detail.

But Analysis and Sources

Adherence to instructions: All the instructions have been followed by GPT-3.5.

Common Strengths: The quality of GPT-3.5's output regarding strengths is not good. Content is lacking.

Common Weaknesses: The quality of content for weaknesses is not good. The responses for both weaknesses and suggestions for improvement are almost the same in spite of the fact the reviewers have described weaknesses and suggestions separately in most of the places.

Common Suggestions for improvement: The suggestions for improvement by GPT-3.5 is fair. The quality is not good. The responses for both weaknesses and suggestions for improvement are almost the same in spite of the fact the reviewers have described weaknesses and suggestions separately in most of the places. Coverage: GPT-3.5 has also mentioned missing references although the answer is not correct. GPT-3.5 has answered all the questions by taking into account three reviews.

Scores:

1. Adherence to instructions: 10/10
2. Common strengths: 5/10
3. Common weaknesses: 5/10
4. Common suggestions for improvement: 6/10
5. Coverage: 8/10 

\({}^{*}\)

\({}^{*}\)
 

The most important feature of the proposed method is that it can be used to identify a new class of features. The first stage involves the input data distribution and the second stage involves the output data distribution conditioned on this model instead by randomized smoothing in adversarial models. The second stage involves learning the original data distribution conditioned on the nested data. The paper demonstrates the effectiveness of this approach on 1D and 2D toy problems, as well as image datasets such as MNIST, CelebA, and CIFAR-10. The paper also provides theoretical justifications for the approach through two theorems.

(a) Strengths: The papers a straightforward idea that its well-motivated, clearly explained, and supported by good empirical evaluation. The paper also provides strong theoretical justifications for the approach. The results of the experiments are reliable, and the paper is well-written and easy to read.

(c) Weaknesses: The paper could benefit from more careful discussion and placement in the literature. The authors should provide more detailed explanations of the regularization effect provided by the smoothing process only if could increase the generalization capability of the model. The authors should also be more precise in their descriptions of related work and avoid confusing terminology. In addition, the paper could benefit from experiments featuring normalizing flows, and the axis ticks in Figures 2 and 3 are too small.

(d) Common improvements suggested: The authors should provide more detailed explanations of the regularization effect provided by the smoothing process, discuss related work carefully, and avoid confusing terminology. The paper could benefit from experiments featuring normalizing flows, and the axis ticks in Figures 2 and 3 are too small. (e) Common missing references: The paper could benefit from adding the Kernel Inspection Distance.

(f) Media/Night/Source

Adherence to instructions: All the instructions have been followed by GPT-3.5.

Common Strengths: The strengths in GPT-3.5's response are correct but the content is inadequate.

Common weaknesses: The weaknesses mentioned by GPT-3.5 are better in response to strength both in terms of quality and quantity.

Common Suggestions for improvement: The suggestions for improvement by GPT-3.5 correct but insufficient.

Coverage: It seems that GPT-3.5 has considered all the reviews for responding to the above instructions. It also responds to missing reference although it is incorrect.

Scores:

1. Adherence to instructions: 10/10

2. Common strengths: 6/10

3. Common weaknesses: 7/10

4. Common suggestions for improvement: 5/10

5. Coverage: 8/10 

\({}^{-}\).

\({}^{-}\).

 

messages["role": "system," content": "You are a meta-review assistant. ",

"role; "user", "content": 1.)n< \(R1>\backslash n2\backslash n<\) \(R2>\backslash n3\backslash n<\) \(R3>\)',

"role; "user", "content": "Give me a meta-review by answering following bullets."

" - Mention core contributions with common contributions first."

" - Mention strengths with common strengths first."

" - Mention weaknesses with common weaknesses first."

" - Motion the common improvements suggested."

" - State the common missing references, if any:".]

### Core contributions:

* The paper proposes a two-stage generative modeling approach for improving the sample quality of autoregressive models.
* The authors demonstrate that adding Gaussian noise and posterior modeling of the smoothed data leads to improved sample quality compared to directly fitting the data distribution.
* The paper includes strong theoretical grounding and empirical evaluation on several datasets.
* Strengths:
* The paper is well-written and easy to follow.
* The idea of leveraging randomized smoothing for density estimation is interesting and novel.
* The theoretical analysis is sound and provides a strong foundation for the proposed method.
* The empirical evaluation shows significant improvements in sample quality on several datasets.

Weaknesses:

* The optimal noise level for randomized smoothing is not thoroughly analyzed.
* The paper focuses only on Gaussian noise and does not explore other types of noise.
* The discussion of related work could be improved, particularly in terms of the relationship to denoising diffusion and score-matching approaches.
* The paper lacks a discussion on how to assess whether conditionals are better fitted than before.

Common improvements suggested:

* Provide a more in-depth discussion of the regularization effect in Theorem 2 and how it increases the generalization capability of the model.
* Carefully discuss the relationship to denoising diffusion and score-matching approaches in the related work section.
* Include experiments with normalizing flows to demonstrate the method's potential applicability beyond autoregressive models.
* Add an assessment of whether conditionals are better fitted than before.

Common missing references:

- None mentioned.

Fixed Analysis and Source

Adherence to instructions: All the instructions have been followed by GPT-3.5.

Common Strengths: The strengths in GPT-3.5's response are correct but the content is inadequate.

Common Weakensess: The weaknesses mentioned by GPT-3.5 are better in response to strength both in terms of quality and quantity.

Common Suggessions for improvement: The suggestions for improvement by GPT-3.5 correct and better than responses to strengths and weaknesses.

Coverage: It seems that GPT-3.5 has considered all the reviews for responding to the above instructions.

Scores:

1. Adherence to instructions: 10/10
2. Common strengths: 5/10
3. Common weaknesses: 7/10
4. Common suggestions for improvement: 8/10
5. Coverage: 10/10 

messages--["role": 'system", content?". "As a meta-reviewer,"

"default a meta review by answering the following bullet points: "

" What is the summary of core contributions? Provide answer with supporting evidence."

" Which common strengths are referred to in the reviews? Support your answer with explanation."

" What common weaknesses are described in the reviews? Give evidence in support of the reply"

" What suggestions for improvement have been provided by three reviews? Explain the basis for the answer."

" Which missing references are mentioned in the reviews? Answer with explanation will be desirable.",

"role:" "user", "content?": '1.\n< \(R1>\) n2.\n< \(R2>\) n3.\n< \(R3>\).]

Summary of core contributions: The paper proposes a two-stage generative modeling approach that improves the sample quality of autoregressive models. The first stage involves smoothing the input data distribution by adding Gaussian noise and learning a distribution over the noised data. The second stage involves learning the original data distribution conditioned on this nosied data. The authors demonstrate the effectiveness of this approach on various datasets such as MNIST, CIFAR-10, CelebA, and synthetic datasets.

Common strengths referred to in the reviews: All reviewers agree that the paper proposes an interesting and novel idea. The paper is well-written, and the presentation is easy to understand. The authors provide a good motivation for the problem, and the theory is well-founded. The empirical evaluation is comprehensive and convincing, and the results show that the proposed method improves the sample quality of autoregressive models.

Common weaknesses described in the reviews: The main weakness of the paper is the lack of thorough analysis of the optimal noise for randomized smoothing. The authors only use Gaussian noise, and it is not clear whether this approach is effective for discrete data. The paper could also be improved by discussing the Laplacian term in the objective of Theorem 2 in more depth, and by placing the paper in the literature more accurately. Some reviewers also noted that the experiments on denoising and impairing are only qualitative and lack quantitative evaluation.

Suggestions for improvement provided by three reviews: The reviewers suggest the following improvements:

Provide a more thorough analysis of the optimal noise for randomized smoothing, and consider using other types of noise for discrete data.

Discuss the Laplacian term in the objective of Theorem 2 in more depth, and provide a more accurate placement of the paper in the literature.

Include more quantitative evaluation for the experiments on denoising and inpainting.

Consider adding experiments featuring flows to test whether the same idea could be applied to them.

Missing references mentioned in the reviews: One reviewer suggests using Kernel Inception Distance (KID) as an additional evaluation metric to supplement FID.

Extended Analyses and Source.

Adherence to instructions: Almost all the instructions have been followed by GPT-3.5 except the explanation part. The core contributions have not focused only on contributions but also on the process. There is a slight departure from the instructions on this aspect. In all the parts, instruction was to provide explanation. No explicit mention of support, explanation or evidence is given in any of the bullet points.

Common Strengths: The strengths in GPT-3.5's response are correct and the content is quite good.

Common Weaknesses: The weaknesses mentioned by GPT-3.5 are better in response to strength in terms of quality.

Common Suggestions for improvement: The suggestions for improvement by GPT-3.5 correct and better than response to common strengths.

Coverage: It seems that GPT-3.5 has considered all the reviews for responding to the above questions.

Scores:

1. Adherence to instructions: 8/10

2. Common strengths: 10/10

3. Common weaknesses: 10/10

4. Common suggestions for improvement: 10/10

5. Coverage: 10/10 

messages[" role]: "system," content": "You are a meta-review assistant. ", "role"; "user",content"; 1.!n\(<R1>\)=n2,n\(<R2>\)=n3,n\(<R3>^{\prime}\); "role"; "user",content"; "Draft a meta review by answering the following bullet points."

- What is the summary of core contributions? Provide answer with supporting evidence."

- Which common strengths are referred to in the reviews? Support your answer with explanation."

- What common weaknesses are described in the reviews? Use evidence in support of the reply"

- what suggestions for improvement have been provided by three reviews? Explain the basis for the answer."

- Which missing references are mentioned in the reviews? Answer with explanation will be desirable."]

###### Abstract

Summary of core contributions: The paper proposes a two-stage generative modeling approach that involves learning a distribution over noised data and then learning the original data distribution conditioned on this noised data. The authors demonstrate that this method leads to improved sample quality compared to fitting the data distribution directly. The paper also provides theoretical support for the approach and validates it through experiments on various datasets.

Common strengths referred to in the reviews: The paper is well written, and the presentation is clear and easy to understand. The core contributions of the paper are interesting and novel, and the theoretical support provided by the authors is strong. The empirical evaluation of the proposed method is also thorough.

Common weaknesses described in the reviews: Some weaknesses highlighted by the reviewers include a lack of analysis on the optimal noise for randomized smoothing, why only Gaussian noise is used, and the lack of discussion about the properties of the Laplacian term in the objective function. Additionally, some reviewers have suggested that the experiments on denoising and inpainting suffer from a lack of quantitative evaluation.

Suggestions for improvement provided by the reviews: The reviewers suggest that a more in-depth discussion of the properties of the Laplacian term in the objective function and its connection to score matching and denoising methods would be beneficial. Additionally, some reviewers suggest that the paper could be improved by analyzing the optimal noise for randomized smoothing and experimenting with other types of noise. The lack of quantitative evaluation in the experiments on denoising and inpainting is also pointed out, and the reviewers suggest that this could be addressed in future work.

Missing references mentioned in the reviews: One reviewer suggests that references to recent work on denoising diffusion and denoising score-matching approaches should be included. Additionally, the Kernel Inception Distance is suggested as a metric that could be used to further validate the improved sample quality claimed by the authors.

Black Analyse and Sources 

messages=[ "role": "system", content": "Considering yourself to be a meta-reviewer,"

" draft a meta review by answering the following points."

"a) What would you consider to be the most balanced opinion about core contributions and why?"

"b) Which strengths should be considered at your level and why?"

"c) What are the weaknesses that deserve attention at your level and why?"

"d) What suggestions for improvement are worth considering? Provide explanation."

"e) Which missing references do you consider appropriate for meta review? Support with explanation." ,

"role": ""user", "content": '1.n\(<R1>\backslash\)n2\(\backslash\)n\(<R2>\backslash\)n3\(\backslash\)n\(<R3>\):]

CIFERS Responses.

 

\({}^{-1}\)

\({}^{-1}\)