Customized GPT Evaluation for Social Media Poll Generation and User Engagement
Authors: ZEXIN LU, Dai Zehua, Xiaoyu DONG, Yujie Feng, Bo LIU, Jing Li, Xiao-Ming Wu


Reviewer paper summaries: The authors propose a new evaluation strategy/metric (based on LLMs) for LLM-generated texts for the domain of social media polls (for the Chinese language, I believe). They measure the performance of a set of models on two tasks (1. generate poll only, 2. generate poll + answer options) and contrast the scores of traditional n-gram-based metrics to their LLM-based metrics. They include a user-study, showing high agreement of human annotators with their LLM-based metrics.
Summary of strengths: - The study additionally relies on human annotators, which is in my humble opinion crucial when dealing with such a subjective task
- Since traditional n-gram-based metrics are severely flawed for open-ended generation tasks, searching for new/better metrics is important.
Summary of weaknesses: - The whole experimental setup remains very opaque. Neither do the authors include their code in the submission, nor do they state anywhere that they intend to do so. Exact model versions are partially unclear (“latest version of ChatGPT-4”), and details/descriptives of the data set are not included.
- In the limitations they talk about “the randomness in the generation”: This can be controlled via the decoding strategy (which also remains unclear)
- It is imho to some extent unfair to compare Chinese Models to multilingual ones (if my assumption is correct, that all texts are fed to all models in chinese)
Comments, suggestions and typos: - In which language are the questions actually fed to the models? The paper never really states this explicitly. The figure suggest usage of the Chinese language
- On which data were the SFT models fine-tuned? The train set? In l. 371 it just says “with Human annotated data”

Typos/Grammar/etc.
l. 011: “n-gram”
l. 163 “MS MARCO”
l.303/304/316: missing space beefore citations
l. 308: acronym already introduced
l. 378/379/513: use the acronym (LLM)
Overall assessment scores: 2.5