## 3 The Proposed Framework
### Agent Debate for Evaluating  
In traditional couplet generation tasks, the goal is merely to generate a subsequent clause based on a given antecedent clause. So, they can use the existing subsequent clauses as ground truth in the dataset to calculate metrics like BLEU and perplexity scores. Additionally, human evaluation is often employed, assessing the syntactic and semantic alignment of antecedent and subsequent clauses based on their structural patterns and meanings, respectively. However, in the task of PCcG, no ground truth exists, since it is an open question. Human evaluation is prone to subjective bias and is difficult to implement effectively.  
Inspired by the Multi-Agent Debate (MAD) framework [41] that leverages debate, a fundamental characteristic of human problem solving, to encourage divergent thinking in LLMs, in this study, we extend MAD and use external evaluation rules to analyze and evaluate the couplelets. This approach not only meets the creativity demands of this task but also avoids the biases inherent in human evaluation, offering a more objective and valuable assessment.  
Our debate process can be represented by Figure 4. First, we set multiple LLMs as debater agents and one LLM as a judge agent to form a committee. Second, we set the rules depending on the evaluation goal. If we want to obtain a more valuable debate and analysis process, we will set the rules as \"Comparing two couplelets, tit for tat, objective, and give reasons\". Third, deciding which couplet is better is relative. We provide two couplelets. Then, different aspects of them will be evaluated by a specific debater through a comparative method, and the judge will consider all the debater's arguments and conclude.  
Specifically, in the debate process, all debaters participate in the discussion of a specific aspect of the couplelets. The LLM assigned to that aspect initiates the topic, after which all LLMs take turns to present their viewpoints. Finally, the initiating LLM is responsible for summarizing the strengths and weaknesses of the couplelets concerning that specific aspect.  
Figure 4: The process of agent debate for evaluating. The arrow and the highlighted parts need to be filled with the content described in its text according to the specific task.  
It is important to note that the summary is comparative and limited to the assigned aspect only. Once all aspect-specific summaries are completed, the role of the judge is to integrate the information provided by all debaters and generate a final overall assessment. The more intense and comprehensive the debate, the more informative the process becomes, thereby offering stronger support for the judge's final decision. This multi-perspective debate and summarization framework helps to enhance both the accuracy and consistency of the final evaluation. The prompt for debaters is as follows:  
_Please compare and analyze the advantages and disadvantages of the following two couplets in terms of <aspect>: <couplet1>, <couplet2>, and give reasons._  
where the <aspect> is the evaluation direction that the LLM is responsible for. The prompt for the judge is as follow:  
<debaters' output> Please synthesize the above opinions and give a conclusion indicating which couple is better.  
where the <debaters' output> is the output of all debaters.