## 4 Experiments
### Experimental Steps
#### 4.2.2 Our Extensions and Improvements  
Building upon the baseline framework of _LLM Agora_[32], we introduced several enhancements to improve the performance and robustness of the multi-agent debate process. These modifications address key limitations of the baseline by incorporating dynamic weighting, consistency-based evaluations, and adaptive dataset augmentation. Below, we outline our approach and emphasize the changes compared to the baseline.  
1. Batch-wise inference with dynamic weighting Unlike the baseline, which processes questions uniformly, we divide the dataset into batches of 50 questions each, inspired by techniques in adaptive learning frameworks. For each batch: * Error log: During the inference stage, each model logs its errors based on comparisons with the ground truth. * Dynamic weight adjustment: After processing a batch, the error rate \(\epsilon\) of each model is calculated. To incorporate this information into subsequent debates, we compute the model's weight for the next batch using the following formula: \[w=\ln\left(\frac{1-\epsilon}{\epsilon}\right),\] (11) a weighting mechanism inspired by boosting techniques [33].
2. Weighted contributions in the debate process The baseline framework does not account for individual model reliability in the debate. In our approach: * Each model communicates its cumulative historical error rate before presenting its arguments, enabling transparency and accountability. * During the debate, model contributions are weighted using the dynamic weights \(w\), ensuring that more reliable models have a greater influence on the final consensus.
3. Consistency-based evaluation and adaptive dataset augmentation To address potential inconsistencies in model outputs, we introduce a post-debate evaluation step, inspired by methods in ensemble learning [34]. After the debate: * Consistency calculation: We calculate the consistency \(C\) of the models' outputs using the formula: \[C=\frac{\sum_{i<j}w_{i}\cdot w_{j}\cdot\delta(o_{i},o_{j})}{\sum_{i<j}w_{i} \cdot w_{j}},\] (12) where \(w_{i}\) and \(w_{j}\) are the weights of models \(i\) and \(j\), and \(\delta(o_{i},o_{j})\) is an indicator function equal to 1 if \(o_{i}=o_{j}\), and 0 otherwise. * Adaptive dataset augmentation: If \(C\) falls below a predefined threshold, we utilize a summarization model [35] to retrieve relevant information from the dataset. The model generates refined responses by incorporating additional context from the top 1000 questions.
4. Visualization of trends To evaluate the effectiveness of our modifications, we analyzed the trends in model weights, accuracy, and consistency across batches, specifically the following: * Model weight dynamics: We plot the evolution of model weights across batches, demonstrating the impact of error-based adjustments on model contributions.  
* Batch accuracy trends: We analyze accuracy improvements across batches, showcasing the cumulative benefits of dynamic weighting and consistency-based evaluation.
* Consistency trends: We track consistency values to understand how adaptive dataset augmentation addresses disagreement among models.  
Comparison to baseline  
Our approach introduces several key changes to the baseline:  
* Dynamic weighting: While the baseline treats all models equally, we dynamically adjust weights based on batch-wise error rates [33].
* Consistency metrics: Unlike the baseline, we calculate and utilize consistency to identify disagreements and guide adaptive augmentation.
* Dataset augmentation: The baseline relies solely on pre-existing responses, whereas we actively retrieve and integrate additional context from the dataset, inspired by recent work on dynamic retrieval [36].  
These changes make the debate process more adaptive and performance-driven, improving the overall accuracy and consistency of model outputs.