Entropy Variation and Information Competence: Enhancing Predictive Accuracy of Collaborative Language Models
Authors: Edward Chang


Reviewer paper summaries: The paper is about a framework that sets up two LLMs in a debate-style dialogue to exchange opinions and through information-theoretic metrics reach a middle-ground solution that both agree upon. The authors speculate that the middle-ground solution is of higher quality and it is proven experimentally that this setup improves the accuracy of the ensemble model compared to any of the two models alone. Their framework is based on three tools: in-context learning, dual entropy setup and information-theoretic metrics. In-context learning is used to ask the LLM to provide justifications for its answers, keep an adversarial stance against the opposing LLM and to configure the entropy level of the LLM's response (certainty-level). They introduce what they call a Dual Entropy setup, where one model has a higher information entropy value than the other, which they configure using the hyperparameters: temprature, top-k and their own introduced hyperparameter "contentiousness". They prove that with the Dual Entropy setup the combined model maximizes the lower bound of the combined entropy. They use several information-theoretic metrics, which they explain thoroughly, to monitor and especially terminate the dialogue when they decide that agreement has been reached.
Summary of strengths: 1. Information-theoretic Metrics.
The information theoretic metrics (CE, Entropy, JS, KL, MI, WD) as presented in Table 1. could be very useful in LLM-dialogue moderation and specifically the detection of agreement between parties as was used in this work.
2. Dual Entropy setup.
If accompanied with empirical proof, then this idea could actually be very insightful and inspire future work either in LLM-dialogue systems or LLM architecture design in general.
Summary of weaknesses: 1. In-context Learning: In-context learning is not the same as conditional statistics and they should not be used interchangeably.
2. Dual Entropy: The so-called proof that the combined model has better accuracy is a proof that the combined model maximizes the lower bound of the combined entropy. Experiments that test the hypothesis that lower entropy results in higher accuracy should be added.
3. Some ideas presented are unclear and it is unclear how they are combined together (examples include: CRIT, ARA, contentiousness).
4. No ablation study to validate empirically the hypotheses in the paper.
Comments, suggestions and typos: In general, the language used in the text is acceptable. Unfortunately, some crucial elements of the text are unclear, have been omitted entirely, or are only addressed in the Appendix. In addition, all of the graphics presented have been distorted and the labels are unclear.
More specifically the following need more explanation:
1. What is "Contentiousness" exactly and how is it applied in the prompt? 
Contentiousness is mentioned that is a number between 0 and 1 and it is applied in the prompt given to the LLM to configure the amount of entropy of the LLM's response. The LLM is assigned to reply with contentiousness Delta=0.9 and that's the prompt? It is neither clearly defined in the text nor in the citation.
2. How are CRIT and ARA algorithms used in the presented setup?
3. How are the outcomes of the algorithm explainable?
4. The probabilities that the models generate in their output (for each potential disease) are not generated by a regression module that has access to the LLM's hidden states as usually should be done to get proper probabilities, they are rather arbitrary next-word-predictions of the LLM.

There is a duplicate figure (figure 1.a. and figure 1.b.)

###############
Comments after Author Discussion
The author discussion helped clarify some details concerning algorithms taken from previous work. The author also promised to create an ablation study soon. The review does not change until the ablation is complete.
Overall assessment scores: 2.5