Unconditionally Probable: A Multi-Class Classification Framework for Prompt Selection without Ground Truth Labels
Authors: Thomas McCullough, Leslie Delval, Ram Dantu


Reviewer paper summaries: This paper presents a method to optimize prompt selection for LLMs in multi-class classification tasks without relying on labeled data. The authors use mutual information (between prompt and label) to determine how well a prompt can predict class labels. This approach helps identify the most informative prompts. Additionally, they apply techniques to minimize dependencies between labels, ensuring that the selected prompts are effective across various categories. Clustering techniques are also employed to group similar prompts and select the best ones from these groups.

The method was tested on a dataset of 80 college assessments, each with 25 binary labels representing different knowledge, skills, abilities, and tasks (KSATs). The experimental results indicate that reducing label dependencies enhances the correlation between mutual information and model accuracy, thereby improving the performance of LLMs in multi-class classification tasks. This approach is particularly useful in scenarios where labeled data is scarce or unavailable.

The paper's contributions include a novel framework for prompt optimization and a new dataset to support further research. This method makes LLMs more practical and effective in real-world applications, enhancing their accuracy and utility without requiring extensive labeled data.
Summary of strengths: 1. The authors introduce a novel and practical methodology for optimizing prompt selection in multi-class classification tasks without the need for ground truth labels, by leveraging mutual information and dependency minimization techniques.

2. The empirical results are insightful and demonstrate the effectiveness of the proposed method. Using a novel dataset comprising 80 college assessments with 25 binary labels, each showcases the practical application and scalability of the framework. This dataset and the proposed methodology offer a valuable resource for future research and development.

3. The paper is well-organized and clearly presents related literature, methodological details, and empirical validation.

The framework's ability to enhance the accuracy of LLMs in scenarios where labeled data is scarce or unavailable makes it highly relevant and useful. This practical applicability, combined with the empirical contributions, positions the paper as a good addition to the field.
Summary of weaknesses: 1. The study focuses primarily on a specific dataset of 80 college assessments. While the results are promising, broader validation on more diverse datasets and domains is needed to confirm the generalizability and robustness of the proposed framework.

2. The paper does not thoroughly explore the practical implications and potential real-world applications of the proposed framework. A more detailed discussion on how this method can be integrated into existing NLP systems and its impact on real-world tasks would enhance its relevance and applicability.

3. Some sections of the paper, particularly the methodology, could benefit from clearer explanations and more accessible language. Providing step-by-step instructions or pseudocode for implementing the proposed framework would make the approach more accessible to a broader audience.

4. The framework relies on certain assumptions and simplifications, such as the conditional independence of labels, which may not always hold in real-world scenarios. Discussing these assumptions' limitations and potential impacts would provide a more balanced view of the framework's applicability.
Comments, suggestions and typos: See Weaknesses.
Overall assessment scores: 3