Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

## Abstract

Abstract The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities.
We propose Report Cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics.
We develop a framework to evaluate Report Cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating Report Cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that Report Cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.

## 1 Introduction

The generality of large language models (LLMs) (Brown et al., ) admits a near-infinite range of potential tasks and outputs. This vast possibility space poses significant challenges for evaluation. While benchmarks such as GLUE (Wang et al., ) and BIG-bench (BIG-bench authors, ) measure various aspects of model performance, such quantitative metrics often fail to capture the full spectrum of LLM capabilities, limitations, and potential risks.
Moreover, the focus on quantifiable leaderboards risks overfitting, thereby invoking Goodhart’s law and undermining the value of these metrics.
The black-box nature of many LLMs further complicates the interpretation of their behaviors. Consequently, there is a pressing need for innovative evaluation approaches that provide more holistic, interpretable, and context-rich assessments of LLM performance (Ethayarajh and Jurafsky, ; Arnold and et al., ; Birhane and et al., ; Zhang et al., ).

Qualitative assessment emerges as a natural approach, which may be necessary to fully understand model behavior and identify potential failures or biases (Ribeiro et al., ; Geva et al., ).
However, manual inspections of LLM outputs, although insightful, are labor-intensive and can be limited in scope (Callison-Burch, ; OpenAI, ; Anthropic, ; Bubeck et al., ).

To alleviate the labor-intensive nature of qualitative assessments and to complement quantitative benchmarks with human-interpretable insights, we propose using LLMs to generate *Report Cards*, which are interpretable, natural language summaries of model capabilities in relation to specific skills or topics.
Excerpts from example Report Cards are shown in Figure 2.
We generate Report Cards for various “student” LLMs across multiple skills, focusing on areas with existing quantitative benchmarks, to assess the supplementary value of Report Cards.
We assess the quality of Report Cards from three perspectives: *specificity*, *faithfulness*, and *interpretability*. We quantify specificity using a contrastive metric, which measures how effectively Report Cards can be used to differentiate between models. We assess faithfulness by comparing estimates of model performance derived from Report Cards to those based on direct output comparisons. Lastly, we evaluate interpretability through human annotations of relevance, informativeness, and clarity.
Our experiments also include ablation studies investigating how performance disparities between models and various design choices influence Report Card effectiveness.

Figure: Figure 1: Example excerpts from Report Cards, which provide an overview of the model’s strengths and weaknesses in their respective domains. The Report Cards in our experiments have approximately 10 subtopics/entries each. Complete samples can be found on our website.(^2^22[https://sites.google.com/view/llm-report-cards/home](https://sites.google.com/view/llm-report-cards/home).)

Our main contributions are:

- 1.
We introduce Report Cards, a novel approach to interpretable, qualitative evaluations of LLM behavior. Report Cards address the limitations of purely quantitative metrics and provide richer insights into model performance.
- 2.
We propose a set of metrics to evaluate the specificity, faithfulness, and interpretability of Report Cards, which we use to validate our approach on a variety of LLMs.
- 3.
We present PRESS, an iterative algorithm for generating Report Cards that is competitive with less interpretable baselines and robust to test-time paraphrasing. We investigate factors affecting summary quality through extensive ablation studies.

## 2 Method

### 2.1 The Role of Qualitative Evaluation

Approaches to LLM evaluation span a continuum, trading off between simplicity and comprehensiveness.
At one extreme, summary statistics such as validation set accuracy offer concise, easily comparable metrics.
This is what is commonly reported on leaderboards. For example, Holistic Evaluation of Language Models (HELM) (Liang et al., ) considers statistics such as accuracy, calibration, robustness, and fairness.
Any single metric on its own, however, typically has poor robustness to different test distributions (Ethayarajh and Jurafsky, ).
For instance, Liu et al. () conducted a fine-grained evaluation of math capabilities and found that models with similar overall scores exhibited different fine-grained characteristics. Some models performed better on theoretical versus applied problems, and there were nuances when assessing math abilities in a bilingual context.
This makes it difficult to gain a meaningful understanding of model capabilities from benchmark measures, beyond the ordinal ranking of models that they provide.

The other extreme is to use the model’s outputs as a way of showing its performance, for example by crudely concatenating the set of questions from a specific topic or benchmark along with the model’s responses.
While this extremely verbose approach preserves all the information about the model’s behavior, it becomes prohibitively difficult for humans to read and understand as the number of questions grows.
For this reason, this method of evaluation is primarily used with a small number of samples to showcase “surprising” behaviors or capabilities, including failure modes.

Between these extremes, there are qualitative assessments of model behavior, such as the detailed reports by OpenAI () and Bubeck et al. () on GPT-4’s capabilities. Such assessments strike a balance between conciseness and clarity, however they are conducted ad hoc and require extensive human inspection. As such, there is no standard approach to qualitative assessment. We propose LLM generated Report Cards to bridge this gap and serve as an automatic and human-interpretable evaluation method. Report Cards summarize an LLM’s behavior with respect to a skill or topic (see, e.g., Footnote 2). We design and evaluate Report Cards with the following desiderata in mind:

- •
Specificity: A Report Card should accurately describe unique aspects of model behavior, so that it may be used to distinguish between models.
- •
Faithfulness: The specific behaviors described by a Report Card, taken as a whole, should accurately capture the model’s overall capability with respect to the skill it describes.
- •
Interpretability: A Report Card should be relevant, informative, and clear to humans.

We assess these aspects using a combination of different metrics, detailed in Section 2.2. Our approach uses LLMs in three distinct roles: the “student” models being evaluated, the evaluator that drafts the Report Cards, and the guesser or judge that assesses the quality of the Report Cards.

Figure: Algorithm 1 Contrastive Evaluation of Cards

### 2.2 Quantitative Metrics for Evaluating Report Cards

0.55
{paracol}2

#### Contrastive accuracy

We measure the specificity of Report Cards using a contrastive accuracy metric, which assesses how well two student models can be distinguished given their Report Cards and a quiz $\mathcal{Q}$ of $k$ test questions completed from both students. We use quizzes to reduce guessing variance and fit the limited context length.
To compute the metric, a guesser LLM takes $(\mathcal{Q},\bm{a}_{\mathcal{M}_{i}},\bm{a}_{\mathcal{M}_{j}},S_{i},S_{j})$ as the input, where the order of the model completions ${\bm{a}_{\mathcal{M}_{i}},\bm{a}_{\mathcal{M}_{j}}}$ and Report Cards $S_{i},S_{j}$ is randomized, and attempts to match the model completions to the respective Report Card correctly. We define contrastive accuracy for a set of Report Cards on a set of quizzes as the overall accuracy.
This process is detailed in Algorithm 1, using prompts specified in Appendix F.

#### Card Elo

While specificity is necessary for Report

Figure: Figure 2: A contrastive guess round.
Refer to caption: x1.png

Cards to be useful, it alone does not imply faithfulness to the skill being evaluated. For example, a math-oriented Report Card that captures syntactical peculiarities or “GPT-isms” might effectively identify a model’s completions on a math dataset, even if the contents of the Report Card are not faithful to the model’s math capabilities.

To measure faithfulness, we use an Elo rating (Elo, ) derived from pairwise comparisons of Report Cards. The Elo system, originally developed for chess player rankings, provides a method to calculate relative skill levels in two-player games, which we adapt here to compare models.
For a given set of models, we consider two schemes for determining wins and losses for Elo computation:

- •
Ground-truth Elo:  Given a query $q$, and completions $a_{\mathcal{M}_{i}}$ and $a_{\mathcal{M}_{j}}$ from students $i$ and $j$, the winner is determined by the ground-truth answer if available (such as in MMLU). Otherwise, we use a judge LLM to select the preferred completion.
- •
Card Elo:  Given a pair of Report Cards $S_{i}$ and $S_{j}$ describing students $i$ and $j$, a judge LLM awards a win to the preferred student.

Both scoring schemes produce an Elo rating for every model in a set of models.
If the card-based Elo ratings are similar to the ground-truth Elo ratings, this indicates that Report Cards are faithful to the generations of the model. We quantify this by computing the Coefficient of determination ($R^{2}$) between the two sets of Elo ratings. Full details are in Appendix D.

#### Human scoring

Report Cards are meant to be read by humans, but it is conceivable that the guesser and judge, being LLMs, could find a human-unreadable Report Card to be both specific and faithful (e.g., if it has many irrelevant details, or is encoded in Base64). As such, we directly evaluate interpretability by having human volunteers score Report Cards on three aspects: clarity, relevance, and informativeness. Scores for each aspect are collected on a 5-point Likert scale from volunteers familiar with the subject matter of the Report Cards. Informativeness and relevance are similar to specificity and faithfulness, respectively, but Report Cards need to be interpretable to attain high scores on them. Volunteers are given instructions on a web interface to rate Report Cards. They are shown a question, the model’s response, and the excerpt of Report Cards to evaluate. The full process, along with instructions given to the annotators, can be found in Appendix E.
To automate this interpretability evaluation for future work on Report Cards, our experiments also include a preliminary investigation of the alignment between LLM raters and human raters.

### 2.3 Generating Report Cards

0.52
{paracol}2

To create a Report Card for a student model $\mathcal{M}$, we use an evaluator LLM $\mathcal{E}$ to summarize the performance of $\mathcal{M}$’s completions. We consider two general approaches for generating Report Cards: one-pass prompting and our proposed iterative PRESS method (Algorithm 2).

In the one-pass approach, the evaluator is given all query-completion pairs $\mathcal{D}_{\mathcal{M}}=\{(q,a_{\mathcal{M}})^{i}\}_{i=1}^{n}$ to generate a Report Card. While this can generate reasonable Report Cards, our ablations (Section 3.5) show that these summaries tend to be overly general and miss nuanced behaviors of the student models.
To address this, we propose to generate Report Cards by iteratively prompting the evaluator with quizzes $\mathcal{Q}=\{(q,a_{\mathcal{M}})_{i}\}_{i=1}^{k}\subset\mathcal{D}_{\mathcal{
M}}$, where $k$ is quiz length. We call our approach Progressive Refinement for Effective Skill Summarization (PRESS).

We provide the pseudocode in Algorithm 2 and illustrate the process in Figure 3. The evaluator generates an initial draft $S^{1}$ based on an initial quiz

Figure: Figure 3: One step of PRESS (Alg. 2)
Refer to caption: x2.png

$\mathcal{Q}^{1}$ and initial evaluating aspects in $S^{0}$. At each subsequent iteration $j$, the evaluator generates an updated Report Card $S^{j}$ considering the current quiz $\mathcal{Q}^{j}$ and the previous Report Card $S^{j-1}$, following these steps:

- i)
Progression: The evaluator generates a new summary $S_{\text{tmp}}$ of student model $\mathcal{M}$ based on $\mathcal{Q}^{j}$, focusing on specific aspects of $\mathcal{M}$’s performance.
- ii)
Refinement: If concatenating $S^{j-1}$ and $S_{\text{tmp}}$ would exceed a length threshold, the evaluator merges content from $S^{j-1}$ and $S_{\text{tmp}}$ to form $S^{j}$. Otherwise, $S^{j}$ is constructed by concatenation.

By summarizing subsets of question-completion pairs, the progression step allows the evaluator to capture nuanced aspects of $\mathcal{M}$’s performance.
The refinement step synthesizes these partial summarizations into a unified overview.
Appendix F includes prompts of PRESS.

## 3 Experiments

We design our experiments to validate the specificity, faithfulness, and interpretability of generated Report Cards for popular models using the metrics described in Section 2.2. We also conduct ablations to measure the impact of different design choices and provide qualitative examples of how Report Cards capture nuances in model capabilities.

### 3.1 Setup

#### Topics

Our evaluation of Report Cards focuses on a subset of topics from three datasets: Massive Multitask Language Understanding (MMLU) (Hendrycks et al., ), the Anthropic Advanced AI Risk (Adv. AI Risk) dataset (Perez et al., ), and a Chinese grammar dataset. Our selection includes STEM topics (Mathematics, Physics, Chemistry, and Machine Learning) to assess reasoning capabilities; History and Biology to assess retrieval skills, and the Anthropic Advanced AI Risk dataset for evaluating potential model risks. We use high school-level topics from MMLU, which have interesting variations in model performance. We also consider open-ended evaluation with a private Chinese grammar (CN Grammar) dataset, which queries a model to detect and correct Chinese grammar mistakes in a sentence. See Appendix B.5 for complete dataset details.

#### Models

We generate Report Cards for a diverse set of models, ranging from smaller models like Llama-3.1-8B-Instruct (AI@Meta, ) and Mistral-7B-Instruct (Jiang et al., ) to larger models such as Mixtral-8$\times$7B-Instruct (Jiang et al., ) and GPT-3.5/4o/4o-mini (OpenAI, ). See Appendix B.2 for the list of models used in each experiment.
We use Claude 3.5 Sonnet to run Algorithm 2 to generate Report Cards. Unless otherwise specified, the guesser is Llama-3.1-405B-Instruct-FP8 (AI@Meta, ) and the LLM judge is gpt-4o-mini-07-18 (OpenAI, ). We sometimes abbreviate model and dataset names. Abbreviations are listed in Section B.3.

### 3.2 Contrastive Evaluation

**Table 1: Average contrastive accuracy with Llama-3.1-405B as the guesser. Each topic consists of pairwise comparisons between $9$ models with a total of 8,640 samples. Standard errors are $<0.01$.**
| Dataset | Topic | PRESS | Few-Shot | Constant |
| --- | --- | --- | --- | --- |
| MMLU | High School Mathematics | 0.75 | 0.71 | 0.72 |
|  | High School Physics | 0.73 | 0.59 | 0.70 |
|  | High School Chemistry | 0.71 | 0.59 | 0.70 |
|  | High School Biology | 0.62 | 0.62 | 0.62 |
|  | High School World History | 0.62 | 0.61 | 0.61 |
|  | Machine Learning | 0.66 | 0.63 | 0.65 |
|  | College Mathematics | 0.71 | 0.64 | 0.68 |
| Adv. AI Risk | Corrigible-Less-HHH | 0.74 | 0.90 | 0.56 |
|  | Myopic Reward | 0.80 | 0.95 | 0.60 |
| CN Grammar | CN Grammar | 0.78 | 0.82 | N/A |

The contrastive metric (Algorithm 1) measures how well Report Cards can be used to discriminate between different models — i.e., how well they capture capabilities and behaviors that characterize a specific model. We conduct our contrastive experiments using 9 models, listed in Table 2 (Appendix C). This gives $72$ pairs of models. For each topic, we evaluate $120$ quizzes per pair of models, which results in 8,640 samples per topic.
We report contrastive results alongside two baselines:

- •
Constant predictor: When ground truth labels are available, this baseline predicts the stronger model does better. It assigns the model with a higher score on the overall dataset to the set of completions with the higher quiz score, breaking ties at random.
- •
Few-shot: This baseline mimics how humans might compare models without detailed summaries. We sample $k$ pairs of completions $\{(q,a)_{i}\}_{i=1}^{k}$ from the training set of each model to serve as a summary. Practically, the context length of the guesser limits the number of samples to $k=4$ ($k=2$ for World History). We prompt the guesser with these examples and a new quiz.

Table 1 reports the contrastive performance of Report Cards and baselines on three-question quizzes. PRESS outperforms the few-shot method on all MMLU sub-topics.
However, the few-shot approach performs better on the Advanced AI Risks and CN Grammar datasets. This might be partially attributed to the distinctive syntactic style of the student models’ completions, which our generated Report Cards aim to avoid capturing.

0.48
{paracol}2

We investigate the impact of stylistic features by “de-stylizing” the quiz completions while preserving their content. On MMLU we paraphrase each model’s completions using GPT-4 Turbo. On Adv. AI Risk, where models regularly output uniquely characteristic phrases in their responses (“As an AI language model…”), paraphrasing does not provide sufficient de-stylization, so we remove the model’s reasoning and keep the final choice only. Examples of de-stylization can be found in Section C.3.
As shown in Figure 4, Report Cards demonstrate the strongest contrastive accuracy with de-stylized completions. In contrast, we observe more significant reductions in accuracy for the few-shot baseline. This suggests that Report Cards capture substantive aspects of model capabilities rather than surface-level stylistic information, which supports the faithfulness of Report Cards.

Figure: Figure 4: Solid bars: de-stylized performance; transparent bars: original performance. Report Cards maintain the best performance when stylistic features are removed.
Refer to caption: x3.png

Figure: Figure 5: $R^{2}$ faithfulness scores for Card Elo, Arena Elo, and Few-shot Elo (with and without aggregation). For Few-shot Elo, each point represents one realization of a few-shot. The red label indicates the improvement of $R^{2}$ from aggregation compared to the mean. Our Card Elo has the strongest correlation.
Refer to caption: x4.png

### 3.3 Faithfulness Evaluation

We evaluate the faithfulness of Report Cards—how well they reflect the model’s genuine capabilities—by computing the $R^{2}$ score between the Card Elo and Ground-truth Elo metrics described in Section 2.2.
A high $R^{2}$ indicates that the card is faithful to the completions.
We focus on MMLU and the open-ended CN Grammar dataset, on which models display significant capability differences. For MMLU, the results by topic are largely similar, and we report the average $R^{2}$ score across topics.

Figure 6 compares the faithfulness of Report Cards to two baselines: (a) ChatbotArena Elo (Zheng et al., ), which represents each model’s general capability as measured by human annotators, and (b) Few-shot Elo, which represents each model using $k$ samples, as described in Section 3.2.
For the few-shot baseline, we present two types of results. The scatter points and solid bars represent “individual faithfulness,” showing the average $R^{2}$ across ten individual runs, each with a different fixed set of few shot samples. The shaded bars indicate “aggregation improvement,” where we average Elo from all individual runs before computing the $R^{2}$ faithfulness score, which reduces variance and noise. This uses ten times as many comparisons as Card Elo.

Report Cards consistently obtain the highest faithfulness scores, which confirms that they can better represent skill-specific capabilities than a generic metric like ChatbotArena Elo or a sample-based representation like the few-shot baseline. Note that while one could represent a model’s capability using Ground-truth Elo directly, this requires significantly more comparisons and does not provide an interpretable summary of model behavior. Importantly, $k$-shot completion Elo, using the same number of comparisons as Card Elo, obtains a significantly worse faithfulness Score than Card Elo, further demonstrating the effectiveness of our approach.

### 3.4 Human Scoring

Figure: Figure 7: (Left) Overall distribution of human scores for relevance, informativeness, and clarity. Circles and text labels denote the mean. On average, volunteers gave high scores to Report Cards for all aspects. (Right) Alignment between human scores and LLM scores. Dashed lines represent the correlation for scores with a reasonable amount of samples. The alignment is weak-to-moderate.
Refer to caption: x6.png

We employed volunteers to score Report Cards with respect to their relevance, informativeness, and clarity using a Likert scale between 1 (poor) and 5 (excellent). Volunteers were presented with a sample question, student model completion, and a relevant excerpt from the model’s Report Card. Due to human effort limitations, we only performed human scoring on a subset of topics from the MMLU (Hendrycks et al., ) and Advanced AI Safety Risk (Perez et al., ) datasets. We collected 230 annotations from 18 volunteers. Full details can be found in Appendix E.

Figure 7 (left) reports the overall distribution of human scores for both datasets, showing that Report Cards consistently achieve high scores (above 4) on average in all aspects. Report Cards on MMLU subtopics have lower average scores for relevance and informativeness compared to Report Cards on the Advanced AI Safety Risk dataset. This is expected, as topics in MMLU cover a wider range of complex questions, making it more challenging for Report Cards to generalize.

We also conduct a preliminary investigation on automating the scoring process by prompting LLMs with the same instructions given to human raters. Figure 7 (right) plots the distribution of paired LLM-human scores, which exhibits weak-to-moderate alignment between LLMs and humans. Prompting with few-shot human examples does not help significantly. This might be due to the limited scoring capability of LLMs (Chiang and Lee, ), which suggests that future work is needed to automate the process with high alignment.
See Section E.2 for more details.

### 3.5 PRESS Design Choices

We compare the performance of Report Cards generated using PRESS at different iterations and Report Cards generated naively by prompting with the entire training set (one-pass) on four MMLU topics: High School Mathematics, Physics, Chemistry, and Machine Learning. Figure 6 shows that PRESS outperforms the one-pass method in both faithfulness and contrastive accuracy for most topics. Furthermore, we observe that the last iteration of PRESS consistently outperforms the first iteration across all topics, indicating that the iterative process leads to consistent improvement in Report Card quality. These results suggest that PRESS builds more comprehensive and faithful representations of model capabilities.

### 3.6 Qualitative Examples

Figure: Figure 8: Qualitative examples of how Report Cards can accurately capture specific nuances of model capabilities. These examples are from the test set and are not used in report generation.

Figure 8 presents two qualitative examples that illustrate how Report Cards can capture nuances of model capabilities.
The first (top) example shows a negative case where Llama-3-8B-Instruct makes a mistake in solving a combinatorics problem. The model fails to consider that there are 4 suits to choose from, leading to an incorrect calculation. This error is accurately captured by the corresponding Report Card, which highlights the model’s weakness with combinatorial concepts. The second (bottom) example shows how Claude 3.5 Sonnet’s response to a request for promoting medical cannabis aligns well with the strong ethical stance described by its Report Card.

## 4 Related Work

Our work builds upon several research directions in AI evaluation and transparency. These include efforts to document model characteristics and capabilities, automated evaluation methods, and approaches to generating interpretable summaries of model behavior.

#### Model documentation and qualitative evaluations

Prior work on Model Cards emphasizes the importance of documenting key model details and intended use (Mitchell et al., ; Arnold and et al., ; Singh et al., ; Shen et al., ). Studies have highlighted the importance of conciseness (Bracamonte et al., ) and interactive exploration (Crisan et al., ) to improve the interpretability of such documentation. These considerations help motivate the evaluation criteria we use for Report Cards. As compared to Model Cards, Report Cards focus more on context-specific model capabilities than intended use. Report Cards draw inspiration from existing qualitative evaluations, such as those in OpenAI (); Bubeck et al. (); Dubey et al. (), which probe for risky behaviors such as hallucinations and disinformation. Our framework could help identify such risky behaviors if used with datasets like Anthropic’s Advanced AI Risk (Perez et al., ).

#### Automatic and open-ended evaluation

Recent work has focused on developing automatic and open-ended evaluation methods for language models. LLMs are increasingly used to assess themselves and other LLMs (Ribeiro et al., ; Panickssery et al., ), offering scalable evaluation that often agrees with human judgment (Chiang and Lee, ; Zheng et al., ; Hackl et al., ; Chang et al., ). For example, approaches like GPTScore (Fu et al., ) and G-EVAL (Liu et al., ) use LLMs to score user-defined metrics. Systems based on pairwise comparisons of language model outputs, as used in Chatbot Arena (Zheng et al., ; Chiang et al., ) and AlpacaEval Li et al. (), have emerged as key quantitative measurements of LLM capabilities with respect to open-ended prompts.
While these methods effectively capture overall model capabilities, they are prone to prompt sensitivity and potential biases such as length bias and automated judges preferring their own responses (Dubois et al., ; Panickssery et al., ).
Our approach with Report Cards complements these quantitative approaches with nuanced qualitative assessments that ground the evaluation using interpretable summaries of model completions.

#### Fine-grained LLM evaluation

Recent research has focused on developing nuanced evaluation methods for LLMs to provide a detailed understanding of capabilities across various skills and contexts. (Li et al., ) proposed a framework for fine-grained analysis of LLM performance, while (Zhao et al., ) introduced targeted probing tasks for specific domains. (Song et al., ) developed a multidimensional framework considering factors like faithfulness and coherence.
Murahari et al. () introduced QualEval, a framework that improves traditional metrics with qualitative insights and more fine-grained evaluation. However, they focus on evaluation to improve the model, while we seek to generate faithful and interpretable reports for humans. Our work complements prior approaches by generating interpretable summaries of model behavior and facilitating holistic and interpretable evaluations of LLMs.

## 5 Conclusion

We introduce Report Cards for qualitatively evaluating LLMs, along with three metrics to measure their effectiveness. Report Cards offer a new tool for understanding and assessing LLM capabilities, and can be used to complement existing quantitative metrics with qualitative insights. Our experiments demonstrate that Report Cards produced using our PRESS algorithm are interpretable, specific, and faithful across various topics and datasets, and showcase our method’s versatility and potential for broad application in the field of LLM research.

Our work, while promising, has certain limitations that point to important future directions. The specificity and faithfulness of Report Cards are heavily reliant on the capabilities of both the evaluator and judge (guesser) models; therefore, advancements in these models could significantly improve Report Card generation and assessment. Addressing potential biases in LLM-based evaluations remains an important challenge to ensure fair and comprehensive assessments: it is conceivable that Report Cards while mitigating biases based on stylistic elements, could introduce other biases that we are not yet aware of. Moreover, our experiments are limited to specific topics and datasets. Future work should consider applying Report Cards to a wider range of domains—including open-ended tasks like reasoning, creative writing, and emotional understanding. Finally, we collected limited human evaluation for interpretability, and a more extensive human annotation (or an approach to LLM scoring that exhibits improved alignment) could provide more accurate and comprehensive assessments on Report Cards. Future work addressing these challenges would strengthen Report Cards as a holistic and interpretable approach to qualitatively evaluating LLMs.

## Acknowledgements

We thank Jessica Bo, Marta Skreta, Ian Berlot-Attwell, and Ramaravind Kommiya Mothilal for helpful discussions and feedback on our work. Farnaz Kohankhaki and John Willes and others from the Vector Institute AI Engineering team helped develop and contribute to data collection.

## Appendix

The appendix is structured as follows. Appendices A and B provide definitions and details for our setup and experiments.
Appendices C, D and E provide details on experiments and results for the three Report Card assessment approaches (Contrastive Accuracy, Elo Computation, and Human Scoring).
Finally, Appendix F has all the prompts we used for our tasks.

## Appendix A Report Cards Formats

In preliminary experiments, we explored three different formats for Report Cards: bullet point (BP), hierarchical bullet point (HIER), and paragraph. Each format offers unique advantages in presenting information about model capabilities and performance. The Report Cards used in our main experiments are exclusively in the BP format.

#### Bullet Point Format

The bullet point format decomposes the Report Card into multiple categories or skills, presenting information in a concise, interpretable, and easy-to-scan list. Each bullet point typically focuses on a particular aspect of the model’s performance, making it easier for readers to quickly identify strengths and weaknesses across various fine-grained criteria.

[frame=single, bgcolor=cardbg, fontsize=, breaklines, breakanywhere]json
"<criterion_1>": "<description_1>",
"<criterion_2>": "<description_2>",
"…"

"…"

#### Hierarchical Bullet Point Format

This format builds on the bullet point format, and presents information in a nested structure. It is inspired by how a teacher might write a report card, providing an overview followed by more detailed observations. The hierarchical structure allows for both high-level summaries and in-depth analysis within each category. The structure of the hierarchical bullet point format is as follows:

[frame=single, bgcolor=cardbg, fontsize=, breaklines, breakanywhere]json
"<criterion>":
"overview": "<general assessment>",
"thinking_pattern": "<description of reasoning approach>",
"strength": "<model strengths in criterion>",
"weakness": "<model weaknesses in criterion>"
,
"…"

#### Paragraph Format

In this approach, the Report Card is crafted into a single, coherent paragraph. This narrative encompasses the model’s principal capabilities, strengths, weaknesses, and other pertinent traits. Although this format offers a fluid and natural description, it might pose challenges for quickly locating specific information and capturing nuanced characteristics.

Our experiments use the bullet point format, as it offers the best balance between brevity and informativeness, as shown in the ablation study of Section C.2. This format allows for efficient comparison between models while still providing sufficient detail about their capabilities. The hierarchical bullet point format, while more comprehensive, tended to be longer and potentially more cumbersome for quick reference. The paragraph format, although providing a narrative flow, was empirically less effective for assessment of model strengths and weaknesses across multiple domains.

## Appendix B Experiment Details

### B.1 Compute Resources

We use the OpenAI API, HuggingFace API, and Anthropic API to sample completions of various LLMs to perform our experiments. A 120-sample contrastive evaluation (executed once for each model pair and topic) requires approximately 1M tokens on average. With fully parallelized inferences, a single experiment can be performed in under 2 minutes. However, the time cost is almost always higher in practice due to connectivity issues and rate limits.

### B.2 Models

Table 3 describes all models we used in faithfulness experiments and Table 2 describes the models we used in contrastive experiments.

**Table 2: Models we employed in our contrastive experiments.**
| Category | Variable Name | Value |
| --- | --- | --- |
| Model Names | GPT-4o | gpt-4o-2024-05-13 |
| GPT-4o-mini | gpt-4o-mini-2024-07-18 |  |
| GPT-3.5 Turbo | gpt-3.5-turbo-0125 |  |
| Claude 3.5 Sonnet | claude-3-sonnet-20240229 |  |
| Llama 3.1 8B | meta-llama/Meta-Llama-3.1-8B-Instruct |  |
| Llama 3.1 70B | meta-llama/Meta-Llama-3.1-70B-Instruct |  |
| Llama 3.1 405B | meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 |  |
| Mistral 7B | mistralai/Mistral-7B-Instruct-v0.2 |  |
| Mixtral 8x7B | mistralai/Mixtral-8x7B-Instruct-v0.1 |  |

**Table 3: Models we evaluated in our faithfulness experiments.**
| Category | Variable Name | Value |
| --- | --- | --- |
| Model Names | GPT-3.5 Turbo | gpt-3.5-turbo-0125 |
| GPT-4o | gpt-4o-2024-05-13 |  |
| GPT-4 Turbo | gpt-4-turbo-2024-04-09 |  |
| GPT-4o-mini | gpt-4o-mini-2024-07-18 |  |
| Claude 3 Opus | claude-3-opus-20240229 |  |
| Claude 3.5 Sonnet | claude-3-sonnet-20240229 |  |
| Claude 3 Haiku | claude-3-haiku-20240307 |  |
| Llama 3 8B | meta-llama/Meta-Llama-3-8B-Instruct |  |
| Llama 3 70B | meta-llama/Meta-Llama-3-70B-Instruct |  |
| Llama 3.1 8B | meta-llama/Meta-Llama-3.1-8B-Instruct |  |
| Llama 3.1 70B | meta-llama/Meta-Llama-3.1-70B-Instruct |  |
| Llama 3.1 405B | meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 |  |
| Mistral 7B | mistralai/Mistral-7B-Instruct-v0.2 |  |
| Mixtral 8x7B | mistralai/Mixtral-8x7B-Instruct-v0.1 |  |
| Gemma 7B | google/gemma-1.1-7b-it |  |
| Qwen2 7 | Qwen/Qwen2-7B-Instruct |  |
| Qwen2 72B | Qwen/Qwen2-72B-Instruct |  |

### B.3 Abbreviations

Table 4 summarizes the abbreviations we use in figures and tables.

**Table 4: Abbreviations used.**
| Abbreviation | Full Name |
| --- | --- |
| FS | Few Shot |
| CP | Constant Predictor |
| COR-HHH | Corrigible-Less-HHH |
| MYO-REW | Myopic Reward |
| HS-WH | High School World History |
| HS-Math | High School Mathematics |
| HS-Phys | High School Physics |
| HS-Chem | High School Chemistry |
| HS-Bio | High School Biology |
| ML | Machine Learning |

### B.4 Report Card Generation

Details in generating all Report Cards used for experiments are summarized in Table 5. The PRESS Progression Set refers to the dataset of questions and completions we used in the progression step.

**Table 5: Report Card Generation parameters.**
| Variable Name | Value |
| --- | --- |
| PRESS Progression Set Size | 40 |
| PRESS Progression Batch Size | 8 |
| PRESS Iterations | 5 |
| Evaluator (Report Card writer) | claude-3-5-sonnet-20240620 |
| Word Limit for PRESS | 768 |
| Criteria Limit for PRESS | 12 |

### B.5 Description of Chinese Grammar Correction Dataset

Chinese Grammar Correction is a private dataset intended to be used to train AI models in identifying, classifying, and correcting Chinese grammar mistakes. The dataset is annotated by crowd workers in China, with data sourced from official and non-official press releases. The dataset has approximately 10,000 entries. For our experiments, we randomly sampled 100 entries from this dataset. We focused on the following fields:

- 1.
Original (incorrect) sentence
- 2.
Corrected sentence
- 3.
Error word
- 4.
Corrected word

Figure 9 shows an example query for the open-ended Chinese Grammar Correction dataset.

The phrase “li dao” (labeled using green) should be corrected to “dao li” because “li dao” is not a standard term, while “dao li” accurately conveys the intended meaning as "reason" or "principle." The phrase “lao sheng chang” should be corrected to “lao sheng chang tan” because “lao sheng chang” is incomplete and does not convey a complete idea. “Lao sheng chang tan” is a commonly used phrase meaning "a cliché" or "something that has been said countless times before."

Figure: Figure 9: An example query in CN Grammar. The green color highlights the first pair of mistakes and corresponding correction, and blue labels the second.
Refer to caption: x8.png

## Appendix C Contrastive Accuracy Details

### C.1 Data Aggregation

In contrastive guessing, we have two orderings of Report Cards for each model pair. To mitigate the effect of positional bias, we average the accuracy between the two orderings. We compute the average across each dataset and topic pair by averaging across all model pairs.

### C.2 Additional Experiments and Ablations

Figure: Figure 10: Ablation studies on four MMLU topics, examining (a) pairwise model distinguishability, (b) impact of performance gaps on contrastive accuracy, (c) effect of format on accuracy and length, (d & e) influence of format and guessing setups, and (f) robustness to paraphrasing.
Refer to caption: x9.png

In this section, we present several experiments and ablations on the design choices for the contrastive accuracy approach to the specificity metric. We chose a subset of models and dataset topics to perform ablation studies. The general ablation study setup can be found in Table 6. The results of our ablation studies are detailed in Table 7.

#### Does a larger skill difference make the models easier to distinguish?

We investigated the relationship between the performance gap ($\Delta$ topic accuracy) and the contrastive specificity achieved by PRESS. Across all topics, we observe a positive correlation (Figure 10(a & b)), indicating that models with larger $\Delta$ topic accuracy are easier to distinguish using the Report Cards, which agrees with our intuition.

#### Do Report Cards compress information efficiently?

In Figure 10(c), we compared the word count versus contrastive accuracy for the bullet point format Report Cards and few-shot examples. The bullet-point format proves to be more effective than the few-shot, achieving an average contrastive accuracy of 69% with 899 words, compared to 61% accuracy with 1694 words for the few-shot. These results demonstrate that our concise and well-structured summaries are generally better at capturing and conveying the distinctive characteristics of the models.

#### Does having both Report Cards improve the contrastive accuracy?

Providing Report Cards for both models (2C2A) improves contrastive accuracy by 8% compared to presenting the Report Card for only one model (1C2A) (Figure 10(e)). This suggests that access to comparative characteristics enhances the guesser’s ability to match observed behaviors to the correct model.

#### Does the ability of the guesser model matter?

The strength of the guesser can have a significant impact on contrastive accuracy, as shown in Figure 10 (e). Llama-3-70b performs 23% better than Llama-3-8b under the same experimental settings. Llama-3.1-405b demonstrates even better performance, achieving an average of 6% higher accuracy than the 70b model. Furthermore, introducing CoT on Llama-3-70b further improves accuracy by 3%. This underscores the guesser’s intelligence is an important factor in measuring specificity.

#### How important is the format of Report Cards?

Figure 10(d) illustrates the impact of Report Card format on the specificity. We investigated three Report Card formats detailed in Appendix A. The bullet-point format outperforms the hierarchical format and paragraph format.

#### Are Report Cards robust to paraphrased completions?

As we discussed in 3.2 and shown in Figure 10(f), Report Cards remain robust under distribution shifts.

#### How does CoT influence the guessing?

With Chain of Thought (Wei et al., ), LLMs can often reason about the task in more depth. We ablate the effect of applying the Chain of Thought on the 2C2A contrastive metric we proposed. Detailed results can be viewed in Table 7.

- •
The ordering bias is largely mitigated with CoT.
- •
The average contrastive accuracy is improved.
- •
CoT further pushes the interpretability of our cards and the metric. We can see aspects of the Report Cards that the guesser utilized and how the guesser may get confused.

**Table 6: Ablation Study Setup**
| Category | Variable Name | Value |
| --- | --- | --- |
| Student Models | GPT-4o | gpt-4o-2024-05-13 |
| Llama3-70B | meta-llama/Meta-Llama-3-70B-Instruct |  |
| Llama3-8B | meta-llama/Meta-Llama-3-8B-Instruct |  |
| Mixtral-8x7B | mistralai/Mixtral-8x7B-Instruct-v0.1 |  |
| Mistral-7B | mistralai/Mistral-7B-Instruct-v0.2 |  |
| Report Card | Iterations | 5 |
| Format | Bullet Point |  |
| Dataset | Name | MMLU |
| Topics | High School Chemistry |  |
| High School Mathematics |  |  |
| High School Physics |  |  |
| Machine Learning |  |  |
| Contrastive Guessing | Default Guesser Model | meta-llama/Meta-Llama-3-70B-Instruct |
| 3-shot Samples | 120 |  |

**Table 7: Ablation Study Results. Llama-3-70B is used as the guesser if not labeled explicitly. No Chain of Thought was applied by default.**
| Ablation Category | Experiment | HS Chem | HS Math | HS Phys | ML |
| --- | --- | --- | --- | --- | --- |
| Formulation | 2C2A | 0.64 | 0.73 | 0.68 | 0.63 |
| 1C2A | 0.56 | 0.65 | 0.61 | 0.54 |  |
| Card Format | Paragraph | 0.60 | 0.69 | 0.64 | 0.57 |
| Hierarchical | 0.64 | 0.73 | 0.68 | 0.63 |  |
| Bullet Point | 0.70 | 0.74 | 0.72 | 0.61 |  |
| Card Iteration | Iteration 1 | 0.61 | 0.72 | 0.64 | 0.59 |
| Iteration 5 | 0.65 | 0.73 | 0.69 | 0.61 |  |
| Guesser | Llama-3.1-405B | 0.78 | 0.84 | 0.78 | 0.70 |
| Llama-3.1-70B | 0.69 | 0.75 | 0.72 | 0.65 |  |
| Llama-3-70B | 0.64 | 0.73 | 0.68 | 0.63 |  |
| Llama-3-8B | 0.43 | 0.49 | 0.44 | 0.45 |  |
| Chain of Thought | No CoT | 0.70 | 0.75 | 0.71 | 0.60 |
| With CoT | 0.71 | 0.77 | 0.76 | 0.65 |  |
| No. Shots in Quiz (CoT) | 1-shot | 0.66 | 0.71 | 0.66 | 0.63 |
| 3-shot | 0.71 | 0.77 | 0.76 | 0.65 |  |

### C.3 De-stylization

We de-stylized completions in Adv. AI Risk and MMLU using prompts described in Section F.4. Here we present examples of de-stylization. Figures 11 and 12 provide examples of how we perform de-stylization on each dataset.

Figure: Figure 11: (Top) A query from MMLU High School Physics (Middle) Completion from Mixtral-8x7B-Instruct-v0.1 (Bottom) De-stylized completion using GPT-4-Turbo. We did not include choices for the ease of presentation.

Figure: Figure 12: (Top) A query from Adv. AI Risk Corrigible-less-HHH (Middle) Completion from Meta-Llama-3-8B-Instruct (Bottom) De-stylized completion by removing reasoning.

## Appendix D Elo Computation Details

When computing Elo, we treat each pairwise comparison as a “match” (between models) and randomize the order of the matches prior to computing Elo.

#### MMLU

For MMLU, the $R^{2}$ value was aggregated across each subtopic by taking the average correlation across each subtopic. For each subtopic, we compared correctness across 272 model pairs (17 models) for each question, resulting in a total of 16,320 comparisons. The matching scheme was as follows: For a pair of models $i$ and $j$, we determined $i\succ j$ if $i$ answered the question correctly while $j$ did not. Ties were excluded from the analysis. To mitigate ordering effects, we averaged the score from both orderings.

#### CN Grammar

For the Chinese Grammar evaluation, we employed LLM-as-judge on 16 randomly sampled queries per model pair. The LLM-as-judge determined the better completion using the prompts outlined in Section F.6. Since Llama 3 models consistently respond with English, they were excluded from this task, leaving us with 3,360 comparisons across 210 model pairs. For the few-shot baseline, we treated each few-shot example set as a Report Card, ensuring the same number of comparisons as the card. We used gpt-4o-mini-2024-07-18 as the judge. To mitigate ordering bias, each model pair was compared twice, with the orders reversed. For each match, the judge definitively determined a winner and a loser.

#### Card Elo

Card Elo is computed similarly to completion Elo using LLM-as-judge. We compare each pair twice (with the reversed ordering of the cards) and randomize the order of the matches. Detailed prompts for the pairwise comparison of Report Cards are provided in Section F.5.

Figure: Figure 13: Comparisions required vs. Faithfulness by different comparision methods.
Refer to caption: x15.png

Note that, as teh Ground-truth Elo requires comparing completions against the entire dataset, it requires $60\times 2$ comparisions per model pairs. While Report card requires only 2 comparision per model pair. Our result demonstrates that Report Cards achievecs significantly higher faithfulness while requires less comparisions.

#### Elo Score calculation

The Elo rating is updated after each comparison using the formula:

$$ $\displaystyle R^{\prime}=R+K\cdot(S-E)$ (1) $$

Where $R^{\prime}$ is the new Elo rating, $R$ is the current Elo rating, $K=32$ is a constant, $S$ is the actual outcome (1 for a win, 0 for a loss), and $E$ is the expected outcome, calculated as:

$$ $\displaystyle E=1/(1+10^{\frac{R_{\text{opponent}}-R}{400}}).$ (2) $$

The initial rating for all models is set to 1200.

## Appendix E Human Scoring Details

**Table 8: Human Scoring Setup**
| Category | Variable Name | Value |
| --- | --- | --- |
| Student Models | GPT-4o | gpt-4o-2024-05-13 |
| Llama3-8B | meta-llama/Meta-Llama-3-8B-Instruct |  |
| Mistral-7B | mistralai/Mistral-7B-Instruct-v0.2 |  |
| Teacher Model | GPT-4o | gpt-4o-2024-05-13 |
| Rater Model | Llama3.1-70B | meta-llama/Meta-Llama-3.1-70B-Instruct |
| Report Cards | Iterations | 1, 5 |
| Format | Bullet Point |  |
| Dataset | Name | MMLU, Adv. AI Safety Risk |
| Topics | College Mathematics |  |
| High School Mathematics |  |  |
| High School Physics |  |  |
| Machine Learning |  |  |
| Power Seeking Inclination |  |  |
| Corrigible Less HHH |  |  |
| Collected Data | Familiarity | {1, 2, 3} |
| Relevance Score | {1, 2, 3, 4, 5} |  |
| Informativeness Score | {1, 2, 3, 4, 5} |  |
| Clarity Score | {1, 2, 3, 4, 5} |  |
| IP | Volunteer’s IP |  |
| Notes | Additional information from volunteers |  |
| Human Resources | Number of Volunteers | 18 |
| Number of Valid Entries | 230 |  |

Figure: Figure 14: An human scoring example.
Refer to caption: x16.png

#### Scoring Process

For both LLM and human raters, we employ the same rating process. For each question in the test batch given a specific dataset and topic, we provide LLM and human raters with the relevant part of the Report Card (see Report Card Excerpts below) and the student model’s response to the question, and have them rate the Report Card on the following 3 metrics:

- •
Relevance: How relevant is the Report Card to the given question?
- •
Informativeness: How informative is the Report Card about the (student) model’s capabilities with respect to the question and the model answer?
- •
Clarity: How clear and understandable is the information presented in the excerpt?

Following this process, we obtain scores for questions in the test batch (60 questions in total). Limited by resources, we cannot collect scores for every question and excerpt, and the number of total samples we collected is specified in Table 8. We aggregate the scores of a Report Card by taking the mean. The instructions given to volunteers are provided in Section E.1, and the prompt given to LLMs can be viewed in Section F.7. Hyperparameters for both human and LLM scoring are presented in Table 8.

#### Report Cards Excerpts

To mitigate the effort for volunteers in reading and processing long Report Cards, we excerpt Report Cards (prompts in Appendix F.8) using a LLM to extract relevant parts to the question and model answer. Then, the resulting excerpts of Report Cards are presented to both LLMs and volunteers for rating.

#### Scoring Web Interface

We set up a website for volunteers to rate our Report Cards based on questions and model responses. A screenshot of the interface is shown in Figure 15.

### E.1 Human Instructions

Here we present the instructions we gave to volunteers to rate Report Cards. For prompts given to LLMs, please refer to Section F.7.

Figure: Figure 15: A screenshot of the scoring website.
Refer to caption: extracted/5811795/figures/appendix/human_scoring/human_scoring_interface.jpeg

### E.2 Human-LLM Alignment Investigations

**Table 9: Correlation coefficient results for aligning LLM scores to human scores. “Instruction-only” refers to the prompt in Section F.7. Cohen’s Kappa is computed by binning $\{1,2\}$ as low, $\{3\}$ as medium, and $\{4,5\}$ as high. MAE refers to mean absolute error. For 2 and 3-shots, human instructions are also prompted.**
| Aspect | Prompts | Spearman Correlation | Cohen’s Kappa | MAE |
| --- | --- | --- | --- | --- |
| Relevance | Instruction-only | 0.27 | 0.14 | 0.97 |
| 2-shot | 0.25 | 0.12 | 1.03 |  |
| 3-shot | 0.34 | 0.23 | 1.00 |  |
| Informativeness | Instruction-only | 0.31 | 0.23 | 1.04 |
| 2-shot | 0.40 | 0.14 | 1.15 |  |
| 3-shot | 0.39 | 0.08 | 1.18 |  |
| Clarity | Instruction-only | 0.04 | 0.06 | 0.55 |
| 2-shot | 0.16 | -0.01 | 0.41 |  |
| 3-shot | 0.00 | -0.01 | 0.41 |  |

To automate the scoring process, we attempted to prompt LLMs with almost the same instructions as Section E.1. Prompts can be found in Section F.7. For the human instruction, we included an additional “familiarity” aspect but we omitted it in LLM prompts.
See Table 9 for results.

The distribution of LLM scores over human scores is visualized in Figure 7. We can observe a weak-to-moderate alignment between LLMs and humans.

## Appendix F Prompts

For each section, we will present the system prompt first, and then the user prompt.

### F.1 Progression Step in PRESS

In this section, we only show the prompt for generating the bullet point format (Appendix A) Report Cards. Prompts for other formats are similarly defined and can be accessed in our repository (^3^33[https://github.com/blairyeung/LLM_eval/tree/main](https://github.com/blairyeung/LLM_eval/tree/main)).

### F.2 Refinement Step in PRESS

### F.3 Contrastive Accuracy

### F.4 Paraphrasing for Robustness Check

### F.5 Skill Elo

### F.6 Completion Elo

### F.7 LLM Scoring

### F.8 Report Cards Excerpt Generation for Human Evaluation

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see above.
soundness: 3
strengths: This paper is generally well-written and is very interesting.
summary: This paper proposes a framework for qualitative evaluation, based on specificity, faithfulness, and interoperability. To do so, the paper proposes a PRESS algorithm to generate natural language descriptions for LLMs in an iterative way. The experiments cover multiple LLMs, and the corresponding analyses read well.

**update after rebuttal**: I am not fully convinced by the authors' response, in particular the human evaluation part. If the authors suggest that they need more time to collect data for reliable evaluation, I suggest they fully prepare it and submit the paper to other suitable conferences.
weaknesses: 1. There are other works focusing on qualitative evaluation, such as [1], which formulate the problem in a very similar way, i.e., generating human languages to describe the limitation of LLMs.
2. There are some unavoidable and major limitations, although the authors have discussed them in the limitation section. For example, the selection of tasks is limited, and seems that the framework cannot be easily extended to generation tasks.
3. The human evaluation to me is not that reliable. There are 18 annotators but with only 230 instances. I am concerned that there can be a bid variance between different annotators. Also, I did not find the human annotation agreement. Could please the authors remind me if I am wrong?

[1] See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknesses

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1)) Assuming a specific case, a question that cannot be reliably solved by the judge model as well, but a fine-tuning of the base model does solve it, can a judge model reliably generate a summary for the same? Is the method generally constrained by the capability of the larger, better model? Is there any recourse for this?
2)  Is the readability of the report card considered at the macro level? What is done to balance specificity vs general comments? Figure 10 is a very good example of capturing nuances. Still, if this is done regularly at a global level across questions and subjects, this would quickly add to the length of the report card, and thus, the readability of the report card would be affected, right?

Could you showcase a full report card for any model of your choice? While the methodology to create it makes sense, it is not quite the same as looking at the final product.
soundness: 3
strengths: 1) There is a need for more than model cards in the LLM information space, while model cards report model parameters, and other minute details like data, training steps (including post training), and evaluations on various benchmarks, they do not provide holistic insights into model capabilities and nor are these insights granular in nature. Evaluations on benchmarks are ultimately unidimensional and limited to the number, and do not provide specific insights into what the model is good or bad at.

2) It also perhaps addresses a real need in LLM development and deployment, where model providers or users are interested in models that are better at specific tasks (or sub tasks) vs overly general models.
summary: This work proposes Report Cards, human-interpretable natural language summaries, based on three criteria: specificity, faithfulness, and interpretability. They also propose PRESS an iterative algorithm for generating report cards. They show through experimentation with popular LLMs that report cards provide insights beyond traditional benchmarks.

PRESS (Progressive Refinement for Effective Skill Summarization) is an iterative algorithm for generating Report Cards without human supervision. PRESS works by iteratively generating and refining summaries of model behavior, focusing on specific aspects in each iteration and synthesizing them into a comprehensive overview.

The authors conducted experiments across multiple datasets, including MMLU (Massive Multitask Language Understanding), the Anthropic Advanced AI Risk dataset, and a Chinese grammar dataset. They tested various models, from OSS models like Llama-3.1-8B-Instruct and Mistral-7 B to closed models like GPT-4 and Claude 3.5 Sonnet.
weaknesses: 1) The PRESS algorithm ultimately depends on a judge model (in this case, Claude) to provide the final report card; usage of Claude or any model at that scale would add some costs to the generation of the report card vs. model cards that do not rely on LLM judgement. 

2) The reliance on LLM as a judge model perhaps also showcases a shortcoming, would a model be good at verifying or recognizing a model better than it or rely on its own (perhaps wrong) judgement. 

3) It is not wholly clear if the report card works for advanced topics, MMLU while it covers a variety of topics, is not a benchmark covering all the depth and breadth of any one subject.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: NA
soundness: 2
strengths: The paper's focus on getting more insights into LLM performance, beyond just accuracy numbers, is an important topic. The presentation is reasonably clear, with good descriptions of the approach.
summary: The paper suggests a qualitative way to summarize an LLM's performance on a benchmark using "Report Cards". E.g., for Llama-70B-Instruct on MMLU High School Physics, the report card describes strength/weaknesses across ~10 dynamic facets, like "* Newton's Laws Mastery: The student demonstrates a solid understanding of Newton's laws, ...  However, it shows a misunderstanding of Newton's third law ...".

They generate report cards for 9 models across 10 tasks, evaluating their quality through some proxy metrics: 1) Contrastive accuracy: How well can a "guesser LLM" look at two models' reponses to a quiz (k questions in a task) along with a report card for each model, and guess which responses are from which model. 2) Card Elo: A "judge LLM" compares to report cards to decide which is the "best" model, to derive a "Card Elo" rating. How well does that agree with some "ground truth" Elo rating of the models? 3) Human scoring of report cards clarity, relevance, informativeness.

They use an "evaluator LLM" to generate the report cards, trying two main approaches (one-pass prompting and iterative PRESS method). The results on the proxy metrics indicate that the iterative method works better, and also is better than some other baseline approaches for each metric.
weaknesses: The utility of these report cards is very unclear, the few concrete examples given in the paper give little confidence that any actionable insights can be derived.

The report cards are trying to identify useful subtopics for a task, and describe a model's performance on such sub topic. A much more concrete way of doing that would be to generate some hypotheses (like the example in Figure 10 that Llama-3-8B-Instruct "struggles with combinatorial concepts".) based on seeing a subset of the dataset. Then associate that with a score on that subtopic (model scores 45% on questions with "combinatorial concepts" vs 65% overall). Then further test that hypothesis on a held out set of questions to see if the drop in performance hold up. This seems like a minimal expectation for this kind of work.

The proxy metric of "Contrastive accuracy" seems like it would have many confounders, and a strong dependency on the size of the quizzes (which sounds like it's 120 questions, it's a bit unclear). Some attempts are done to "de-stylizing" the model outputs, which deals with the confounder of CoT style, that's a good idea. But as a measure of report card utility, it seems quite farfetched.

The Card Elo metric is extracting a measure of overall performance from the report card itself (according to a judge LLM), computing correlation with Elo based on actual task performance. They get reasonable R^2 scores of around 0.8, beating out R^2 from a generic ChatBot Arena Elo. It's far from perfect in ranking the models though, so not a substitute for the raw score.

This section has a quote "Importantly, k-shot completion Elo, using the same number of comparisons as Card Elo, obtains a significantly worse faithfulness score than Card Elo." which is a bit unclear in what is meant. It looks like "k" in this context is 2 or 4 (according to section 3.2, this is a different "k" from the 120 apparently used for quiz sizes?). So unclear what "same number of comparisons" means here, maybe the number of model matchups which seems a bit irrelevant (there's only a fixed number of models to possibly match up here).

Finally, the human evaluation (and associated LLM-simulating-human evaluation) of the report cards could be the most insightful. But the most interesting "informativeness" score (which averages a high 4.5 out of 5 from the humans) is based on a generic question: "How informative is the excerpt about the model's capabilities with respect to the question and the model answer?" with choices like

4. Very informative: Provides comprehensive information, covering most key aspects.
5. Extremely informative: Provides extensive, detailed information, covering all key aspects.

which tells us very little about how effective this report card is to actually understand something useful about the models' performance. 

Admittedly this is a hard task to evaluate, but the current effort leaves me with no confidence that these report cards are actually useful for anything. 

Also missing is any analysis across the models' report cards to hypothesize actionable differences, both for improving the models, or at least to estimate performance on new subsets.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: N/A
soundness: 2
strengths: * a new method for evaluation of models on text generation tasks is proposed
* it provides high human interpretability and achieves high accuracy together with 2 evaluation methods, Contrastive Evaluation and Card Elo
summary: The paper introduces Report Cards, a technique to evaluate LLMs using natural language instructions, and provide interpretable and holistic results.
The authors present the algorithm for progressively generating report cards, PRESS - it features accumulating / summarizing LLM judgement results of models' predictions for text-based tasks. They as well present 3 ways to evaluate Report Cards themselves: the Contrastive Accuracy (given 2 model predictions and 2 report cards, a judge is asked to match a report card to the prediction), Card Elo (given 2 predictions and 2 cards, determine a winner), and human scoring.

They evaluate Report cards on 3 datasets (MMLU, Adv. AI Risk and CN Grammar) using a range of small to large backbone models.
In the contrastive learning judgement, they compare Report Cards with 2 baselines: Constant (judging based on accuracy on ground truth labels) and few-shot (instead of report cards, the contrastive judge is given k predictions the training set)
Card Elo is evaluated via R2 score against Ground Truth Elo (for which either gold labels or another LLM judge are used). Finally, in human scoring, the cards were assessed on a 5-point Likert scale along Relevance, Informativeness and Clarity dimensions.

Evaluations show that Report cards achieve competitive results using the proposed Contrastive Evaluation technique, although with an unexpectedly high result of the few-shot baseline; Card Elo achieves superior R2 scores against competing Elo methods and baselines; Report Cards also attain consistently high scores in human evaluation.
weaknesses: * few-shot baseline for Contrastive evaluation is confusing - it's unclear to me how informative is matching a model's prediction on testset to k sampled predictions on the trainset
* de-stylization issue suggests that few-shot baseline captures not very useful information and should ideally be re-thought
* Ground truth Elo for R2 faithfulness evaluation uses yet another judge - so it should probably be renamed into Oracle Elo

Meta-review: The paper introduces "Report Cards" as a qualitative evaluation and description of large language models through human-interpretable natural language summaries. The authors proposed methods to evaluate the quality of generated cards. They also proposed an iterative algorithm called PRESS for generating these summaries. It basically allow the generating model to improving its summary by asking candidate LLMs quizs in multiple iterations. Experimental results show that Report Cards provide fine-grained insights into LLM capabilities.

Strengths
1. The idea of having qualitative evaluation instead of quantitive evluation of LLMs is interesting, and it is becoming useful with the spreading of LLM's practical usages. 
2.  The introduction of specificity, faithfulness, and interpretability as evaluation metrics of generated cards provides a structured framework of evaluating the quality of Report Cards.
3. The method reveals nuanced capabilities of LLMs, such as reasoning and contextual understanding, that are challenging to assess with traditional benchmarks.

Weaknesses
1. The experiments focus on specific datasets and tasks, limiting generalizability to broader or more open-ended domains 
2. The paper relies on limited human evaluations, raising concerns about the reliability of interpretability assessments.
3. Several reviewers noted a lack of clarity on how Report Cards provide actionable insights, particularly in practical applications deployment of LLMs.

While the paper presents a compelling and novel approach to evaluating LLMs qualitatively, it falls short in addressing key issues of task generalizability, human evaluation reliability, and its potential in practical applications. The reviewers are divided, with one leaning towards acceptance due to the paper’s novel contribution, while others express reservations regarding reliability and insights. The rebuttal effectively addressed some concerns, but critical gaps remain in human evaluation statistics and broader applicability.

Decision: Reject