reviews: 
Comments: This paper introduces SaMer, a novel scenario-aware multi-dimensional evaluator designed to assess the quality of Large Language Model (LLM) responses to open-ended questions. The work addresses a significant challenge in LLM evaluation: the complex, multifaceted nature of response quality assessment, which varies significantly across different types of queries and use cases. While traditional LLM evaluators typically provide only overall scores or rankings, with some offering limited multidimensional evaluations across fixed dimensions like harmlessness, honesty, and helpfulness, different scenarios require different evaluation criteria. The key innovation of SaMer lies in its ability to provide both overall and fine-grained assessments of LLM-generated responses, with a flexible framework that adaptively identifies and prioritizes relevant evaluation dimensions based on the query type. This approach contrasts with existing methods that rely on predetermined evaluation dimensions. To support this framework, the authors developed a large-scale fine-grained preference dataset encompassing 42 evaluation dimensions across 36 scenarios, with scenarios categorized into five main types based on Maslow's hierarchy. Three graduate students created dimension definitions, and the final dataset contains 135,402 total data points, with 2,000-5,000 samples per scenario.
The dataset construction started with the collection of preference data (win/lose/tie labels) from multiple sources, including custom datasets built using LLM-generated annotations. Scenario annotation utilized a custom-trained LLaMA-3-8B classifier on [Li et al., 2023] data, enriched with GPT4-o-mini annotations for target scenarios without previously existing annotations. The sampling process initially gathered 6,000+ instances per scenario, followed by GPT-4o-mini verification of scenario labels and final balancing to 2,000-5,000 samples per scenario. Fine-grained preference annotation was performed using GPT-4o.
The model architecture consists of a frozen pre-trained LLaMA-3-8B (pre-trained for rewarding using [Wang et al., 2024]) as the text encoder, with its original output projection layer removed. On top of this encoder, the model employs three specialized MLP heads: a quality dimension prediction layer performing multi-label classification using ZLPR loss, a pairwise preference learning layer using margin ranking loss, and a weighting layer for overall scoring also utilizing margin ranking loss. The training process combines these losses (L_{zlpr}, L_{dim}, and L_{o}) through weight hyperparameters lambda.
The evaluation of SaMer was conducted across 9 benchmarks spanning 3 families. For single rating benchmarks, the assessment used Pearson, Spearman, and Kendall-Tau correlations, while preference selection tasks were evaluated using accuracy metrics for pairwise and fine-grained comparisons. The results demonstrate SaMer's superior performance compared to existing baselines, showing strong adaptability across diverse scenarios while providing interpretable, fine-grained evaluations.
Strengths: - The paper is clear and well-structured, effectively communicating complex ideas through a logical flow and professional presentation.
- The fine-grained preference dataset represents a valuable contribution to the field.
- The model is efficient and practical, making it accessible for deployment and fine-tuning.
- The evaluation is thorough and comprehensive, benchmarking against both proprietary models and open-source alternatives.
Weaknesses: - From a methodological standpoint, the paper's novelty is limited. While it combines multiple components to create a comprehensive evaluation system, the individual techniques used are standard approaches in the field, and many data resources are derived. The main innovation lies in the application rather than the methodology itself.
- The hyperparameter analysis lacks depth and transparency. For example, the authors fix lambda values to combine the different losses without exploring or discussing their impact on model performance. Similarly, other hyperparameters are presented with single values, offering no insights into their selection process or their effects on the model's behavior. This absence of ablation studies or sensitivity analyses makes it difficult to understand the robustness of the approach.
- The use of GPT-4o-based models on this scale for intensive annotation and filtering is resource-intensive and appreciable. However, this heavy reliance can introduce potential biases and raises questions about the overall training dataset quality.
- There is no discussion of verification steps or quality control through human analysis to ensure the artificial annotations' reliability. This is particularly important given the subjective nature of many evaluation dimensions.
- The supplementary materials lack data samples, making it impossible for reviewers to assess the quality and characteristics of the dataset.
- The metrics section (5.2) requires a more detailed explanation.
- The paper does not clearly provide information about code availability, model checkpoint distribution, or dataset licensing.
Recommendation: 8
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 3