reviews: 
Comments: This paper presents SaMer, a new framework to evaluate LLM responses with more flexibility. Instead of using static criteria for evaluating open-ended generation, SaMer adapts its approach based on the context, whether it's looking at a creative response, answering a factual question, or offering advice. To achieve this, the authors created a large-scale dataset covering a variety of real-life scenarios so SaMer could learn to recognize what matters most in different situations. The framework is set up with three parts: one to pick relevant criteria, one to score based on those criteria, and one to weigh them for an overall score. 

Experiments indicate that SaMer performs competitively, achieving improvements over baseline models across single rating, pairwise comparison, and fine-grained dimension-specific benchmarks. It also sheds light on which factors are key in different scenarios. Overall, SaMer brings a more adaptable, context-sensitive way to evaluate LLM responses, making it better suited to assess responses based on what's actually needed in each situation.
Strengths: 1. The motivation for a more fine-grained, multi-dimensional evaluation of LLM responses is reasonable and relatively under-explored, as current evaluations usually rely on a single overall score.
2. The authors have built a large-scale preference dataset from existing corpora, covering diverse scenarios and annotating each with corresponding evaluation dimensions.
3. Experimentally, the model shows performance improvements over baselines across several settings.
Weaknesses: **Major Issues:**

1. **The choice of evaluation dimensions for each scenario is unclear**: The authors mention that three graduate students annotated the evaluation dimensions for each scenario. However, this task seems highly subjective and challenging. From Table A2, many dimensions appear vague or redundant. For instance, why doesn’t "data analysis" include "Accuracy"? Why is "Logic" missing for "reading comprehension"? “Recommendation” seems subjective, so why does it include "Objectivity"? "Accuracy" and "Admit Uncertainty" also seem somewhat contradictory, yet both are included in five scenarios. "Being Friendly" and "Audience Friendly" seem overlapping. Overall, the annotations lack clarity and intuitiveness. The authors could improve this by defining these dimensions and including the guidelines used for annotation in the paper. Additionally, reporting inter-annotator agreement with discussion would be helpful.

2. **The predicted evaluation dimensions are still pre-defined**: Although SaMer claims to flexibly predict relevant evaluation dimensions per scenario, it actually selects from a set of 42 dimensions defined in previous studies. This limits SaMer's ability to generalize to broader, unforeseen scenarios.

3. **No evaluation of selected evaluation dimensions**: The experiments in this paper focus on overall response preference or specific fixed dimensions. However, one of SaMer's primary contributions is automating the prediction of relevant dimensions for each scenario, yet there is no experiment supporting SaMer’s effectiveness in this. A potential experiment could involve having LLMs generate reasonable evaluation dimensions for given scenarios as a baseline, then comparing those results with SaMer's using human judgment as a metric.

4. **Missing related work**: Multi-dimensional and explainable evaluations for text generation are not new ideas, and it would be helpful for the authors to include comparisons and discussions with prior works such as [1-3].

**Minor Issues:**

1. **Potential unfair comparisons**: The training of SaMer heavily relies on GPT-4o for preference annotation, while some baselines may not have used this newer and potentially stronger teacher model, which could lead to unfair comparisons.
2. **Lack of efficiency discussion**: Since SaMer first predicts evaluation dimensions before aggregating them into an overall score, discussing runtime and efficiency comparisons would help the community better understand the practical efficiency of this framework.

**References:**

[1] Zhong et al. Towards a Unified Multi-Dimensional Evaluator for Text Generation. EMNLP 2022.

[2] Liu et al. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. EMNLP 2023.

[3] Xu et al. INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback. EMNLP 2023.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3