Kati8/ICLR_2023_2025 huggingface dataset:
authors: Kehua Feng
authors: Keyan Ding
authors: Jing Yu
authors: Yiwen Qu
authors: Zhiwen Chen
authors: chengfei lv
authors: Gang Yu
authors: Qiang Zhang
authors: Huajun Chen
reviews: 
Comments: This paper proposes an evaluation model (SaMer) that assess the quality of responses based a 42 pre-selected dimensions and gives an overall evaluation by weighting scores on different dimensions. The experimental results show that the proposed model achieves the best efficacy on single rating datasets and the fine-grained comparion data curated by the authors, and it also achieves a comparable performance with ArmoRM on pairwise comparison datasets.
Strengths: - Compared to past evaluators that assessed overall quality, the fine-grained evaluation method proposed in this paper provides more fine-grained evaluation results. The weights across different dimensions offer a certain level of interpretability.

- Since this model can implicitly determine the required dimensions based solely on the query without task information, this evaluator is task-agnostic and possesses a certain degree of generality.
Weaknesses: - The evaluator's ability to understand different dimensions is questionable. Firstly, the 42 dimensions are fixed, and the model did not even encounter these dimension definitions during training (nor did the paper provide their specific definitions). Secondly, the experimental results do not demonstrate whether the evaluator can accurately determine which dimensions are needed when encountering out-of-domain data. Compared to similar work like X-Eval [1], which allows users to input personalized dimension definitions, it is hard to say that this model has a clear advantage.

- The paper employs three loss functions but lacks ablation studies to investigate the impact of choosing $\lambda_1$ and $\lambda_2$ on model performance, making it difficult for readers to understand the individual contributions of these designs.



[1] X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects (Liu et al., NAACL 2024)
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3