LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Models
Authors: Chongyan Sun
Ken Lin
Shiwei Wang
Hulong Wu
Chengfei Fu
Zhen Wang
Venue: COLM (2024)
Keywords: Domain-Specific Large Language Model, Human Evaluation, Holistic Framework, End-to-end Protocol, Logistic Industry
TL;DR: This paper introduces LalaEval, a holistic framework designed for the human evaluation of domain-specific large language models and presents its deployment with real-world outcomes in the logistics industry.

Abstract: This paper introduces LalaEval, a holistic framework designed for the human evaluation of domain-specific large language models (LLMs). LalaEval proposes a comprehensive suite of end-to-end protocols that cover five main components including domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, and thorough analysis and interpretation of evaluation outcomes. This initiative aims to fill a crucial research gap by providing a systematic methodology for conducting standardized human evaluations within specific domains, a practice that, despite its widespread application, lacks substantial coverage in the literature and human evaluation are often criticized to be less reliable due to subjective factors, so standardized procedures adapted to the nuanced requirements of specific domains or even individual organizations are in great need. Furthermore, the paper demonstrates the framework's application within the logistics industry and a comparative analysis of LLMs for the logistics domain use, highlighting the framework's capacity to elucidate performance differences and guide model selection and development for domain-specific LLMs. Through real-world deployment, the paper underscores the framework's effectiveness in advancing the field of domain-specific LLM evaluation, thereby contributing significantly to the ongoing discussion on LLMs' practical utility and performance in domain-specific applications.

Published as a conference paper at COLM 2024
LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Models
Chongyan Sun Ken Lin", Shiwei Wang", Hulong Wu', Chengfei Fu², Zhen Wang?
'The Chinese University Hong Kong Data Science Group, Huolala sunchongyan@l ink. cuhk. edu. hk [adam. lin, three. wang,
senter. sangee wu, eden. fu, waw.kjcccooo...wagggg.......
Abstract
This paper introduces LalaEval, holistic framework designed for the hu- man evaluation of domain-specific large language models (LLMs). LalaEval proposes a comprehensive suite of end-to-end protocols that cover five main components including domain specification, criteria establishment, bench- mark dataset creation, construction of evaluation rubrics, and thorough analysis and interpretation of evaluation outcomes. This initiative aims fill crucial research gap by providing a systematic methodology for conducting standardized human evaluations within specific domains, a practice that, despite its widespread application, lacks substantial coverage the literature and human evaluation are often criticized to be less reliable due to subjective factors, so standardized procedures adapted to the nu- anced requirements of specific domains even individual organizations are in great need. Furthermore, the paper demonstrates the framework' ss application within the logistics industry, presenting domain-specific eval- uation benchmarks, datasets, and comparative analysis of LLMs for the logistics domain use, highlighting the framework's capacity to elucidate performance differences and guide model selection and development for domain-specific LLMs.. Through real-world deployment, the paper un- derscores the framework's effectiveness in advancing the field of domain- specific LLM evaluation, thereby contributing significantly the ongoing discussion on LLMs' practical utility and performance in domain-specific applications.
1 Introduction
The rise of large language models (LLMs) represents significant step towards artificial general intelligence (Bubeck et al., 2023), showcasing their powerful ability to understand and produce natural language. Although these models have generally been designed for wide-ranging use, one of their most promising applications lies within specific domains such as medicine (Lee et al., 2023; Wang et al., 2023a; Thirunavukarasu al., 2023), law (Cui et al., 2023) and finance (Li et al., 2023; Wu et al., 2023). For businesses looking to integrate LLMs into their operations, the focus naturally shifts towards models' capabilities in certain industries, as conversations within a particular industry are more prevalent and relevant than broad conversations when in real-world business setting (Guo & Yu, 2022; Zhao al., 2023).
The process of evaluating these domain-specific models is crucial. Typically, evaluations range from automatic techniques, e.g. Zheng et al. (2024), to human evaluation, with the latter widely regarded as the most comprehensive method. Human evaluation's importance lies in its unparalleled ability to grasp the intricacies of language and context, aspects that automatic methods might miss (Novikova et al., 2017), and more importantly in commercial use, human evaluation also reflects true human preference of businesses' stakeholders, so when developing and comparing LLMs, particularly those meant for domain-specifff real-world business use, human evaluation becomes essentially gold standard. It plays

---

Published as a conference paper at COLM 2024
critical role in measuring how these models perform for commercial use and guides the development process, ensuring the models meet real-world interaction standards.
Given the importance of human evaluation (Chang et al., 2023), creating standardized framework for human evaluation is vital. This framework, which includes detailed methods for gathering data, setting evaluation rubrics, defining metrics well as establishing respective evaluation protocols, is essential for ensuring evaluations are consistent, accurate and relevant. It specifically addresses the need for domain-specifie LLMs to be closely aligned with actual business requirements, filling the gap between theoretical capabilities and practical application. By focusing on the case study of the logistics domain, our study introduces a detailed evaluation framework designed for a specific domain. This framework not just an example of the evaluation within this industry, also showcases standardized human evaluation framework of domain-specific LLMs in general.
This research makes meaningful contribution to the discussion on evaluating LLMs, espe- cially for domain-specific uses. By outlining the creation of a human evaluation framework that matches commercial needs, we set path for a more focused, efficient, and practical use of LLMs in business contexts. Our deployed framework in the logistics domain demon- strates the framework's value to improve the relevance, functionality, and adoption of LLMs in specific domains.
2 Related Work
Chang et al. (2023) presents an extensive survey on the evaluation of LLMs, providing detailed taxonomy of evaluation aspects, including what, where, and how to evaluate. General-purpose models, such as OpenAl's GPT and Google's Gemini, typically do not serve downstream tasks directly. Thus, their evaluation often involves a multifaceted approach (Bang et al., 2023), incorporating a variety of tasks, as demonstrated by Bai et al. (2024); Bian et al. (2023); Liang et al. (2022).
In contrast, the most economically potent applications for commercial, domain-specific use are arguably found in customer operations, i.e. service chatbots (Chui et al., 2023). Accordingly, this paper concentrates on developing an evaluation framework for domain- specific LLMs, particularly for chatbot conversations. Although initially designed for single- round conversations, this framework can be readily extended to multi-round interactions.
There a significant body of work addressing where evaluate. Numerous benchmarks and datasets have been established to test the capabilities of LLMs (Hendrycks et al., 2020; Xu al., 2023; Gu et al., 2023). These efforts provide essential benchmarking datasets and results for mainstream LLMs, facilitating unified comparison. However, these benchmarks are often not directly applicable to specific company uses due to differences in domains, knowledge, applications, etc. This paper distinguishes itself by offering frameworks and protocols to guide the construction of benchmarks and datasets within commercial organi- zations for specific domains.
Another stream of literature focuses on how to evaluate. Many benchmarks adopt automatic evaluation methods to obtain results, ranging from traditional metrics (Chin-Yew 2004) to more LLM-based judgments (Wang et al., 2023b). The advantages of automatic evaluation over human evaluation are clear: it is more standardized and objective, easier to scale, and less costly. However, for non-standard tasks like open generation, human evaluation proves to be more reliable and better aligned with general human preferences (Novikova et al., 2017). Moreover, evaluating domain-specific LLMs often requires profound domain knowledge and even internal organizational knowledge. The human evaluation is typically criticized by its cost and susceptibility to subjective bias, but within the context of commercial use of domain-specific LLMs, company can often afford the expense of human evaluators and organize effective training. Moreover, domain-specific LLMs typically have large real- world influence (e.g. chatbot serving millions of consumers) and have significant impact on firm operation. In these widespread high-stake circumstances, the resources required are reasonable compared to the potential benefits of ensuring best practice. Nevertheless, there is very little literature offering holistic framework to systematically guide the implementation
2

---

Published as a conference paper at COLM 2024
of human evaluation from domain specification and evaluation dataset construction to evaluation rubrics, metrics, and reporting of results. This paper aims to bridge this gap.
Methodology
3.1 Overview
LalaEval encompasses five main components to establish comprehensive protocols guiding the evaluation process of LLMs: (1) Domain Specification, (2) Crileria Establishment, (3) Bench- mark Dataset Creation, (4) Construction of Evaluation Rubrics, and (5) Analysis and Interpretation of Evaluation Results.
1. Domain Specification: This component involves defining the scope of specific fields of interest, largely influenced by an organization's 's goals or objectives with LLMs. It establishes the evaluation process's 's boundaries.
2. Criteria Establishment: This component defines the LLMs' capability dimensions for evaluating performance, effectiveness, or suitability. This ensures that evaluations are based on relevant, objective, and consistently applied measures.
3. Benchmark Dataset Creation: This component entails developing standardized tests and compiling carefully curated data collections from scrutinized information sources. allows for evaluation under consistent conditions, facilitating compara- tive measurement and analysis.
4. Construction of Evaluation Rubrics: This component describes the careful design of grading schemes that detail specific guidelines for measuring various performance aspects. It provides structured framework train human evaluators.
5. Analysis and Interpretation of Evaluation Results: This component involves system- atically examining data collected from the evaluation process to minimize intrap- ersonal, interpersonal, intramodel, and intermodel variability. It aims to derive meaningful insights and guide decision-making. ensuring the constructive applica- tion of outcomes.
3.2 Domain specification
In the conceptualization of LalaEval, the first step entails defining the domain that is usually inherently derived from the industries within which organization operates. The breadth of such an industry definition, however, remains an area for exploration. For instance, should the evaluation scope for medical LLM include only radiology or extend to the broader field of medicine? Should the focus for financial LLM be confined to bonds, or should it cover the wider realm of capital markets?
We propose a hierarchical structuring of subdomains for LalaEval, which organizes these subdomains by their relative significance, ranging from narrow to broad scopes. Employing backward induction, we begin with a highly specific subdomain pertinent to the organi- zation, progressing towards a more generalized industry definition. By adhering the principle of mutual exclusivity and collective exhaustiveness, we enumerate the most gran- ular subdomains and their parallel subdomains, progressively ascending to encompass broader subdomains. This process is iterated multiple times until sufficiently broad domain scope is achieved.
employ qualitative prioritization to define both the scope and the hi- erarchical significance of each subdomain. This prioritization may adopt either a linear progression from the most specific to the most broad domain tree-like structure that includes parallel subdomains at various levels, contingent upon business imperatives.
Subsequently, 1
Figure presents case by hierarchical domain specification in the logistics industry, with the final chosen subdomains highlighted. We establish both the scope and priorities for subdomains, ranking them from high (PO) to low (P3) priority: PO (Intracity Freight
3

---

Published as a conference paper at COLM 2024
E-commerce Warehouse
Logistics Real Estate
Warehousing
Warehouse Equipment
Warehouse Service
Intracity Freight Transportation (PO)
Express Delivery
Intracity Transportation (P1)
Instant Delivery and Others
General Logistics (P3)
Transporation (P2)
Road Transportation (P1)
Express Freight(Less-Thnn-Loos (LTL)
Intercity Transportation (P1)
Truckload (FTL)
Wattewaay/Rillwy//Air Transportation
Supply Chain Warehousing and Distribution Services
Cross-border Logistics Services
Logistics Service
Integrated Logistics Services
Smart Logistics Services
Other Logistics Services
Figure 1: The hierarchical domain specification in the logistics industry
Transportation), P1 (Road Transportation; Intracity and Intercity Transportation), P2 (Trans- portation), and P3 (Logistics). This prioritization informs the overall evaluation process, emphasizing the critical importance of intracity freight transportation in our case while also acknowledging the integral roles of road transportation and general logistics.
This hierarchical specification of subdomains and their prioritization not only informs but also directs the subsequent components of LalaEval.
3.3 Criteria Establishment
3.3.1 General Capability
The objective establishing criteria for general capabilities to evaluate the performance of LLMs a wide array of natural language tasks not confined to any specific domain. This includes their capability in comprehending and generating natural language, recogniz- ing contextual cues, sustaining coherence throughout conversations, and processing and conveying information with accuracy.
The rationale for evaluating such general capabilities includes (1) Foundation for domain capability: A solid foundation in general capability crucial for the performance of any LLM, providing the groundwork upon which domain capability is constructed. A lack in general language capability can impede LLMs' capability to understand or generate responses accurately within specialized domains. (2) Flexibility and Adaptability: LLMs that exhibit strong general language capabilities demonstrate enhanced flexibility and adaptability. This vital for applications requiring comprehension of inputs from diverse domains or the integration of new knowledge without the need for extensive retraining. (3) Understand- ing and Reasoning: Evaluating general capabilities aids identifying LLMs' capacity for understanding complex queries, reasoning, and generating coherent responses that are contextually appropriate. These attributes are indispensable for practical applications.
To systematically evaluate these general capabilities, we have adapted from (Xu et al., 2023) and outlined dimensions general capabilities the following, and we also show example questions and defined difficulty levels in Appendix A.
Semantic Understanding: Crucial for LLMs is understanding the basic meaning of language, and special terms such idioms and cultural references, to ensure meaningful user interaction. Evaluation involves questions that test understanding across linguistic occasions.
4

---

Published as a conference paper at COLM 2024
Contextual Conversation: LLMs need to remember past interactions to maintain coherent conversations, essential for sustained engagement in customer service or conversational applications. Multi-round questions can evaluate this skill.
Answer Completeness and Coherence: Responses must be clear, concise, and address queries directly, ensuring outputs are practical and user-centric, especially in infor- mative or decision-support settings.
Factuality: Accuracy in responses, especially for questions expecting definitive answers, is critical. This is vital in industries like logistics where errors can have significant consequences. Verification against trusted sources evaluates this aspect.
Creativity: The ability to generate creative content, such marketing material or innovative responses, highlights LLMs' capacity for engaging content creation. This is evaluated through creativity requests with certain constraints.
Logical Reasoning: Tasks requiring numerical or logical deduction evaluate this capability, relevant for problem-solving in industries like logistics. Evaluation includes mathematical logic puzzles.
3.3.2 Domain Capability
The domain capability focuses on LLMs' expertise within certain industries, evaluating its understanding of domain-specific terminologies, concepts, regulations, and operational nuances. It also evaluates LLMs' capability provide insightful, accurate responses to domain-specific queries.
The rationale for evaluating domain-specifet capability is (1) Domain-Specifit Performance: The principal motive is to ascertain the efficacy of LLMs in logistics-specific applications. imperative for LLMs to not merely comprehend general language but also to exhibit profound understanding of specialized domain knowledge and intricacies. (2) Practical Usabilify: LLMs endowed with strong logistics domain capabilities hold greater practical value for industry professionals and businesses. Such LLMs are capable of providing more accurate and relevant insights, thereby facilitating enhanced decision -aaiing processes and operational efficiency. (3) Customized Solutions: Evaluating this capability allows for the development of more customized, domain-specific solutions that can address specific chal- lenges and needs within the logistics domain, providing competitive edge to businesses leveraging these LLMs.
Focusing on the dimensions of factuality and creativity, we have outlined and illustrated subdimensions of capabilities in logistics domain in the following, and we also show example questions and defined difficulty levels in Appendix B.
Conceptual and Terminological Understanding: Knowledge of specific terms and con- cepts is fundamental in accurately interpreting and responding to industry-related queries. This necessitates an evaluation that includes questions derived from the logistics domain's 's lexicon and operational nuances.
Company Information: LLMs should be conversant with key players in the industry and relevant corporate data, reflecting the model's 's utility as an informative resource. Questions about major companies, their operations, and market positions can evaluate this aspect.
Legal and Policy Knowledge: Given the regulatory environment surrounding logistics operations, LLMs must be adept at navigating legal and policy-related queries. This can be evaluated through questions that require the LLM to reference specific regulations guidelines applicable to logistics.
Industry Insights: The ability to provide informed opinions data about the logistics market, trends, and future outlooks showcases LLMs' depth of knowledge and analytical capabilities. Crafting scenarios that ask for analysis or predictions based on current data evaluates this competency.
Company-specific Knowledge: For LLMs deployed by specific companies, such as Huolala, understanding the company's services, history, and strategic vision is
5

---

Published as a conference paper at COLM 2024
crucial. This ensures the LLM can serve effective chatbot. Questions tailored to the company's 's operations and strategy evaluate this knowledge.
Creative Capability in Logistics Confext: Beyond generic creativity, the ability to gener- ate content specifically tailored to the logistics industry's 's stylistic and contextual requirements is valuable for marketing and customer engagement purposes. Ques- tions requiring domain-specific creative responses can evaluate this capability.
3.4 Benchmark Dataset Creation
The primary goal of establishing a high-quality benchmark dataset is to develop an evolving bank of question-anse aaeest (QA) pairs for human evaluation. Existing public benchmarks and datasets are not utilized due to (1) the absence of benchmarks tailored to the specific domain; (2) the proprietary benchmark's closer alignment with -emann-speccfic needs; and (3) the lack of continuous updates in most public datasets, which fails to meet the timely knowledge requirements for the business application of LLMs. Concurrently, this dataset aims to include a broad spectrum of capabilities outlined in Section 3.3, thereby ensuring uniform evaluation of LLMs' performance across diverse metrics.
The benchmark dataset are developed through the process outlined in following steps:
1. Accuntulation of the Raw Corpus: Guided by the previous components, we system- atically collect and compile original texts and corpora from verified sources. This repository of raw corpus forms the groundwork for generating insightful and pertinent QA pairs.
2. Production of QA Pairs: (1) Development of the Question Plan: Formulate a struc- tured plan outlining the desired number of QA pairs, categorized by various levels of difficulty and capabilities. This plan should be iterative to allow for the con- tinuous enhancement of the dataset. (2) Selection of Question Designers: Identify and appoint individuals responsible for question design, ensuring they possess an in-depth understanding of the evaluation framework and access to the raw corpus. (3) Creation of QA Pairs: The question designers will distill relevant information from the raw corpus or other authoritative sources to craft QA pairs. Each pair will include question and its corresponding standard answer, for instance, "Q: How many hours are there in day? A: 24 hours." Importantly, the source of information for each QA pair must be documented to maintain traceability and credibility.
3. Quality Inspection and Database Entry: Following the generation of QA pairs, thorough quality inspection is conducted to verify that the pairs adhere to prede- fined criteria. QA pairs that pass this inspection are incorporated into the dataset, whereas those failing to meet the standards are redirected back to the designers for refinement.
The process of proposing QA pairs for the benchmark dataset is critical component in the evaluation of LLMs, particularly within specific domains such as the logistics industry. Through systematic approach to compiling raw corpora, creating relevant QA pairs, and executing rigorous quality inspections, this initiative seeks to establish a comprehensive dataset. This dataset not only lays the foundation for thorough evaluation of LLMs but also facilitates the continuous improvement and benchmarking of LLMs, thereby ensuring their relevance and applicability in real-world scenarios.
3.5
Construction of Evaluation Rubrics
The objective of the construction of evaluation rubrics is to systematically evaluate the performance of various LLMs using the benchmark dataset from Section 3.4. This process is designed to guide the training of human evaluators and ensure more consistent outcomes, both intraperson and interperson.
The general evaluation scale is 0-3 points. If a response contains any incorrect information, will get 0 points. 1-3 points measure the degree of correctness, completeness, creativity etc. Meanwhile, special consideration is given to the timeliness of responses, recognizing the

---

Published as a conference paper at COLM 2024
importance of current knowledge and the LLMs' ability to reflect the most recent information or anticipate future developments. Appendix C presents the preceding general grading principle and the special consideration about timeliness.
The evaluation rubrics for evaluating the general capabilities and logistics domain capabili- ties of LLMs are comprehensively outlined in Appendices D and E, respectively.
3.6 Analysis and Interpretation of Evaluation Outcomes
After trained human evaluators ready (see Appendix G for details of human evaluator training process), we randomly draw a subset from the dataset based on agreed difficulty level and quantities in the grading process. To keep integrity, we employ a single-blind procedure, wherein model responses to the QA pairs are anonymized and presented in a randomized order to panel of at least three human evaluators. This procedure miti- gates bias, ensuring that human evaluators cannot infer the origin of the responses, thus maintaining objectivity. Table 1 shows the demo of human evaluators' interface.
Standard
Grading Principle
Position
Position
Position
Position
Question 1
Question content]
[Principle content]
Model 4 Response
Model
Response
Model 2 Response
Model Response
Question
Question content]
Principle content]
Model 3 Response
Model Response
Model 4 Response
Model Response
Table 1: The demo of human evaluators' interface
The evaluation results are systematically compiled into a four-dimensional table that en- capsulates the evaluated capabilities, question numbers, human evaluator IDs, and grades allocated to each LLM for every question like Table 2. This structured data allows for analysis across different dimensions of model performance. Grades for each model within specific capabilities are aggregated and normalized to 100 points, offering a detailed view of model capabilities. Moreover, more comprehensive grading can be achieved through the weighted aggregation of detailed grades, reflecting the LLMs' overall performance spectrum.
Capability Dimension
Question Number
Evaluator Number
Model Grade
Model Grade
Model Q Grade
Dimension 1
Question 1 Question
Evalustor 1 Evaluator 2
ASi
ASa1
ASgn
Dimension
AS
AS
ASqu
Dimension Dimension
Question & Question &
Evaluator Evaluator
ASW
A5
A5qu A (8
AS120 1)
A5
Dimension
Question Ky
Evaluator n
AS
ASqukil
Table 2: Four-dimensional table of evaluators' grading results. ASaki represents the grade given by evaluator to the response of the 9%h model for the kn, question.
The grade calculation for model y involves single dimension j grade calculation using the
formula Grade(gi) - En AS
then aggregating across dimensions to compute the
E IS₁
total grade as Grade(g) - Grade((q))) This process evaluates a LLM's capability across various dimensions and overall, by weighing each dimension's grade, equally or otherwise. Reporting includes both the grades for different dimensions and the total grade, facilitating a comprehensive assessment of each LLM's performance. Repeating these steps for all models under evaluation provides comparative analysis of their strengths and weaknesses.
Our methodology also aims reduce the subjective factors of human evaluators much as possible, incorporating automated dispute analysis and checks for grade stability and reliability. This phase is crucial for identifying low-quality grades and questions and
7

---

Published as a conference paper at COLM 2024
attributing grading fluctuations to transparent factors, such evaluator inconsistency or changes in LLM responses. The detailed dispute analysis and grade fluctuation analysis methods are described in Appendix F.
3.7 Overall Deployment Structure
After describing the five main conceptual components of LalaEval, we now turn the practicalities of deployment, presenting approach that brings our framework into opera- tional reality, ensuring comprehensive evaluation process for domain-specific LLMs. This integration strategy, designed to foster standardization, as well efficiency and adaptability, is succinctly illustrated in Figure 2. Through this streamlined depiction, the operational blueprint that guides the application of LalaEval is presented, facilitating clear under- standing of its modular architecture and the dynamic interactions essential for evaluation. The results presented in Section 4 are also generated using this deployment strategy.
Domain Domain Specification and Criteria Establishment
Deliverable
Benchmark Dataset
Dimension Explanation
Damain Specification
Capability by Dimension
Data Collection
QA Pair Production
Seed Question
Quality Check
Evalluatoon Guideline
Quality Check
Initial
Establishment
Information Source
Source
Quality Check
Question Proceclure
Standardized Procedure
High Quality Corpus
Staff
Grading Question
Question
Analysis of Evaluation Outcomes
Rubrics
Evaluation Outcome
Dispute Analysis
Question Selection
Randomization Anorymizationn
Front-Eat Display
Outcome Access
Result Analysis
Fluctuation Analayals
Evaluation Report
Candidate LLMs List Confirmation
Responses API
Evaluator Grading
Summary
Interpretation
Figure 2: The overall deployment diagram of LalaEval
4 Results
To demonstrate the effectiveness and the practical application of Lalalival, we present the zero-shot evaluation results of various LLMs including OpenAl's GPT-4 (without web access), Baidu's Ernie Bot (with web access), and our proprietary LLM vl, v2, and v3 (PLLM1/2/ /23). PLLM1/ /2/3 are iterations fine-tuned from the ChatGLM2-6B foundation, incorporating web access, aetrieeal-augmented generation, or combination of both, respec- tively.
Our evaluation demonstrated in two distinct result sets: (1) The accuracy of the LLMs (Table 3), where we present the percentage of responses achieving non-zero grades across different capability dimensions. This metric serves as a direct indicator of each LLM's capability to generate relevant and accurate response. (2) The normalized average grades (Table 4) for each LLM across capability dimensions, providing more quantitative view of LLM performance. The complete results of grades and disagreement by all capability dimensions are included in Appendix H. The results reflect various levels coverage of capability dimension defined in Section 3.3 where Domain- Conain-Factuallly is defined as the average
8

---

Published as a conference paper at COLM 2024
capability within the domain except creative capability, enabling finer-grained comparison across LLMs. The results only represent evaluation under our LalaEval framework, focusing more capabilities in the logistics domain.
Capability Dimension
GPT-4
Ernie Bot
PLLM3
PLLM2
PLLM1
Dactui-Factuality
45.0%
84.0%
93.2%
92.0%
86.2%
Domain
54.2%
86.7%
87.0%
86.0%
81.2%
General
80.0%
89.3%
68.0%
63.0%
62.7%
Overall
67.1%
88.0%
77.5%
74.5%
71.9%
Table 3:
The accuracy of evaluated LLMs by narrower to broader capability dimension coverage
Capability Dimension
GPT-4
Ernie Bot
PLLM3
PLLM2
PLLM1
Factin-Factuality
38.8
79.7
88.7
81.4
90.3
Domain
48.1
81.8
80.6
74.5
81.9
General
77.0
87.1
59.4
59.1
63.6
Overall
62.6
84.4
70.0
66.8
72.7
Table 4: The normalized average grades of evaluated LLMs by narrower to broader capability dimension coverage
The comparative analysis, illustrated in Tables 3 and 4, highlights landscape of LLM performance across different capabilities. GPT-4 and Ernie Bot exhibit closely matched strengths in general capabilities, significantly outperforming the proprietary LLMs. How- ever, the narrative shifts within the logistics domain, where proprietary LLMs, alongside Ernie Bot, demonstrate superior performance over GIT-4, possibly due to the lack of infor- mation used to fine-tune proprietary LLMs. Notably, within this domain, the proprietary LLMs marginally surpass Ernie Bot in factuality, showing their refined capability. These insights serve not only to benchmark the current state of LLMs within the company but also to direct future development efforts.
Future Work
LalaEval may have some potential limitations, including evaluator subjectivity, data selec- tion bias, dynamic changes in domain knowledge, and scalability concerns. To address these challenges and further enhance LalaEvalls effectiveness, several avenues for future work can be proposed.
Beyond procedures described in Appendix G, exploring more structured training protocols across domains can further improve the reliability and generalizability LalaEval.
Standardized Training and Assessment:
Enhanced Dataset Representation: Ensuring the benchmark dataset is comprehensive and representative in the domain can minimize biases arising from data selection. This step involves continuous refinement and expansion of dataset sources to cover diverse scenarios.
Adaptability to Dynamic Domains: Given the evolving nature of domain-specific knowledge, continuous updates to evaluation protocols and benchmarks are essen- tial. This adaptive approach helps LalaEval stay relevant and effective in terms of dynamic industry changes.
Automation Integration: Supplementing human evaluation with automated methods and support systems can streamline the evaluation process. Automation can assist in handling large volumes of data and tasks, enhancing scalability while maintaining evaluation reliability.

---

Published as a conference paper at COLM 2024
Robust Support Systems: interpreting ambiguous cases or complex scenarios can further improve evaluation consistency and reliability.
Developing robust support systems to assist evaluators in
6 Conclusion
This study introduces LalaEval, novel framework for evaluating domain-specific LLMs, with focus on standardizing human evaluations. By detailing comprehensive methodol- ogy spanning domain specification to results analysis, LalaEval addresses the critical gap in the standardized human evaluations of domain-specific LLMs, exemplified through its application in the logistics domain.
LalaEval's deployment showcases its capability to illuminate performance differences among LLMs, guiding model selection and development for domain-specific applications. LalaEval not only advances the field of LLM evaluation by establishing end-to-end human evaluation protocols but also emphasizes the importance of aligning LLM capabilities with practical needs.
LalaEval is also a general human evaluation framework which are disentangled with any specific domain. The results of only logistics domain are reported because LalaEval was developed and finalized when exploring standardized human evaluation framework in this domain. LalaEval has been applied to other domains such HR /TT maintanance/ text-to- SQL/ telrattenn and brought business value. It is believed those who are interested in employing LalaEval to evaluate other domain-specific LLMs can easily implement it.
In summary, LalaEval represents a significant step forward in the evaluation of domain- specific LLMs, offering structured and human-centric approach to understanding LLM performance. Its contributions lay the groundwork for future research and application of LLMs across various domains, highlighting the evolving need for evaluation methodologies that are as dynamic and specialized as the LLMs they seek to evaluate.
References
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, al. Benchmarking foundation models with language-model-at an-examiner. Advances in Neural Information Processing Systems, 36, 2024.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multi- modal evaluation of chatgpt reasoning, hallucination, and interactivity. ar preprint ar arXit:2302.04023 2023.
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. Chatgpt is a knowl- edgeable but inexperienced solver: An investigation of commonsense problem in large language models. arXiu preprint ar Xio:23303.16421, 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. Xr preprint ar arXim:2303.1121 122122, 2023.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Téchnology, 2023,
Lin Chin-Yew. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out, 2004, 2004.
Michael Chui, Eric Hazan, Roger Roberts, Alex Singla, and Kate Smaje. The economic
potential of generative ai. 2023.
10

---

Published as a conference paper at COLM 2024
Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases, ar aiv preprint arXiv:2306.1699,, 2023.
Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Qianyu He, Rui Xu, et al. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation. ar Xie preprint ar Xit::2306.05788, 2023.
Xu Guo and Han Yu. On the domain adaptation and generalization of pretrained language models: A survey. arXiv preprint arXiv:2211.03154, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Peter Lee, Sebastien Bubeck, and Joseph Petro. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine, 385(13):1233-1229, 2023.
Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen, Large language models in finance: A survey. In Proceedings of the Fourth ACM International Conference on Al in Finance, pp- 374-382, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Xiv preprint Xii::2211.09110, 2022.
Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. Why we need new evaluation metrics for nlg. arv preprint ar atXix:170700683 2017.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 199391193, 2023.
Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. Huatuo: Tuning llama model with chinese medical knowledge. arxiv preprint ar xrXiv:2304.06975, 2023a.
Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for Ilm instruction tuning optimization. arXie preprint ar wxxo:2306.0008/1 2023b.
Shijie Wu, Ozan Trsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint ar XxXoo:3303.1756, 2023.
Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. Superclue: A comprehensive chinese large language model benchmark. ar Xiv preprint ar XrX::2237..10020 2023.
Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowd- hury, Li Yun, Hejie Cui, Zhang Xuchao, Tianjiao Zhao, al. Domain specialization as the key to make large language models disruptive: A comprehensive survey. ar xiv preprint NXX::23055.8703 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at COLM 2024
A Details of General Capability
Capability Dimension
Definition
Difficulty Level Definition
Example Question
Simple: Questions straightforward and easy to understand.
Intermediate: Questions have higher level of difficulty, such as involving metaphorical mesnings..
Simpic: Introduce Shenzhen.
Intermediate: What are the two clouds in physies?
Semantic Understanding
The capabiiit to correctly under- stand content of the question and Docalble possitle proverbs and idioms,
Difficult: What does "Laoyang Paper Expen- mean?
Questions involve Chinese cultural context.
Simple: Referring to previous round swers or questions.
Interndull:: Referring to answers or questions from the last rounds.
Contextual Conversation
The capability of memory com- textual connection in multi-round con- versations
NA
Diffiuut:: Repeating questions deviating from the topic before returning to
Simple: Input Chinese. intermadizte:
input English and Traditional Chinese. Diffcedt:: Inputs containing multiple ques- tions.
Simpie: Intreduce yourself
Response Completeness and Coherence
Consistency in lan- guage, garbled characters, terruption, peated answers and complete response
Intermediatt: The main character is called Mike. Please write a short story.
Difficut:: [Three different questions]
Simple: Can obtained through information retrieval, mainly commen sense questions. Intermoliatle: Requires multiple retrieval als or large amount of information. Diffcaatt; Requires analyzing lot of informa- fiom and making judgments.
Simple: Which side emperors of Ming Dy- Internediati: nasty and hriefly introduce their main achieve- ments and failures.
Factuality
Contradiction to ab- jective facts hal- lucination
Difficutt: Which company had better revenue growth year,
Google?
Simple: High degree of freedom in eveation.
Simpic Write poem.
intermediate: Creation with 1-2 constraints. Difficalt: Creation with least constraints.
Intermadinte: Write poem about spring. Diffcadt: Write seven speing.
Creativity
Generate text that required style, qua- trains, poems and slogans
seven-character character poem about
Simple: Elementary simple arithmetic..
internidiahe: Involve junior high school mathematics.
Simpic: 3, what
Intermediate: The chicken and rabbit in the same cage problem. Diffcctt:: Permutations and combinations.
Logical Reasoning
Mathematical prob- lemns and logical lations
Diffcuut: Involve high school mathematics or identify wrung/ unanuweroble questions.
Table Definition, difficulty level and examples questions of general capability
12

---

Published as a conference paper at COLM 2024
B Details of Logistics Domain Capability
Capabilly Dimension
Sulsdmmession
Delisition
Diliialty Level Deliiilon
Example Question
Single Accescibie through coonmon querees
Define the basic wonoepts and ter- mimalogy the logtsbes domain. Foess vari- ous sub-domains the logistic indus-
dusa in Ew logatic domain
prmmanly movoling basic conceptual ques-
Sinple What is refrigerated track? / What is intrecity frright trapprttttion? What and the main types of vehicles used in intracity freight transpoctation Ietermadiate: What are for standards dillerent types veticies in intracit lrsight transportiimm
Interwadale:: Requires multiple indormacion retrievals, the use of more specialized (be- yond simple internet wanches) informatiom setrieral channels, involves large amount informatioon
Conceptual and Tomainological
spectalized questions.
dill queryinn wethedss sach compurative cummarization and anal- yais of meults.
Diffcull: Necessiittes numerouus queries spu-
information,
fenences between tracking tochnologges traciy freight traneportaiionand the courser industry?
What
similarities dif-
Siuple Accescible through commmnn queries, petmurrlyy invalcing bask facts about the com- pany with minimal anforatiin requised. Iuterwedatt:: Requires muiltiple searches, or the use mone specialized (bwyond simple internet wearchee) infoemation retrinval chen- nels, swolves significatt amount
Simple IPO date Company A. Indorwedlaiic revenue composittion pany in 2023.
Company Informa- tion
Answer questions related to major companiis upstream and downstreom seciors).
Dijicall, Necssalttts of information, querying methods, requines prerssing such comparative timn and mulyyss of noults.
seacchs and
Diffioule comparison Company and 2023.
growth
substantial
volves
Simple Accessible through comnmon queries, involving introductions to laks and regula- dions, such as the main application aroas of these lanos regulations
Siuple: What are freight-reiated laws China?
Legal and Pollcc Knowledge
Answer questions related regalation and
Iuternddutt: Requires multiple information searches combinations, involves spe- citic lega) previsions. of specific and aogulations wolld
Intermediate:: What are the track restriction rules Shanshess?
Difliwit: Caven la speciticc toal scanariol. you
Diffcaii:: The
conneponding legal pro-
repolations
indua-
Simple Accessible through common mainly conceening basic indusity knowladge and Intormation.
Simple Which are the londing companies the courter industry?
Intermedaate: Roquires multiple information retrievals, ur the use of more specielized fnot just simple interned wainchs) tion retneval chamnels, or entails significatt amount information, or movoves some level analysis.
s informa-
Ieterassfuti: What the market size distribu- toon of the leacing companies in the courier industry?
Indusity Insights
Answer quastions about the macro strachum, curnent status, and develop- ment logis- inxlustr.
Dificail What are future decelopement pronpeets of Company A the courter indas- try?
Dificad: Based available indormation, receescheees combired
funecasting, similar swthods ausuine.
Simple Accesaible through common queries, mainly involving overall knowledge the company without dalving into specitic busi- ness details.
Intormsllate: Requires multiple information wandhas, or the use of more specidized (bu- yond simple internet scarches) information retrieval charmels, or involves large amount information.
Simple When woas Huolela founded??
Intermnaiatte What was Huslale's sa's revenue in 2022 Diffrait: What are the future development prospects of Huolals's service line
Answer quistions related to Hoclals.
more spectallzed methods, and requites combined amelytical resseening
Dificait: Information retrioval
Simple Accesaible through common primarlly involving basic business guestions. Intermedate:: Involves multiple indormation retrievals, mainly for more detailed break- down and understanding of business ques- timn.
Simple What of vebii los available muving home in Stamaham? Intermediate: What types of costs are sesoci- ated with movinng home in Shenchen? Difiianit What type of vahick is suiiable for a typiccl 3 s-persoo family to use for moving home??
From the user's is per- spectiee answer questices related to Ilsoiala's services posed by usem.
Dificnl: Requires a coriain level of indervtuar and pudpment in combirstionn with busiress knowledge.
Sinple Accessible through common queries, Interwsdalt::
sutriovals, mainly multiple quastionns. down and undtenstandnn basiness ques- times
involving
Simgle Whitt stups for cept job? Indewwdiate What are important conei-- erations for diiivers during the aeceptance pencoss?
detailed break-
From the drtver's perspactive swar quastions
Knowchadge about Huolala
Dyfccat: Requites a cerialin level of indorence and judgment in combiration with business krowlsigg..
Factuality
reloted to Finciala's services posed by drivers.
Diffcautt: Llow can driver reccive mare jobs?
Simple We need to send text message to users about coupon. Please help design a script.
Simpie notification seripts. Juteraadistc Marketing advertising pro- copy incolves cnativitty Difkcall for nowenrh, scenanies.
Intermediate: Saptember season, company's 's help design promitional script for potential
home-mmoving
Crnaivity
Conative Capahil- Context
Gemenate seripts advectising,
Logistics
merketing and market
and other complex communication
Diffiel:: We need to conduct surveys through phone calls with drivers who have stopped ing the plation, plens halp design script make the drivers more willing to participato.
research
Table 6: Definition, difficulty level and examples questions of logistics domain capability
13

---

Published as a conference paper at COLM 2024
C
Grading Principle
C.1 General Grading Principle
Factual Questions:
If the response contains incorrect information, grade 0.
If the response is correct but less complete than the standard answer, grade
If the response is fully consistent with the standard answer's points, grade
If the response is fully consistent with the standard answer's s points and provides additional correct information, grade
Open-ended Questions:
If the answer contains incorrect information, grade O.
In the absence of errors, the answer is judged by evaluators based on depth and breadth compared the standard answer to grade or
C.2 Special Consideration
Timeliness of Responses:
If the response does not specify clear time but matches the standard answer, it is considered correct.
If the response does not specify clear time and does not match the standard answer, it is considered a vague answer and judged incorrect.
If the response includes explicit time that different from the question's 's time, but the response would be correct from the given time point, it is considered correct.
D Grading Rubrics of General Capability
Capability Dimension
Rubrics
0: Incorrect understanding of the question
Semantic Understanding
1: Correct Understanding of the question
0: No contextual conversation capability Limited contextual conversation capability
Contextual Conversation
2: Good contextual conversation capability
3: Excellent contextual conversation capability
0: Incomplete response or incomsistent language
Response Completeness and Coherence
1: Complete
0: Incorrect information
Correct information incomplete
Factuality
2: Correct information and complete
No consistency with requirements
1: Limited consistency with requirements
Creativity
Complete consistency with requirements limited artistic conception
3: Complete consistency with requirements and stronger artistic conception
Incorrect result
Logical Reasoning
1: Correct result
Table 7: Grading rubrics of general capability evaluation
14

---

Published as a conference paper at COLM 2024
E
Grading Rubrics of Logistics Domain Capability
Capability Dimension
Subdimension
Rubrics
1: Incorrect information
Conceptual and Terminological Understanding
1: Correct information but incomplete
2: Correct information and complete
Incorrect information
Company Information
Correct information but incomplete
information complete
Incorrect information
Factuality
Legal and Policy Knowledge
1: Correct information but incomplete
2: Correct informalion and complete
0: Incorrect information
Industry Insights
1: Correct information but incomplete
2: Correct information and complete
Incorrect information
Company-pecific Knowledge
information incomplete information
1: Limited comsistency with requirements
Creativity
Creative Capability in Logisties Context
2: Complete conaistency with requirements but not in logistics context
3: Complete consistency with requirements and in logistics contest
Table Grading rubrics of logistics domain capability evaluation
F Dispute Analysis and Grade Fluctuation Analysis
F1 Dispute Analysis
1. Evaluator Dispute: ators.
automatically identifies potential low-quality grades and evalu-
(a) Define the grading dispute decision function Fay: This function takes the value 0 or 1. When the іn grader has a dispute over the grade for the 9n model on the kih question (QA pair kq), Fikg 1, otherwise Fikg O. The criterion for determining a dispute can be manual rule. For example: if evaluator i's grade > ((= 0), but all other graders' grades = 0 0), it is defined that grader has a dispute over the QA pair, assigning Fitg as 1, otherwise, it is considered as no dispute, assigned as O.
Fitg K,O
(b) Calculate the dispute level of grade i on dimension j: C(ij)
Q
represents the number of LLMs evaluated. This signifies the proportion of QA pairs that grade has disputed grades within that dimension.
(c) Calculate the overall dispute level of grader i: C(i) - C(( Here
J represents the number of dimensions, w the weight of dimension j. and
Σ| "; = 1. If no particular dimension is of special interest, all dimensions can be equally weighted.
(d) Repeat the previous steps to calculate the dispute level for all graders. (e) List the total dispute level of all graders, dispute levels in different dimensions, and identify graders with high dispute levels. Review the disputed grades of high-dispute graders, and a manual review confirms significant issues with their grades, conduct retraining for the disputed graders and consider invalidating their grading results.
2. Question Dispute: automatically identifies potential low-quality QA pairs.
(a) Define the question dispute decision function Gzy: This function takes the value or 1. When there is a divergence among graders' grades for the gn model on the kt question (QA pair kq), Gky 1, otherwise Gky = 0. The criterion for determining dispute can be manual rule. For example: when half of the graders' grades > 0(= 0), and the other graders' grades = 0(> 0), the QA pair considered to have divergence, assigning Gkg as 1, otherwise it is considered non-dispute, assigned as O.
15

---

Published as a conference paper at COLM 2024
(b) Calculate the dispute level of question k: C(k) 1 (Eq Gkq) + w x
Ei-1 Ei-I Fig where w + w2 1, Fikq is the grading dispute decision function. The reason for including grading dispute that represents a certain level of fluctuation. If grading disputes are not to be considered, can be set to lower weight, or simply let w2 0.
(c)
Calculate the dispute level for all questions: Repeat steps 1-2 for each question.
(d) List the top N disputed questions, manually determine if there are issues with the questions or standard answers. If issues are indeed present, invalidate the question and conduct retraining for the question designers on the standard of question design.
F2 Grade Fluctuation Analysis
If the same LLM shows significant grade changes between two rounds of evaluation, the changes must be attributed and explained.
1. Breakdown of Fluctuation Causes: grading fluctuations can be broken down into 4 reasons (1) changes in the question itself (2) changes in the LLM's response (3) inconsistency in the same evaluator's grading (4) changes in evaluators.
2. Tagging Questions and LLM Responses: Manual tagging is required determine if question has changed; if the question remains the same, then determine if the L.M'' response has changed (if the information in the response consistent, it is considered the same, exact sameness is not required)
3. Single Dimension Grade Breakdown: Based on whether the Q and A of the QA pair are the same and whether the evaluator the same, the fluctuation can be broken down into 6 scenarios shown in Figure 3. Calculating the differences in grades before and after for these 6 scenarios and aggregating them can quantify the contributions of the fluctuation reasons to the overall grade change.
Fluctuation
Evaluator
Attribution
Same Evaluator
Different Evaluator
QA pair
Response
Different Question
Differett Question Evalustor
Different Question Difterent Evalutor
Grading
Same Question, Different Response
Question Difierent Reeponse Same Evalustor
Same Question Different Reeponse Different Evaluator
Same Question, Response
Same Question Same Resporee
Same Question
Same Ressonse Different Evalustor
Same Evaluator
Figure The six scenarios of single dimension grade breakdown
4. Calculating the contribution of the fluctuation reasons to grade changes: Based these 6 scenarios, the accuracy grade of the LLM in certain dimension can be broken down into parts: Accuracy of a certain dimension = W₃1 Wp1 Adg + Wg1 Wp2 Augglp2 + W₁2 Wp1 Aug Wg2 Wp2 Aug + W33 Wp1 Aug + Wa Wp2 Aug where ql, 92, q3 respectively
16

---

Published as a conference paper at COLM 2024
represent three scenarios: different questions, same questions and different responses, and same questions and same answers. pl, p2 respectively represent scenarios where the evaluators are the same where the evaluators are different. Wq1 indicates the proportion of QA pairs where the questions are different from another round of evaluation. Wq2 indicates the proportion of QA pairs where the questions are the same as in another round of evaluation, but the responses are different. Wq3 indicates the proportion of QA pairs where both the question and the responses are the same as in another round of evaluation. Wp1 indicates the proportion of evaluators who are the same as those in another round of evaluation. Wp2 indicates the proportion of evaluators who are different from those in another round of evaluation. Avgglp1 represents the average grade given by the same evaluator for QA pairs with different questions. Avgql p2 represents the average grade given by different evaluators for QA pairs with different questions.
Change in accuracy of a certain dimension = [W 91 W2 * Avg + W * W
Auga1p2 + W'2 * W * Aug + W * W
p2 * Aus + W * p1 Aug + [Wg1 Wp1 Augg1p1 + Wg1 * W2 * Auggip2 + Wip Wpt
W W
43 p2 * AAggaa22
Aug + Wq2 * Wp Augu2p2 + Was * Wp1 * Augg3p1 + Wap Wy2 * Aug,3p2)
Ei_ E (W W Auo Wjj Augqipp)))
qi PJ
Wr, E E - l(W,, pj AuS - W A
+ Wi
Wii A Aug
+ (WiiWyp Aug
{[ - Wpl (Avg Wo A
Augelpt)
.1)
+ (W Wp1 (A0g(2p))
A (2)
+ Pp (Avg 93p1 - A0g |......... (3)
+ (Was p2 93p2
Aug9312)
Eh E W Aug
Wq Wpj Aog giinj)
}.......(4)) (4)
If the three types of proportions of QA pairs (Wq) and the two types of proportions of evaluators (Wp) remain unchanged between two rounds of evaluations, then the above expression can be simplified to: Change in accuracy of a certain dimension
WN1 _ (Apg Aug41p)) )......())
(Wip E Wp1 ( 822pj Aug p .)...... (2)
+ (W₂p wp (Aug'sp1 - A0g13p1)
(3)
(Wys (Avg
Aug
(4) As shown above, changes in grades can
be attributed to: changes in questions (1), changes in responses (2), changes in grades by the same evaluators (3) and changes in evaluators (4).
G Details of Human Evaluator Training
1. Selection of Human Evaluators: Human evaluators should be selected from a can- didate pool with domain expertise. The circumstances may vary among different domains. In our case, they were selected from our existing team of professional annotators who regularly perform domain-specific annotation tasks.
2. Training Methodology:
Training sessions should be structured around the defined rubrics. Example-based training sessions should be conducted where annotators are exposed to diverse examples of responses and instructed on how to apply the evaluation criteria consistently.
3. Trial Evaluations: Initial trial evaluations should be conducted to assess the consis- tency within and among evaluators in their grading decisions. Issues identified during trials should be determined whether to be attributed to few evaluators based on measures described in Appendix F. If ambiguity found in grading rubrics, the expression of rubrics should be refined.
17

---

Published as a conference paper at COLM 2024
4. Evaluation Quality Assurance: For evaluators showing inconsistent performance, targeted retraining sessions were implemented. These sessions focused on rein- forcing the rubrics and providing additional examples to clarify evaluation criteria. Post-retraining, evaluators should go through further trial evaluations to validate their improvement.
5. Deployment Criteria: Evaluators were only deployed into the production setting after achieving predefined threshold of consistency in trial evaluations.
H
Evaluation Results of All Capability Dimensions
Capability
Capability Dimension
GPT-4
Ernie Bot
PLLM3
PLLM2
PLLM1
Domain
Conceptual and Termino-
66.0
80.0
82.5
84.5
85.0
logical Understanding
Domain
Company Information
22.5
85.0
92.0
79.0
98.5
Domain
Legal and Policy Knowl-
38.5
67.5
91.5
81.0
91.5
edge
Domain
Industry Insights
48.0
84.5
88.5
85.5
90.0
Domain
Company-speciffe
19.0
81.5
89.0
77.0
86.5
Knowledge
Domain
Creative Capability in Lo-
94.7
92.0
40.0
40.0
40.0
gistics Context
General
Semantic Understanding
98.0
92.0
90.0
82.0
88.0
General
Contextual Conversation
68.0
75.0
20.0
24.0
32.0
General
Answer Completeness
64.0
90.0
54.0
56.0
54.0
and Coherence
General
Factuality
86.0
98.0
77.0
76.0
74.0
General
Creativity
70.0
81.3
55.3
60.7
65.3
General
Logical Reasoning
76.0
86.0
60.0
56.0
68.0
Table 9: The normalized average grades of evaluated LLMs by all capability dimensions
Capability
Capability Dimension
Number of Annotators
Ratio of Disagreement
Domain
Conceptual and Termino-
27.0%
logical Understanding
Domain
Company Information
5
17.0%
Domain
Legal and Policy Knowl-
5
33.0%
edge
Domain
Industry Insights
5
24.0%
Domain
Company-specific
5
17.0%
Knowledge
Domain
Creative Capability in Lo-
5
80.0%
gistics Context
General
Semantic Understanding
5
22.0%
General
Contextual Conversation
22.0%
General
Answer Completeness
5
10.0%
and Coherence
General
Factuality
5
18.0%
General
Creativity
5
40.0%
General
Logical Reasoning
4.0%
Table 10: The ratio of disagreement of evaluated LLMs by all capability dimensions
18


Reviews:
[{"review_id": "OX3dUImplo", "reviewer": "Reviewer_wsJT", "summary": "This paper proposes LalaEval, an human-centered evaluation framework for domain-specific large language models (LLMs). The core principles behind the proposed work consist of five components: domain specification, criteria establishment, benchmark creation, evaluation rubric construction and results interpretation. This work combines the proposed principles and presents a general deployable framework. The authors also present the evaluation results of 3 proprietary models fine tuned on top of ChatGLM along with Ernie Bot, GPT-4. Overall this paper is more of an industry-track position piece than a pure methodological one.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have provided brief point-by-point responses below and will include any additional necessary details in the camera-ready version.\n# Resource Demand\nThe domain-specific LLM deployment typically has large real-world influence (e.g. chatbot serving millions of consumers) and has significant impact on firm operation, so it is highly crucial to evaluate and benchmark domain-specific LLMs. In these widespread high-stake circumstances, the resources required are affordable compared to the potential benefits of ensuring best practice.\n# More Details on Evaluation\n1. In our case, the human evaluators were selected from our existing professional annotator team who regularly handle domain-specific annotation tasks.\n2. Our training was based on our framework's rubrics and example-based training sessions. Besides, we conducted trial evaluations to identify evaluator variability. If the variation was attributed to few evaluators, we would retrain them with rubrics and more examples and then reran trial evaluations to form a local loop. Only after reaching an acceptable threshold, they were deployed into the production setting.\n3. We included solving variability in Appendix F in our submission. We standardize the disagreement and fluctuation and attribute it to model, data, intra-evaluators or inter-evaluators. These quantitative measures of reliability were integrated into the loop of dataset construction, model development, training and interpretation.\n# Detailed Evaluation Results\nWe present the results of 2 example out of 12 dimensions here due to character limit and provide complete results in further comments.\n| Capability Dimension                        | GPT4 | Ernie Bot | PLLM3 | PLLM2 | PLLM1 |\n|-|:-:|:-:|:-:|:-:|:-:|\n| Domain: Conceptual and Terminological Understanding | 66.0 |    80.0   |  82.5 |  84.5 |  85.0 |\n| General: Semantic Understanding                      | 98.0 |    92.0   |  90.0 |  82.0 |  88.0 |\n# Minor Suggestion\nWe will fix them. Our request to open-source is currently going through the internal approval process.\n# Lower Scores of GPT-4 and Ernie Bot\nFrom the detailed results, the lower scores of GPT-4 and Ernie Bot are likely due to the absence of proprietary information used to fine-tune the PLLMs.\n# Handling Subjectivity and Variability\nPlease refer to rebuttal 2(3) and the details are included in Appendix F. We will also include the results of reliability measures in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work is well motivated and I agree with the authors that LLMs, especially in domain-sensitive scenarios, need to have better testing protocols. This paper presents a conceptual framework that allows systematic integration of human judgment in evaluating LLM’s in-domain capabilities. From my perspective, the overall framework seems to be well structured and can be standardized in real-world applications.", "reasons_to_reject": "From my understanding, LalaEval seems to demand significant resources, including time, trained personnel, and high-quality domain-specific data. Despite the nature of this piece being a positional one, I feel that this concept will be more complete if the authors can provide more details on the evaluations. Specifically, slightly more information on the process of selecting human evaluators, the training they receive, and how their evaluations are standardized would be beneficial.\n\nAdditionally, as the authors have outlined in the evaluation framework constructions, there are multiple scoring stages. It would be beneficial to include them in the main evaluation for a more detailed per-model evaluation.\n\nMinor suggestions: \n- Try increasing the font size of figure 1 or otherwise the figure can be difficult to read. \n- Be careful about the use of \\citep and \\citet (e.g. first sentence of Related Work)\n- When describing framework specifications, avoid using specific examples as the bullet points (e.g., change \"Specific Knowledge about \n  Huolala\" to \"Company-specific Knowledge\" in 3.3.2).\n- If possible, consider open-sourcing a prototype.", "questions_to_authors": "For the main evaluation, should I believe that the lower scores of GPT-4 and Ernie Bot are due to the fact they do not have access to potentially proprietary information that is used to fine-tune the in-house ChatGLM? Additionally, how does LalaEval handle the potential subjectivity and variability in human evaluations, and what measures are in place to ensure the consistency and reliability of these evaluations?", "extra_scores": {}}, {"review_id": "Dmppwws1Qd", "reviewer": "Reviewer_dWK1", "summary": "The paper introduces LALAEval, a comprehensive human evaluation framework designed to assess LLMs. The framework incorporates a diverse set of evaluation tasks and metrics to provide a holistic understanding of LLM's performance, addressing the limitations of existing evaluation methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Discussion on Potential Biases or Limitations\nWe will include more details in the camera-ready version. Briefly, LalaEval faces potential limitations such as evaluator subjectivity, data selection bias, dynamic domain knowledge changes, and scalability issues. To mitigate these, we propose implementing standardized training programs with regular assessments for evaluators. Ensuring a representative benchmark dataset can reduce selection bias, while continuous updates will keep the framework current with industry changes. Additionally, supplementing automated methods and support systems can streamline the evaluation process and enhance scalability and reliability.\n# Adaptability of LalaEval to More Domains\nLalaEval is designed as a conceptual framework to evaluate domain-specific LLMs, implementing standardized end-to-end protocols of human evaluation which are disentangled with any specific domain.  We reported only logistics domain in submission because we developed and finalized LalaEval when exploring standardized human evaluation framework in this domain. Currently, we do successfully apply LalaEval now in other domains such as HR/IT maintanance/text-to-SQL/telemarketing and bring business value. We therefore believe those who are interested in employing LalaEval to evaluate other domain-specific LLMs by human can easily implement it.\n# Novelty of LalaEval\nLalaEval is the first holistic human evaluation conceptual framework for domain-specific LLMs, addressing the lack of standardized protocols in existing literature. \nGiven the open-generation nature of LLM output and the high stakes of large-scale real-world domain-specific LLM deployments, LalaEval aims to reduce overall human-in-the-loop evaluation subjectivity, to ensure objective and aligned evaluation results and/or further guide our LLM development. Existing literature primarily mentions human evaluation of specific tasks without detailing implementation, standardization or even protocols (See Section 5.2 of A Survey on Evaluation of Large Language Models, arXiv:2307.03109 for a detailed review). LalaEval fills this gap by proposing protocols for domain specification, criteria/dataset/rubrics construction, result analysis including reliability measurement and a systematic integration of these components.\nFurthermore, our deployed framework in a real-world production setting demonstrates its practical effectiveness, contributing valuable LLM-related industry (esp. non-tech) insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The framework encompasses a diverse range of evaluation tasks and metrics, allowing for a nuanced assessment of LLMs' performance.\n2. The empirical results showcase the efficacy of LALAEval in capturing various aspects of LALMs' behavior and facilitating fair model comparisons.", "reasons_to_reject": "The discussion on potential biases or limitations of the proposed framework, as well as strategies to mitigate them, could be more comprehensive.", "questions_to_authors": "1. I was questioned about the adapability of this framework to various domain applications. \n2. what is the novelty of this proposed framework?", "extra_scores": {}}, {"review_id": "6DzmXG7iw7", "reviewer": "Reviewer_uk9M", "summary": "This paper introduce a holistic framework for the human evaluation of domain specific large language models. This framework have five main components: domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, analysis and interpretation of evaluation results.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Contribution and Comparison to Previous Human Evaluation Paradigms\nLalaEval is the first holistic human evaluation conceptual framework for domain-specific LLMs, addressing the lack of standardized protocols in existing literature. \nGiven the open-generation nature of LLM output and the high stakes of large-scale real-world domain-specific LLM deployments, LalaEval aims to reduce overall human-in-the-loop evaluation subjectivity, to ensure objective and aligned evaluation results and/or further guide our LLM development. Existing literature primarily mentions human evaluation of specific tasks without detailing implementation, standardization or even protocols (See Section 5.2 of A Survey on Evaluation of Large Language Models, arXiv:2307.03109 for a detailed review). LalaEval fills this gap by proposing protocols for domain specification, criteria/dataset/rubrics construction, result analysis including reliability measurement and a systematic integration of these components.\nFurthermore, our deployed framework in a real-world production setting demonstrates its practical effectiveness, contributing valuable LLM-related industry (esp. non-tech) insights.\n# Importance of Human Evaluation\nThe domain-specific LLM deployment typically has high-stake real-world influence (e.g. chatbot serving millions of consumers) and the LLM response has open-generation nature, so it is highly crucial to involve human evaluation in the loop to ensure alignment and tailored to domain needs among other goals. Although automated methods may serve as an important supplement to the human evaluation, it is beyond the scope of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe evaluation framework for domain specific human evaluation of LLMs is beneficial to the community.", "reasons_to_reject": "1.\tCertain method details are missed. For example, it is hard to tell how to construct a hierarchical domain specification for the logistics industry in Figure 1. The authors mentioned about the methods of the principle of mutual exclusivity, qualitative prioritization and so on, yet these methods are not well described and would be hard to understand if the readers do not have related background.\n\n2.\tIt would be better if the authors could release the dataset and evaluation code they’ve created in this work.\n\n3.\tThere could be more analysis to the evaluation results. For example, GPT4 underperform all the compared language models in all the results. How could we explain this phenomenon and is there any reasons that lead this result?", "questions_to_authors": "1.\tHow could we evaluate if the proposed LalaEval is working better than other evaluation metrics or frameworks?\n\n2.\tThe font in the figures is a little small and it needs to be set larger.", "extra_scores": {}}, {"review_id": "3CIXwCm3cU", "reviewer": "Reviewer_Mk4S", "summary": "This paper introduces LalaEval, a holistic framework for conducting standardized human evaluations of domain-specific large language models (LLMs). The framework consists of five main components: domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, and analysis and interpretation of evaluation outcomes. The paper demonstrates the framework's application within the logistics industry, presenting domain-specific evaluation benchmarks, datasets, and a comparative analysis of LLMs. The results highlight the framework's effectiveness in elucidating performance differences and guiding model selection and development for domain-specific LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have provided brief point-by-point responses below and will include any additional necessary details in the camera-ready version.\n# Generalizability to Other Domains\nLalaEval is designed as a conceptual framework to evaluate domain-specific LLMs, implementing standardized end-to-end protocols of human evaluation which are disentangled with any specific domain.  We reported only logistics domain in submission because we developed and finalized LalaEval when exploring standardized human evaluation framework in this domain. Currently, we do successfully apply LalaEval now in other domains such as HR/IT maintanance/text-to-SQL/telemarketing and bring business value. We therefore believe those who are interested in employing LalaEval to evaluate other domain-specific LLMs by human can easily implement it.\n# Contribution and Comparison to Previous Human Evaluation Paradigms\nLalaEval is the first holistic human evaluation conceptual framework for domain-specific LLMs, addressing the lack of standardized protocols in existing literature. \nGiven the open-generation nature of LLM output and the high stakes of large-scale real-world domain-specific LLM deployments, LalaEval aims to reduce overall human-in-the-loop evaluation subjectivity, to ensure objective and aligned evaluation results and/or further guide our LLM development. Existing literature primarily mentions human evaluation of specific tasks without detailing implementation, standardization or even protocols (See Section 5.2 of A Survey on Evaluation of Large Language Models, arXiv:2307.03109 for a detailed review). LalaEval fills this gap by proposing protocols for domain specification, criteria/dataset/rubrics construction, result analysis including reliability measurement and a systematic integration of these components.\nFurthermore, our deployed framework in a real-world production setting demonstrates its practical effectiveness, contributing valuable LLM-related industry (esp. non-tech) insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a crucial research gap by providing a systematic methodology for conducting standardized human evaluations within specific domains, which is essential for the practical application of LLMs in real-world business settings.\n- The five-component framework is comprehensive and well-structured, covering all essential aspects of domain-specific LLM evaluation, from domain specification to the analysis and interpretation of evaluation results.\n- The case study of the logistics industry demonstrates the framework's practical utility and its ability to generate valuable insights for businesses looking to integrate LLMs into their operations.", "reasons_to_reject": "- The case study focuses on the logistics industry, which may limit the generalizability of the findings to other domains. Additional case studies across various industries would strengthen the paper's claims about the framework's broad applicability.\n- The paper does not provide a detailed comparison of the LalaEval framework to existing evaluation methodologies, making it difficult to assess its relative advantages or disadvantages.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xvd1S6q832", "reviewer": "Reviewer_qpET", "summary": "1. This paper proposes LalaEval, a framework designed to standardized and systematic human evaluation for domain-specific LLMs. The framework consists of 5 components including domain specification, criteria setting, benchmark dataset creation, construction of evaluation rubrics, and analysis of evaluation outcomes.\n\n2. The framework’s efficacy is demonstrated through a case study in the logistics industry, showing how it helps elucidate performance differences among LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have provided brief responses and will include any additional necessary details in camera-ready version.\n# Validation of More Models\nWe present the overall grades for 6 additional models, 4 foundation and 2 fine-tuned.\n|| GPT-3.5|ChatGLM-6B|Vicuna-13B|Qwen-7B|Fine-tuned Qwen-7B|Fine-tuned Baichuan-13B|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Overall|46.70|33.18|20.22|65.40|71.62|57.11|\n# Validation of More Domains\nLalaEval is designed as a conceptual framework to evaluate domain-specific LLMs, implementing standardized end-to-end protocols of human evaluation, disentangled with any specific domain. We reported only logistics domain because we finalized LalaEval when exploring standardized human evaluation framework in this domain. Currently, we do successfully apply LalaEval now in other domains such as HR/IT maintanance/text-to-SQL/telemarketing and bring business value. We therefore believe those who are interested in employing LalaEval can easily implement it.\n# Intermediate Steps\nWe present the results of 2 example out of 12 dimensions here due to character limit and provide complete results in further comments.\n| Capability Dimension|GPT4|Ernie Bot|PLLM3|PLLM2|PLLM1|\n|-|:-:|:-:|:-:|:-:|:-:|\n| Domain: Conceptual and Terminological Understanding|66.0|80.0|82.5|84.5|85.0|\n| General: Semantic Understanding|98.0|92.0|90.0|82.0|88.0|\n# Human Evaluation Details\nWe had 5 annotators and the overall disagreement ratio is 23.8%. More details will be provided in further comments.\n# Contribution and Comparison to Previous Paradigms\nLalaEval is the first holistic human evaluation conceptual framework for domain-specific LLMs, addressing the lack of standardized protocols in existing literature. \nGiven the open-generation nature of LLM and the high stakes of large-scale real-world domain-specific LLM deployments, LalaEval aims to reduce overall human-in-the-loop evaluation subjectivity, to ensure objective and aligned evaluation results and/or further guide the LLM development. Existing literature primarily mentions human evaluation of specific tasks without detailing implementation, standardization or even protocols (See Section 5.2 of arXiv:2307.03109). LalaEval fills this gap by proposing protocols for all five components and a systematic integration.\nFurthermore, our deployed framework in a real-world production setting demonstrates its practical effectiveness, contributing valuable LLM-related industry (esp. non-tech) insights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Constructing a standardized framework for human evaluation processes is an intriguing idea. Relevant research can help formalize the assessment procedures in the field of large language models, leading to more reliable evaluation outcomes.\n\n2. This study provides a detailed description of each module of LalaEval, comprehensively considering  the processes involved in the human evaluation procedure.", "reasons_to_reject": "1. My major concern is that the experiments in this article struggle to demonstrate the advantages and effectiveness of the proposed evaluation framework. On one hand, the author conducted only a case study involving three models, lacking validation across a broader range of domains and models. Additionally, the experimental results presented do not clarify how LalaEval surpasses previous human evaluation paradigms. On the other hand, the author only displayed the final evaluation results and conclusions, lacking presentation and validation of intermediate steps. There is also a lack of detailed description of the human evaluation details, such as the number of evaluators involved, inter-annotator agreement, and so forth.\n\n2. As an academic paper, I think that the framework proposed by this study does not present any fundamental innovation compared to the commonly used human evaluation processes previously established.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper outlines a general framework for designing evaluations for domain-specific LLMs. It appears to be an industry paper, which is grounded in practical concerns about deploying LLMs for real-world use cases. As such, it does not follow the “typical” style of an academic paper, and this is a cause for some reviewer concerns. In particular, reviewers raise concerns about a lack of clear evaluation/baselines for existing approaches, and about the apparent cost of carrying out the proposed framework. The authors’ response is that no clear baselines exist because industry standards do not yet exist, and rather each case is handled in an ad hoc manner. This ad hoc nature of evaluation is the problem the paper attempts to address.

I believe that this paper would be a useful addition to COLM precisely because of its atypical nature. The reviewers’ concerns about evaluation are understandable, but given the current state of LLM research, there is likely more to be gained by exposing differing approaches and perspectives (e.g., from academia and industry, from quantitative and qualitative sciences) than prematurely imposing a specific recipe for research. This paper has value in outlining the key challenges that are faced by those applying LLMs in industry settings, and showcasing the way that these challenges are currently addressed. Such work could provide inspiration or fodder for subsequent, more rigorous, research studies.

[At least one review was discounted during the decision process due to quality]
Decision: Accept