LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Models

Authors: chongyan  sun
ken  lin
shiwei  wang
hulong  wu
chengfei  fu
zhen  wang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 16:18:57

Abstract: This paper introduces LalaEval, a holistic framework designed for the human evaluation of domain-specific large language models (LLMs). LalaEval proposes a comprehensive suite of end-to-end protocols that cover five main components including domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, and thorough analysis and interpretation of evaluation outcomes. This initiative aims to fill a crucial research gap by providing a systematic methodology for conducting standardized human evaluations within specific domains, a practice that, despite its widespread application, lacks substantial coverage in the literature and human evaluation are often criticized to be less reliable due to subjective factors, so standardized procedures adapted to the nuanced requirements of specific domains or even individual organizations are in great need. Furthermore, the paper demonstrates the framework's application within the logistics industry and a comparative analysis of LLMs for the logistics domain use, highlighting the framework's capacity to elucidate performance differences and guide model selection and development for domain-specific LLMs. Through real-world deployment, the paper underscores the framework's effectiveness in advancing the field of domain-specific LLM evaluation, thereby contributing significantly to the ongoing discussion on LLMs' practical utility and performance in domain-specific applications.

Main paper: 
 introduction: 
 The rise of large language models (LLMs) represents a significant step towards artificial general intelligence (Bubeck et al., 2023) , showcasing their powerful ability to understand and produce natural language. Although these models have generally been designed for wide-ranging use, one of their most promising applications lies within specific domains such as medicine (Lee et al., 2023; Wang et al., 2023a; Thirunavukarasu et al., 2023) , law (Cui et al., 2023) and finance (Li et al., 2023; Wu et al., 2023) . For businesses looking to integrate LLMs into their operations, the focus naturally shifts towards models' capabilities in certain industries, as conversations within a particular industry are more prevalent and relevant than broad conversations when in real-world business setting (Guo & Yu, 2022; Zhao et al., 2023) .The process of evaluating these domain-specific models is crucial. Typically, evaluations range from automatic techniques, e.g. Zheng et al. (2024) , to human evaluation, with the latter widely regarded as the most comprehensive method. Human evaluation's importance lies in its unparalleled ability to grasp the intricacies of language and context, aspects that automatic methods might miss (Novikova et al., 2017) , and more importantly in commercial use, human evaluation also reflects true human preference of businesses' stakeholders, so when developing and comparing LLMs, particularly those meant for domain-specific real-world business use, human evaluation becomes essentially gold standard. It plays a critical role in measuring how these models perform for commercial use and guides the development process, ensuring the models meet real-world interaction standards.Given the importance of human evaluation (Chang et al., 2023) , creating a standardized framework for human evaluation is vital. This framework, which includes detailed methods for gathering data, setting evaluation rubrics, defining metrics as well as establishing respective evaluation protocols, is essential for ensuring evaluations are consistent, accurate and relevant. It specifically addresses the need for domain-specific LLMs to be closely aligned with actual business requirements, filling the gap between theoretical capabilities and practical application. By focusing on the case study of the logistics domain, our study introduces a detailed evaluation framework designed for a specific domain. This framework is not just an example of the evaluation within this industry, it also showcases a standardized human evaluation framework of domain-specific LLMs in general.This research makes a meaningful contribution to the discussion on evaluating LLMs, especially for domain-specific uses. By outlining the creation of a human evaluation framework that matches commercial needs, we set a path for a more focused, efficient, and practical use of LLMs in business contexts. Our deployed framework in the logistics domain demonstrates the framework's value to improve the relevance, functionality, and adoption of LLMs in specific domains.2 Related Work Chang et al. (2023) presents an extensive survey on the evaluation of LLMs, providing a detailed taxonomy of evaluation aspects, including what, where, and how to evaluate. General-purpose models, such as OpenAI's GPT and Google's Gemini, typically do not serve downstream tasks directly. Thus, their evaluation often involves a multifaceted approach (Bang et al., 2023) , incorporating a variety of tasks, as demonstrated by Bai et al. (2024) ; Bian et al. (2023) ; Liang et al. (2022) .In contrast, the most economically potent applications for commercial, domain-specific use are arguably found in customer operations, i.e. service chatbots (Chui et al., 2023) . Accordingly, this paper concentrates on developing an evaluation framework for domainspecific LLMs, particularly for chatbot conversations. Although initially designed for singleround conversations, this framework can be readily extended to multi-round interactions.There is a significant body of work addressing where to evaluate. Numerous benchmarks and datasets have been established to test the capabilities of LLMs (Hendrycks et al., 2020; Xu et al., 2023; Gu et al., 2023) . These efforts provide essential benchmarking datasets and results for mainstream LLMs, facilitating a unified comparison. However, these benchmarks are often not directly applicable to specific company uses due to differences in domains, knowledge, applications, etc. This paper distinguishes itself by offering frameworks and protocols to guide the construction of benchmarks and datasets within commercial organizations for specific domains.Another stream of literature focuses on how to evaluate. Many benchmarks adopt automatic evaluation methods to obtain results, ranging from traditional metrics (Chin-Yew, 2004 ) to more LLM-based judgments (Wang et al., 2023b) . The advantages of automatic evaluation over human evaluation are clear: it is more standardized and objective, easier to scale, and less costly. However, for non-standard tasks like open generation, human evaluation proves to be more reliable and better aligned with general human preferences (Novikova et al., 2017) . Moreover, evaluating domain-specific LLMs often requires profound domain knowledge and even internal organizational knowledge. The human evaluation is typically criticized by its cost and susceptibility to subjective bias, but within the context of commercial use of domain-specific LLMs, a company can often afford the expense of human evaluators and organize effective training. Moreover, domain-specific LLMs typically have large realworld influence (e.g. chatbot serving millions of consumers) and have significant impact on firm operation. In these widespread high-stake circumstances, the resources required are reasonable compared to the potential benefits of ensuring best practice. Nevertheless, there is very little literature offering a holistic framework to systematically guide the implementation of human evaluation from domain specification and evaluation dataset construction to evaluation rubrics, metrics, and reporting of results. This paper aims to bridge this gap. 
overview: 
 LalaEval encompasses five main components to establish comprehensive protocols guiding the evaluation process of LLMs: (1) Domain Specification, (2) Criteria Establishment, (3) Benchmark Dataset Creation, (4) Construction of Evaluation Rubrics, and (5) Analysis and Interpretation of Evaluation Results. In the conceptualization of LalaEval, the first step entails defining the domain that is usually inherently derived from the industries within which an organization operates. The breadth of such an industry definition, however, remains an area for exploration. For instance, should the evaluation scope for a medical LLM include only radiology or extend to the broader field of medicine? Should the focus for a financial LLM be confined to bonds, or should it cover the wider realm of capital markets?We propose a hierarchical structuring of subdomains for LalaEval, which organizes these subdomains by their relative significance, ranging from narrow to broad scopes. Employing backward induction, we begin with a highly specific subdomain pertinent to the organization, progressing towards a more generalized industry definition. By adhering to the principle of mutual exclusivity and collective exhaustiveness, we enumerate the most granular subdomains and their parallel subdomains, progressively ascending to encompass broader subdomains. This process is iterated multiple times until a sufficiently broad domain scope is achieved.Subsequently, we employ qualitative prioritization to define both the scope and the hierarchical significance of each subdomain. This prioritization may adopt either a linear progression from the most specific to the most broad domain or a tree-like structure that includes parallel subdomains at various levels, contingent upon business imperatives.Figure 1 presents our case by hierarchical domain specification in the logistics industry, with the final chosen subdomains highlighted. We establish both the scope and priorities for subdomains, ranking them from high (P0) to low (P3) priority: P0 (Intracity Freight This hierarchical specification of subdomains and their prioritization not only informs but also directs the subsequent components of LalaEval.The objective of establishing criteria for general capabilities is to evaluate the performance of LLMs a wide array of natural language tasks not confined to any specific domain. This includes their capability in comprehending and generating natural language, recognizing contextual cues, sustaining coherence throughout conversations, and processing and conveying information with accuracy.The rationale for evaluating such general capabilities includes (1) Foundation for domain capability: A solid foundation in general capability is crucial for the performance of any LLM, providing the groundwork upon which domain capability is constructed. A lack in general language capability can impede LLMs' capability to understand or generate responses accurately within specialized domains. (2) Flexibility and Adaptability: LLMs that exhibit strong general language capabilities demonstrate enhanced flexibility and adaptability. This is vital for applications requiring comprehension of inputs from diverse domains or the integration of new knowledge without the need for extensive retraining.(3) Understanding and Reasoning: Evaluating general capabilities aids in identifying LLMs' capacity for understanding complex queries, reasoning, and generating coherent responses that are contextually appropriate. These attributes are indispensable for practical applications.To systematically evaluate these general capabilities, we have adapted from (Xu et al., 2023) and outlined dimensions of general capabilities in the following, and we also show example questions and defined difficulty levels in Appendix A.• Semantic Understanding: Crucial for LLMs is understanding the basic meaning of language, and special terms such as idioms and cultural references, to ensure meaningful user interaction. Evaluation involves questions that test understanding across linguistic occasions.• Contextual Conversation: LLMs need to remember past interactions to maintain coherent conversations, essential for sustained engagement in customer service or conversational applications. Multi-round questions can evaluate this skill.• Answer Completeness and Coherence: Responses must be clear, concise, and address queries directly, ensuring outputs are practical and user-centric, especially in informative or decision-support settings.• Factuality: Accuracy in responses, especially for questions expecting definitive answers, is critical. This is vital in industries like logistics where errors can have significant consequences. Verification against trusted sources evaluates this aspect.• Creativity: The ability to generate creative content, such as marketing material or innovative responses, highlights LLMs' capacity for engaging content creation. This is evaluated through creativity requests with certain constraints.• Logical Reasoning: Tasks requiring numerical or logical deduction evaluate this capability, relevant for problem-solving in industries like logistics. Evaluation includes mathematical or logic puzzles.The domain capability focuses on LLMs' expertise within certain industries, evaluating its understanding of domain-specific terminologies, concepts, regulations, and operational nuances. It also evaluates LLMs' capability to provide insightful, accurate responses to domain-specific queries.The rationale for evaluating domain-specific capability is (1) Domain-Specific Performance:The principal motive is to ascertain the efficacy of LLMs in logistics-specific applications.It is imperative for LLMs to not merely comprehend general language but also to exhibit a profound understanding of specialized domain knowledge and intricacies.(2) Practical Usability: LLMs endowed with strong logistics domain capabilities hold greater practical value for industry professionals and businesses. Such LLMs are capable of providing more accurate and relevant insights, thereby facilitating enhanced decision-making processes and operational efficiency.(3) Customized Solutions: Evaluating this capability allows for the development of more customized, domain-specific solutions that can address specific challenges and needs within the logistics domain, providing a competitive edge to businesses leveraging these LLMs.Focusing on the dimensions of factuality and creativity, we have outlined and illustrated subdimensions of capabilities in logistics domain in the following, and we also show example questions and defined difficulty levels in Appendix B.• Conceptual and Terminological Understanding: Knowledge of specific terms and concepts is fundamental in accurately interpreting and responding to industry-related queries. This necessitates an evaluation that includes questions derived from the logistics domain's lexicon and operational nuances.• Company Information: LLMs should be conversant with key players in the industry and relevant corporate data, reflecting the model's utility as an informative resource. Questions about major companies, their operations, and market positions can evaluate this aspect.• Legal and Policy Knowledge: Given the regulatory environment surrounding logistics operations, LLMs must be adept at navigating legal and policy-related queries. This can be evaluated through questions that require the LLM to reference specific regulations or guidelines applicable to logistics.• Industry Insights: The ability to provide informed opinions or data about the logistics market, trends, and future outlooks showcases LLMs' depth of knowledge and analytical capabilities. Crafting scenarios that ask for analysis or predictions based on current data evaluates this competency.• Company-specific Knowledge: For LLMs deployed by specific companies, such as Huolala, understanding the company's services, history, and strategic vision is crucial. This ensures the LLM can serve as an effective chatbot. Questions tailored to the company's operations and strategy evaluate this knowledge. • Creative Capability in Logistics Context: Beyond generic creativity, the ability to generate content specifically tailored to the logistics industry's stylistic and contextual requirements is valuable for marketing and customer engagement purposes. Questions requiring domain-specific creative responses can evaluate this capability.The primary goal of establishing a high-quality benchmark dataset is to develop an evolving bank of question-answer (QA) pairs for human evaluation. Existing public benchmarks and datasets are not utilized due to (1) the absence of benchmarks tailored to the specific domain;(2) the proprietary benchmark's closer alignment with company-specific needs; and (3) the lack of continuous updates in most public datasets, which fails to meet the timely knowledge requirements for the business application of LLMs. Concurrently, this dataset aims to include a broad spectrum of capabilities outlined in Section 3.3, thereby ensuring a uniform evaluation of LLMs' performance across diverse metrics.The benchmark dataset are developed through the process outlined in following steps:1. Accumulation of the Raw Corpus: Guided by the previous components, we systematically collect and compile original texts and corpora from verified sources. This repository of raw corpus forms the groundwork for generating insightful and pertinent QA pairs. 2. Production of QA Pairs: (1) Development of the Question Plan: Formulate a structured plan outlining the desired number of QA pairs, categorized by various levels of difficulty and capabilities. This plan should be iterative to allow for the continuous enhancement of the dataset. ( 2) Selection of Question Designers: Identify and appoint individuals responsible for question design, ensuring they possess an in-depth understanding of the evaluation framework and access to the raw corpus.(3) Creation of QA Pairs: The question designers will distill relevant information from the raw corpus or other authoritative sources to craft QA pairs. Each pair will include a question and its corresponding standard answer, for instance, "Q: How many hours are there in a day? A: 24 hours." Importantly, the source of information for each QA pair must be documented to maintain traceability and credibility. 3. Quality Inspection and Database Entry: Following the generation of QA pairs, a thorough quality inspection is conducted to verify that the pairs adhere to predefined criteria. QA pairs that pass this inspection are incorporated into the dataset, whereas those failing to meet the standards are redirected back to the designers for refinement.The process of proposing QA pairs for the benchmark dataset is a critical component in the evaluation of LLMs, particularly within specific domains such as the logistics industry. Through a systematic approach to compiling raw corpora, creating relevant QA pairs, and executing rigorous quality inspections, this initiative seeks to establish a comprehensive dataset. This dataset not only lays the foundation for a thorough evaluation of LLMs but also facilitates the continuous improvement and benchmarking of LLMs, thereby ensuring their relevance and applicability in real-world scenarios.The objective of the construction of evaluation rubrics is to systematically evaluate the performance of various LLMs using the benchmark dataset from Section 3.4. This process is designed to guide the training of human evaluators and ensure more consistent outcomes, both intraperson and interperson.The general evaluation scale is 0-3 points. If a response contains any incorrect information, it will get 0 points. 1-3 points measure the degree of correctness, completeness, creativity etc. Meanwhile, special consideration is given to the timeliness of responses, recognizing the importance of current knowledge and the LLMs' ability to reflect the most recent information or anticipate future developments. Appendix C presents the preceding general grading principle and the special consideration about timeliness.The evaluation rubrics for evaluating the general capabilities and logistics domain capabilities of LLMs are comprehensively outlined in Appendices D and E, respectively.After trained human evaluators are ready (see Appendix G for details of human evaluator training process), we randomly draw a subset from the dataset based on agreed difficulty level and quantities in the grading process. To keep integrity, we employ a single-blind procedure, wherein model responses to the QA pairs are anonymized and presented in a randomized order to a panel of at least three human evaluators. This procedure mitigates bias, ensuring that human evaluators cannot infer the origin of the responses, thus maintaining objectivity. Table 1 shows the demo of human evaluators' interface. i AS 1ki • • • AS qki • • • AS Qki Dimension j Question k Evaluator i + 1 AS 1k(i+1) • • • AS qk(i+1) • • • AS Qk(i+1) . . . . . . . . . . . . . . . . . . . . . . . . Dimension J Question K J Evaluator n AS 1(K J )n • • • AS q(K J )n • • • AS Q(K J )nTable 2 : Four-dimensional table of evaluators' grading results. AS qki represents the grade given by evaluator i to the response of the q th model for the k th question.The grade calculation for model q involves single dimension j grade calculation using the formula Grade(qj) =∑ K j k=1 ∑ n i=1 AS qki ∑ K j k=1 ∑ n i=1 TS k, then aggregating across all dimensions to compute the total grade as Grade(q) = ∑ J j=1 w j Grade(qj) . This process evaluates a LLM's capability across various dimensions and overall, by weighing each dimension's grade, equally or otherwise. Reporting includes both the grades for different dimensions and the total grade, facilitating a comprehensive assessment of each LLM's performance. Repeating these steps for all models under evaluation provides a comparative analysis of their strengths and weaknesses.Our methodology also aims to reduce the subjective factors of human evaluators as much as possible, incorporating automated dispute analysis and checks for grade stability and reliability. This phase is crucial for identifying low-quality grades and questions and attributing grading fluctuations to transparent factors, such as evaluator inconsistency or changes in LLM responses. The detailed dispute analysis and grade fluctuation analysis methods are described in Appendix F.After describing the five main conceptual components of LalaEval, we now turn to the practicalities of deployment, presenting an approach that brings our framework into operational reality, ensuring a comprehensive evaluation process for domain-specific LLMs. This integration strategy, designed to foster standardization, as well as efficiency and adaptability, is succinctly illustrated in Figure 2 . Through this streamlined depiction, the operational blueprint that guides the application of LalaEval is presented, facilitating a clear understanding of its modular architecture and the dynamic interactions essential for evaluation. The results presented in Section 4 are also generated using this deployment strategy.  
results: 
 To demonstrate the effectiveness and the practical application of LalaEval, we present the zero-shot evaluation results of various LLMs including OpenAI's GPT-4 (without web access), Baidu's Ernie Bot (with web access), and our proprietary LLM v1, v2, and v3 (PLLM1/2/3). PLLM1/2/3 are iterations fine-tuned from the ChatGLM2-6B foundation, incorporating web access, retrieval-augmented generation, or a combination of both, respectively.Our evaluation is demonstrated in two distinct result sets: (1) The accuracy of the LLMs (Table 3 ), where we present the percentage of responses achieving non-zero grades across different capability dimensions. This metric serves as a direct indicator of each LLM's capability to generate relevant and accurate response. (2) The normalized average grades (Table 4 ) for each LLM across capability dimensions, providing more quantitative view of LLM performance. The complete results of grades and disagreement by all capability dimensions are included in Appendix H. The results reflect various levels of coverage of capability dimension defined in Section 3.3 where Domain-Factuality is defined as the average capability within the domain except creative capability, enabling finer-grained comparison across LLMs. The results only represent evaluation under our LalaEval framework, focusing more on capabilities in the logistics domain. The comparative analysis, as illustrated in Tables 3 and 4 , highlights a landscape of LLM performance across different capabilities. GPT-4 and Ernie Bot exhibit closely matched strengths in general capabilities, significantly outperforming the proprietary LLMs. However, the narrative shifts within the logistics domain, where proprietary LLMs, alongside Ernie Bot, demonstrate superior performance over GPT-4, possibly due to the lack of information used to fine-tune proprietary LLMs. Notably, within this domain, the proprietary LLMs marginally surpass Ernie Bot in factuality, showing their refined capability. These insights serve not only to benchmark the current state of LLMs within the company but also to direct future development efforts. 
analysis and interpretation of evaluation results: 
 : This component involves systematically examining data collected from the evaluation process to minimize intrapersonal, interpersonal, intramodel, and intermodel variability. It aims to derive meaningful insights and guide decision-making, ensuring the constructive application of outcomes.LalaEval may have some potential limitations, including evaluator subjectivity, data selection bias, dynamic changes in domain knowledge, and scalability concerns. To address these challenges and further enhance LalaEval's effectiveness, several avenues for future work can be proposed.• Standardized Training and Assessment: Beyond procedures described in Appendix G, exploring more structured training protocols across domains can further improve the reliability and generalizability of LalaEval. • Enhanced Dataset Representation: Ensuring the benchmark dataset is comprehensive and representative in the domain can minimize biases arising from data selection. This step involves continuous refinement and expansion of dataset sources to cover diverse scenarios. • Adaptability to Dynamic Domains: Given the evolving nature of domain-specific knowledge, continuous updates to evaluation protocols and benchmarks are essential. This adaptive approach helps LalaEval stay relevant and effective in terms of dynamic industry changes. • Automation Integration: Supplementing human evaluation with automated methods and support systems can streamline the evaluation process. Automation can assist in handling large volumes of data and tasks, enhancing scalability while maintaining evaluation reliability.• Robust Support Systems: Developing robust support systems to assist evaluators in interpreting ambiguous cases or complex scenarios can further improve evaluation consistency and reliability. 
conclusion: 
 This study introduces LalaEval, a novel framework for evaluating domain-specific LLMs, with a focus on standardizing human evaluations. By detailing a comprehensive methodology spanning domain specification to results analysis, LalaEval addresses the critical gap in the standardized human evaluations of domain-specific LLMs, exemplified through its application in the logistics domain.LalaEval's deployment showcases its capability to illuminate performance differences among LLMs, guiding model selection and development for domain-specific applications.LalaEval not only advances the field of LLM evaluation by establishing end-to-end human evaluation protocols but also emphasizes the importance of aligning LLM capabilities with practical needs.LalaEval is also a general human evaluation framework which are disentangled with any specific domain. The results of only logistics domain are reported because LalaEval was developed and finalized when exploring standardized human evaluation framework in this domain. LalaEval has been applied to other domains such as HR/IT maintanance/text-to-SQL/telemarketing and brought business value. It is believed those who are interested in employing LalaEval to evaluate other domain-specific LLMs can easily implement it.In summary, LalaEval represents a significant step forward in the evaluation of domainspecific LLMs, offering a structured and human-centric approach to understanding LLM performance. Its contributions lay the groundwork for future research and application of LLMs across various domains, highlighting the evolving need for evaluation methodologies that are as dynamic and specialized as the LLMs they seek to evaluate. Simple: Accessible through common queries, primarily involving basic conceptual questions in the logistics domain. Intermediate: Requires multiple information retrievals, or the use of more specialized (beyond simple internet searches) information retrieval channels, or involves a large amount of information. Difficult: Necessitates numerous queries and a large volume of information, or involves special querying methods, or requires processing such as comparative summarization and analysis of results. 
 
 Appendix: Simple: What is a refrigerated truck? / What is intracity freight transportation? / What are the main types of vehicles used in intracity freight transportation? Intermediate: What are the fee standards for different types of vehicles in intracity freight transportation? Difficult: What are the similarities and differences between tracking technologies in intracity freight transportationand the courier industry?Answer questions related to the major companies in the logistics industry (or its upstream and downstream sectors).Simple: Accessible through common queries, primarily involving basic facts about the company with minimal information required. Intermediate: Requires multiple searches, or the use of more specialized (beyond simple internet searches) information retrieval channels, or involves a significant amount of information. Difficult: Necessitates numerous searches and a substantial amount of information, or involves unique querying methods, or requires processing such as comparative summarization and analysis of results. Answer questions related to market regulation and legal regulations in the logistics industry.Simple: Accessible through common queries, involving introductions to laws and regulations, such as the main application areas of these laws and regulations. Intermediate: Requires multiple information searches and combinations, or involves specific legal provisions. Difficult: The application of specific laws and regulations in real-world scenarios.Simple: What are freight-related laws in China? Intermediate: What are the truck restriction rules in Shenzhen? Difficult: Given [a specific real-world scenario], can you provide the corresponding legal provisions?Answer questions about the macro structure, current status, and development of the logistics industry.Simple: Accessible through common queries, mainly concerning basic industry knowledge and information.Intermediate: Requires multiple information retrievals, or the use of more specialized (not just simple internet searches) information retrieval channels, or entails a significant amount of information, or involves some level of analysis.Difficult: Based on available information, it necessitates combined analysis, forecasting, and similar methods to acquire.Simple: Which are the leading companies in the courier industry? Intermediate: What is the market size distribution of the leading companies in the courier industry?Difficult: What are the future development prospects of Company A in the courier industry?Answer questions related to Huolala.Simple: Accessible through common queries, mainly involving overall knowledge of the company without delving into specific business details.Intermediate: Requires multiple information searches, or the use of more specialized (beyond simple internet searches) information retrieval channels, or involves a large amount of information. Difficult: Information retrieval involves more specialized methods, and requires combined analytical reasoning.Simple: When was Huolala founded? Intermediate: What was Huolala's revenue in 2022?Difficult: What are the future development prospects of Huolala's service line A?From the user's perspective: answer questions related to Huolala's services posed by users.Simple: Accessible through common queries, primarily involving basic business questions.Intermediate: Involves multiple information retrievals, mainly for a more detailed breakdown and understanding of business questions.Difficult: Requires a certain level of inference and judgment in combination with business knowledge.Simple: What types of vehicles are available for moving home in Shenzhen?Intermediate: What types of costs are associated with moving home in Shenzhen? Difficult: What type of vehicle is suitable for a typical 3-person family to use for moving home?From the driver's perspective: answer questions related to Huolala's services posed by drivers.Simple: Accessible through common queries, primarily involving basic business questions. Intermediate: Involves multiple information retrievals, mainly for a more detailed breakdown and understanding of business questions. Difficult: Requires a certain level of inference and judgment in combination with business knowledge.Simple: What are the steps for a driver to accept a job? Intermediate: What are some important considerations for drivers during the job acceptance process? Difficult: How can a driver receive more jobs?Generate scripts for marketing, advertising, promotions, and market research Simple: Basic notification scripts. Intermediate: Marketing, advertising, and promotional copy that involves more creativity. Difficult: Scripts for user research, customer service, and other complex communication scenarios.We need to send a text message to users about a coupon. Please help design a script. Intermediate: September is the company's home-moving discount season, please help design promotional script for potential homemoving customers. Difficult: We need to conduct surveys through phone calls with drivers who have stopped using the platform, please help design a script to make the drivers more willing to participate. C.1 General Grading Principle Factual Questions:• If the response contains incorrect information, grade 0.• If the response is correct but less complete than the standard answer, grade 1.• If the response is fully consistent with the standard answer's points, grade 2.• If the response is fully consistent with the standard answer's points and provides additional correct information, grade 3.Open-ended Questions:• If the answer contains incorrect information, grade 0.• In the absence of errors, the answer is judged by evaluators based on depth and breadth compared to the standard answer to grade 1 or 2.Timeliness of Responses:• If the response does not specify a clear time but matches the standard answer, it is considered correct.• If the response does not specify a clear time and does not match the standard answer, it is considered a vague answer and judged incorrect.• If the response includes explicit time that is different from the question's time, but the response would be correct from the given time point, it is considered correct. (a) Define the grading dispute decision function F ikq : This function takes the value 0 or 1. When the i th grader has a dispute over the grade for the q th model on the k th question (QA pair kq), F ikq = 1, otherwise F ikq = 0. The criterion for determining a dispute can be a manual rule. For example: if evaluator i's grade > 0(= 0), but all other graders' grades = 0(> 0), it is defined that grader i has a dispute over the QA pair, assigning F ikq as 1, otherwise, it is considered as no dispute, assigned as 0.(b) Calculate the dispute level of grade i on dimension j:C(ij) = ∑ K j k=1 ∑ Q q=1 F ikq K j Q . Qrepresents the number of LLMs evaluated. This signifies the proportion of QA pairs that grade i has disputed grades within that dimension.(c) Calculate the overall dispute level of grader i: C(i) = ∑ J j=1 w j C(ij). Here J represents the number of dimensions, w j the weight of dimension j, and ∑ J j=1 w j = 1. If no particular dimension is of special interest, all dimensions can be equally weighted. (d) Repeat the previous steps to calculate the dispute level for all graders. (e) List the total dispute level of all graders, dispute levels in different dimensions, and identify graders with high dispute levels. Review the disputed grades of high-dispute graders, and if a manual review confirms significant issues with their grades, conduct retraining for the disputed graders and consider invalidating their grading results.2. Question Dispute: automatically identifies potential low-quality QA pairs.(a) Define the question dispute decision function G kq : This function takes the value 0 or 1. When there is a divergence among graders' grades for the q th model on the k th question (QA pair kq), G kq = 1, otherwise G kq = 0. The criterion for determining dispute can be a manual rule. For example: when half of the graders' grades > 0(= 0), and the other graders' grades = 0(> 0), the QA pair is considered to have divergence, assigning G kq as 1, otherwise it is considered non-dispute, assigned as 0.(b) Calculate the dispute level of question k: C(k) = w 1 × (∑ Q q=1 G kq ) + w 2 × ∑ n i=1 ∑ Q q=1 F ikq n, where w 1 + w 2 = 1, F ikq is the grading dispute decision function. The reason for including grading dispute is that it represents a certain level of fluctuation. If grading disputes are not to be considered, w 2 can be set to a lower weight, or simply let w 2 = 0. (c) Calculate the dispute level for all questions: Repeat steps 1-2 for each question. (d) List the top N disputed questions, manually determine if there are issues with the questions or standard answers. If issues are indeed present, invalidate the question and conduct retraining for the question designers on the standard of question design.If the same LLM shows significant grade changes between two rounds of evaluation, the changes must be attributed and explained.1. Breakdown of Fluctuation Causes: grading fluctuations can be broken down into 4 reasons (1) changes in the question itself (2) changes in the LLM's response (3) inconsistency in the same evaluator's grading (4) changes in evaluators.2. Tagging Questions and LLM Responses: Manual tagging is required to determine if a question has changed; if the question remains the same, then determine if the LLM's response has changed (if the information in the response is consistent, it is considered the same, exact sameness is not required)3. Single Dimension Grade Breakdown: Based on whether the Q and A of the QA pair are the same and whether the evaluator is the same, the fluctuation can be broken down into 6 scenarios shown in Figure 3 . Calculating the differences in grades before and after for these 6 scenarios and aggregating them can quantify the contributions of the 4 fluctuation reasons to the overall grade change. = [W ′ q1 * W ′ p1 * Avg ′ q1p1 + W ′ q1 * W ′ p2 * Avg ′ q1p2 + W ′ q2 * W ′ p1 * Avg ′ q2p1 + W ′ q2 * W ′ p2 * Avg ′ q2p2 + W ′ q3 * W ′ p1 * Avg ′ q3p1 + W ′ q3 * W ′ p2 * Avg ′ q3p2 ] -[W q1 * W p1 * Avg q1p1 + W q1 * W p2 * Avg q1p2 + W q2 * W p1 * Avg q2p1 + W q2 * W p2 * Avg q2p2 + W q3 * W p1 * Avg q3p1 + W q3 * W p2 * Avg q3p2 ] = ∑ 3 i=1 ∑ 2 j=1 (W ′ qi W ′ pj Avg ′ qipj -W qi W pj Avg qipj ) = ∑ 3 i=1 ∑ 2 j=1 [(W ′ qi W ′1. Selection of Human Evaluators: Human evaluators should be selected from a candidate pool with domain expertise. The circumstances may vary among different domains. In our case, they were selected from our existing team of professional annotators who regularly perform domain-specific annotation tasks. 3. Trial Evaluations: Initial trial evaluations should be conducted to assess the consistency within and among evaluators in their grading decisions. Issues identified during trials should be determined whether to be attributed to few evaluators based on measures described in Appendix F. If ambiguity found in grading rubrics, the expression of rubrics should be refined.4. Evaluation Quality Assurance: For evaluators showing inconsistent performance, targeted retraining sessions were implemented. These sessions focused on reinforcing the rubrics and providing additional examples to clarify evaluation criteria.Post-retraining, evaluators should go through further trial evaluations to validate their improvement.5. Deployment Criteria: Evaluators were only deployed into the production setting after achieving a predefined threshold of consistency in trial evaluations.  


Decision: The paper outlines a general framework for designing evaluations for domain-specific LLMs. It appears to be an industry paper, which is grounded in practical concerns about deploying LLMs for real-world use cases. As such, it does not follow the “typical” style of an academic paper, and this is a cause for some reviewer concerns. In particular, reviewers raise concerns about a lack of clear evaluation/baselines for existing approaches, and about the apparent cost of carrying out the proposed framework. The authors’ response is that no clear baselines exist because industry standards do not yet exist, and rather each case is handled in an ad hoc manner. This ad hoc nature of evaluation is the problem the paper attempts to address.

I believe that this paper would be a useful addition to COLM precisely because of its atypical nature. The reviewers’ concerns about evaluation are understandable, but given the current state of LLM research, there is likely more to be gained by exposing differing approaches and perspectives (e.g., from academia and industry, from quantitative and qualitative sciences) than prematurely imposing a specific recipe for research. This paper has value in outlining the key challenges that are faced by those applying LLMs in industry settings, and showcasing the way that these challenges are currently addressed. Such work could provide inspiration or fodder for subsequent, more rigorous, research studies.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper proposes LalaEval, an human-centered evaluation framework for domain-specific large language models (LLMs). The core principles behind the proposed work consist of five components: domain specification, criteria establishment, benchmark creation, evaluation rubric construction and results interpretation. This work combines the proposed principles and presents a general deployable framework. The authors also present the evaluation results of 3 proprietary models fine tuned on top of ChatGLM along with Ernie Bot, GPT-4. Overall this paper is more of an industry-track position piece than a pure methodological one.
strength_weakness: strength_weakness: From my understanding, LalaEval seems to demand significant resources, including time, trained personnel, and high-quality domain-specific data. Despite the nature of this piece being a positional one, I feel that this concept will be more complete if the authors can provide more details on the evaluations. Specifically, slightly more information on the process of selecting human evaluators, the training they receive, and how their evaluations are standardized would be beneficial.

Additionally, as the authors have outlined in the evaluation framework constructions, there are multiple scoring stages. It would be beneficial to include them in the main evaluation for a more detailed per-model evaluation.

Minor suggestions: 
- Try increasing the font size of figure 1 or otherwise the figure can be difficult to read. 
- Be careful about the use of \citep and \citet (e.g. first sentence of Related Work)
- When describing framework specifications, avoid using specific examples as the bullet points (e.g., change "Specific Knowledge about 
  Huolala" to "Company-specific Knowledge" in 3.3.2).
- If possible, consider open-sourcing a prototype.
questions: questions: For the main evaluation, should I believe that the lower scores of GPT-4 and Ernie Bot are due to the fact they do not have access to potentially proprietary information that is used to fine-tune the in-house ChatGLM? Additionally, how does LalaEval handle the potential subjectivity and variability in human evaluations, and what measures are in place to ensure the consistency and reliability of these evaluations?
paper_summary: paper_summary: The paper introduces LALAEval, a comprehensive human evaluation framework designed to assess LLMs. The framework incorporates a diverse set of evaluation tasks and metrics to provide a holistic understanding of LLM's performance, addressing the limitations of existing evaluation methods.
strength_weakness: strength_weakness: The discussion on potential biases or limitations of the proposed framework, as well as strategies to mitigate them, could be more comprehensive.
questions: questions: 1. I was questioned about the adapability of this framework to various domain applications. 
2. what is the novelty of this proposed framework?
paper_summary: paper_summary: This paper introduce a holistic framework for the human evaluation of domain specific large language models. This framework have five main components: domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, analysis and interpretation of evaluation results.
strength_weakness: strength_weakness: 1.	Certain method details are missed. For example, it is hard to tell how to construct a hierarchical domain specification for the logistics industry in Figure 1. The authors mentioned about the methods of the principle of mutual exclusivity, qualitative prioritization and so on, yet these methods are not well described and would be hard to understand if the readers do not have related background.

2.	It would be better if the authors could release the dataset and evaluation code they’ve created in this work.

3.	There could be more analysis to the evaluation results. For example, GPT4 underperform all the compared language models in all the results. How could we explain this phenomenon and is there any reasons that lead this result?
questions: questions: 1.	How could we evaluate if the proposed LalaEval is working better than other evaluation metrics or frameworks?

2.	The font in the figures is a little small and it needs to be set larger.
paper_summary: paper_summary: This paper introduces LalaEval, a holistic framework for conducting standardized human evaluations of domain-specific large language models (LLMs). The framework consists of five main components: domain specification, criteria establishment, benchmark dataset creation, construction of evaluation rubrics, and analysis and interpretation of evaluation outcomes. The paper demonstrates the framework's application within the logistics industry, presenting domain-specific evaluation benchmarks, datasets, and a comparative analysis of LLMs. The results highlight the framework's effectiveness in elucidating performance differences and guiding model selection and development for domain-specific LLMs.
strength_weakness: strength_weakness: - The case study focuses on the logistics industry, which may limit the generalizability of the findings to other domains. Additional case studies across various industries would strengthen the paper's claims about the framework's broad applicability.
- The paper does not provide a detailed comparison of the LalaEval framework to existing evaluation methodologies, making it difficult to assess its relative advantages or disadvantages.
paper_summary: paper_summary: 1. This paper proposes LalaEval, a framework designed to standardized and systematic human evaluation for domain-specific LLMs. The framework consists of 5 components including domain specification, criteria setting, benchmark dataset creation, construction of evaluation rubrics, and analysis of evaluation outcomes.

2. The framework’s efficacy is demonstrated through a case study in the logistics industry, showing how it helps elucidate performance differences among LLMs.
strength_weakness: strength_weakness: 1. My major concern is that the experiments in this article struggle to demonstrate the advantages and effectiveness of the proposed evaluation framework. On one hand, the author conducted only a case study involving three models, lacking validation across a broader range of domains and models. Additionally, the experimental results presented do not clarify how LalaEval surpasses previous human evaluation paradigms. On the other hand, the author only displayed the final evaluation results and conclusions, lacking presentation and validation of intermediate steps. There is also a lack of detailed description of the human evaluation details, such as the number of evaluators involved, inter-annotator agreement, and so forth.

2. As an academic paper, I think that the framework proposed by this study does not present any fundamental innovation compared to the commonly used human evaluation processes previously established.

Comments: title: Rebuttal of the New Review (Continued)
Rebuttal of the New Review
Response to the authors
comment: As the discussion period is about to end, we would like to extend our sincere thanks to all the reviewers for their valuable feedback. We also want to take this opportunity to reiterate our main contributions:

1. **Standardizing the Evaluation Process**: Compared to the current literature that often involves collecting datasets without a structured approach and inviting domain experts to score subjectively, we proposed a meta-framework to standardize the end-to-end process for evaluating domain-specific LLMs. Our framework addresses key questions such as, “How shall we define our domain?”, “How shall we set capability criteria and build our own evaluation datasets/benchmarks?”, and “How shall we train domain annotators and analyze their evaluations to minimize subjectivity?”.

2. **Successful Deployment and Impact**: We have successfully deployed our framework within the model selection and development process in our domain. The effectiveness of our framework is proven to be crucial to the implementation of domain-specific LLMs that serve millions of consumers. Furthermore, the deployment of our framework in other domains such as HR/IT maintenance/text-to-SQL/telemarketing has also made significant progress , demonstrating its versatility and impact. We believe that industry insights gained from large-scale real-world evaluation deployment can bring unique value to the community.

Thank you once again for your constructive feedback and consideration.
Thanks for response. However, my concerns about the novelty and its empirical comparison to other evaluation framework still exist. So I tend to keep my score.
# Contribution and Comparison to Previous Framework
There is no standardized end-to-end (from domain specification to criteria/dataset/rubrics construction to result interpretation) human evaluation protocols that meet industry-level domain needs currently, scoring metrics and and corresponding interpretation are only part of LalaEval, so LalaEval is the first holistic human evaluation conceptual framework for domain-specific LLMs, addressing the lack of standardized protocols in existing literature and it is hard to be compared to previous paradigms. Given the open-generation nature of LLM output and the high stakes of large-scale real-world domain-specific LLM deployments, LalaEval aims to reduce overall human-in-the-loop evaluation subjectivity, to ensure objective and aligned evaluation results and/or further guide our LLM development. Existing literature primarily mentions human evaluation of specific tasks without detailing implementation, standardization or even protocols (See Section 5.2 of A Survey on Evaluation of Large Language Models, arXiv:2307.03109 for a detailed review). LalaEval fills this gap by proposing protocols for domain specification, criteria/dataset/rubrics construction, result analysis including reliability measurement and a systematic integration of these components. Furthermore, our deployed framework in a real-world production setting demonstrates its practical effectiveness, contributing valuable LLM-related industry (esp. non-tech) insights.
# Font Size
Thank you for pointing out. We will fix the font size once we are allowed to submit a revision.
Thank you for your new valuable feedback. We have provided brief point-by-point responses below and will include any additional necessary details in the camera-ready version.
# Method Elaboration
We will emphasize that domain specification is primarily conceptual and qualitative, relying on expert knowledge and understanding of the business context. Specifically, we will:
1. Explain the conceptual principles used:
      - Mutual exclusivity: each subdomain should be distinct and non-overlapping
      - Collective exhaustiveness: all relevant aspects of the domain should be covered
2. Elaborate on the qualitative prioritization process:
      - Priorities are assigned based on expert judgment of subdomain relevance
      - Consideration is given to alignment with specific business objectives and focus areas
      - The prioritization is conducted through discussions with domain experts and stakeholders (e.g. managers of the relevant businesses)
3. Provide a step-by-step guide on applying this qualitative methodology in the example logistics industry context:
      - Begin with identifying the most granular, relevant subdomains based on expert knowledge
      - Iteratively combine these granular subdomains into broader, more comprehensive categories
      - Assign priorities to each subdomain and category based on their determined relevance and business alignment
      - Continue this process until a sufficiently comprehensive domain specification is constructed
4. Include concrete examples to illustrate each step of the conceptual process for a general audience: 
      - Demonstrate how "Intracity Freight Transportation" is identified as the most granular, highest priority subdomain
      - Show how it is combined with other subdomains like "Intercity Freight Transportation" into the broader "Road Transportation" category
      - Explain how priority levels are assigned qualitatively based on Huolala's business focus and expert judgment
# Opensource Request
Our request to open-source is currently going through the internal approval process. We will release the dataset and code once it is approved.
# Lower Scores of GPT-4
From the detailed results of all 12 capability dimensions, the lower scores of the foundation model GPT-4 are likely due to the absence of proprietary information used to fine-tune the PLLMs. They performed better on general capabilities.
| Capability         | Capability Dimension                        | GPT4 | Ernie Bot | PLLM3 | PLLM2 | PLLM1 |
|--------------------|---------------------------------------------|:----:|:---------:|:-----:|:-----:|:-----:|
| Domain Capability  | Conceptual and Terminological Understanding | 66.0 |    80.0   |  82.5 |  84.5 |  85.0 |
| Domain Capability  | Company Information                         | 22.5 |    85.0   |  92.0 |  79.0 |  98.5 |
| Domain Capability  | Legal and Policy Knowledge                  | 38.5 |    67.5   |  91.5 |  81.0 |  91.5 |
| Domain Capability  | Industry Insights                           | 48.0 |    84.5   |  88.5 |  85.5 |  90.0 |
| Domain Capability  | Company-specific Knowledge                  | 19.0 |    81.5   |  89.0 |  77.0 |  86.5 |
| Domain Capability  | Creative Capability in Logistics Context    | 94.7 |    92.0   |  40.0 |  40.0 |  40.0 |
| General Capability | Semantic Understanding                      | 98.0 |    92.0   |  90.0 |  82.0 |  88.0 |
| General Capability | Contextual Conversation                     | 68.0 |    75.0   |  20.0 |  24.0 |  32.0 |
| General Capability | Answer Completeness and Coherence           | 64.0 |    90.0   |  54.0 |  56.0 |  54.0 |
| General Capability | Factuality                                  | 86.0 |    98.0   |  77.0 |  76.0 |  74.0 |
| General Capability | Creativity                                  | 70.0 |    81.3   |  55.3 |  60.7 |  65.3 |
| General Capability | Logical Reasoning                           | 76.0 |    86.0   |  60.0 |  56.0 |  68.0 |
I feel much sorry that I attached the wrong review to this submission, I have changed the reviews accordingly. Sorry again for my bad mistake.
# Intermediate Steps (Continued)
We take the opportunity here to further present the complete intermediate per-model evaluation results of all our 12 capability dimensions.
| Capability         | Capability Dimension                        | GPT4 | Ernie Bot | PLLM3 | PLLM2 | PLLM1 |
|--------------------|---------------------------------------------|:----:|:---------:|:-----:|:-----:|:-----:|
| Domain Capability  | Conceptual and Terminological Understanding | 66.0 |    80.0   |  82.5 |  84.5 |  85.0 |
| Domain Capability  | Company Information                         | 22.5 |    85.0   |  92.0 |  79.0 |  98.5 |
| Domain Capability  | Legal and Policy Knowledge                  | 38.5 |    67.5   |  91.5 |  81.0 |  91.5 |
| Domain Capability  | Industry Insights                           | 48.0 |    84.5   |  88.5 |  85.5 |  90.0 |
| Domain Capability  | Company-specific Knowledge                  | 19.0 |    81.5   |  89.0 |  77.0 |  86.5 |
| Domain Capability  | Creative Capability in Logistics Context    | 94.7 |    92.0   |  40.0 |  40.0 |  40.0 |
| General Capability | Semantic Understanding                      | 98.0 |    92.0   |  90.0 |  82.0 |  88.0 |
| General Capability | Contextual Conversation                     | 68.0 |    75.0   |  20.0 |  24.0 |  32.0 |
| General Capability | Answer Completeness and Coherence           | 64.0 |    90.0   |  54.0 |  56.0 |  54.0 |
| General Capability | Factuality                                  | 86.0 |    98.0   |  77.0 |  76.0 |  74.0 |
| General Capability | Creativity                                  | 70.0 |    81.3   |  55.3 |  60.7 |  65.3 |
| General Capability | Logical Reasoning                           | 76.0 |    86.0   |  60.0 |  56.0 |  68.0 |
# Human Evaluation Details (Continued)
We present here the number of annotators and our simplified disagreement analysis of the original five models. The result is at question-model-response level. We define a response as "disagreed" when not all five annotators consistently scored zero(bad response)/positive marks(good response) for each question-model-response.
| Capability         | Capability Dimension                        | Number of Annotators | Ratio of Disagreement |
|--------------------|---------------------------------------------|:--------------------:|:---------------------:|
| Domain Capability  | Conceptual and Terminological Understanding |           5          |         27.0%         |
| Domain Capability  | Company Information                         |           5          |         17.0%         |
| Domain Capability  | Legal and Policy Knowledge                  |           5          |         33.0%         |
| Domain Capability  | Industry Insights                           |           5          |         24.0%         |
| Domain Capability  | Company-specific Knowledge                  |           5          |         17.0%         |
| Domain Capability  | Creative Capability in Logistics Context    |           5          |         80.0%         |
| General Capability | Semantic Understanding                      |           5          |         22.0%         |
| General Capability | Contextual Conversation                     |           5          |         22.0%         |
| General Capability | Answer Completeness and Coherence           |           5          |         10.0%         |
| General Capability | Factuality                                  |           5          |         18.0%         |
| General Capability | Creativity                                  |           5          |         40.0%         |
| General Capability | Logical Reasoning                           |           5          |          4.0%         |
# Detailed Evaluation Results (Continued)
We take the opportunity here to further present the complete intermediate per-model evaluation results of all our 12 capability dimensions.
| Capability         | Capability Dimension                        | GPT4 | Ernie Bot | PLLM3 | PLLM2 | PLLM1 |
|--------------------|---------------------------------------------|:----:|:---------:|:-----:|:-----:|:-----:|
| Domain Capability  | Conceptual and Terminological Understanding | 66.0 |    80.0   |  82.5 |  84.5 |  85.0 |
| Domain Capability  | Company Information                         | 22.5 |    85.0   |  92.0 |  79.0 |  98.5 |
| Domain Capability  | Legal and Policy Knowledge                  | 38.5 |    67.5   |  91.5 |  81.0 |  91.5 |
| Domain Capability  | Industry Insights                           | 48.0 |    84.5   |  88.5 |  85.5 |  90.0 |
| Domain Capability  | Company-specific Knowledge                  | 19.0 |    81.5   |  89.0 |  77.0 |  86.5 |
| Domain Capability  | Creative Capability in Logistics Context    | 94.7 |    92.0   |  40.0 |  40.0 |  40.0 |
| General Capability | Semantic Understanding                      | 98.0 |    92.0   |  90.0 |  82.0 |  88.0 |
| General Capability | Contextual Conversation                     | 68.0 |    75.0   |  20.0 |  24.0 |  32.0 |
| General Capability | Answer Completeness and Coherence           | 64.0 |    90.0   |  54.0 |  56.0 |  54.0 |
| General Capability | Factuality                                  | 86.0 |    98.0   |  77.0 |  76.0 |  74.0 |
| General Capability | Creativity                                  | 70.0 |    81.3   |  55.3 |  60.7 |  65.3 |
| General Capability | Logical Reasoning                           | 76.0 |    86.0   |  60.0 |  56.0 |  68.0 |