QAEval: Mixture of Evaluators for Question-Answering Task Evaluation
Authors: Tan Yue, Rui Mao, xuzhao Shi, SHUO ZHAN, Zuhao Yang, Dongyan Zhao
Accepted at: acl25

QAEval: Mixture of Evaluators for Question-Answering Task Evaluation

Tan Yue1, Rui Mao3, Xuzhao Shi4,
Shuo Zhan3, Zuhao Yang2, Dongyan Zhao1,2,*
1Wangxuan Institute of Computer Technology, Peking University
2State Key Laboratory of General Artificial Intelligence
3Nanyang Technological University
4Beijing University of Posts and Telecommunications
{yuetan,zhaodongyan}@pku.edu.cn, rui.mao@ntu.edu.sg, sxzs@bupt.edu.cn, {zhan0590,yang0756}@e.ntu.edu.sg

Abstract

Question answering (QA) tasks serve as a key benchmark for evaluating generation systems. Traditional rule-based metrics, such as accuracy and relaxed-accuracy, struggle with open-ended and unstructured responses. LLM-based evaluation methods offer greater flexibility but suffer from sensitivity to instructions, robustness issues, and high computational costs. To overcome these challenges, we introduce QAEval, a hybrid framework combining rule-based reliability with LLM-based adaptability. QAEval utilizes two high-quality datasets: QAExtract for short-answer extraction and QAScore for scoring model training. By integrating a Mixture of Evaluators model with Dynamic Load Balancing Optimization, QAEval enables accurate, cost-effective QA evaluation. Experimental results show it outperforms models like GPT-4o and Claude-3, achieving 92.3% accuracy with only 0.6B parameters.

1 Introduction

Question answering (QA) systems (Ojokoh and Adebisi, 2018; Zaib et al., 2022) serve as an important medium of human-computer interaction, with diverse applications, such as information retrieval for solving complex problems (Masry et al., 2022; Liu et al., 2023; Welivita and Pu, 2023). With the advancement of large language models (LLMs), QA tasks have become a crucial benchmark for assessing their performance (Krithara et al., 2023; Mao et al., 2024). Robust QA evaluation becomes essential for understanding the strengths and limitations of LLMs, particularly in reasoning, factual accuracy, and contextual comprehension.

Existing QA evaluation methods exhibit notable limitations in assessing complex and open-ended responses (Wang et al., 2024). Traditional rule-based approaches rely on strict matching criteria to compute accuracy (Kahou et al., 2017), making them struggle with open-ended questions that require contextual understanding. While relaxed-accuracy metrics (Methani et al., 2020) improve flexibility by allowing numerical tolerance, they remain insufficient for capturing contextualized correctness. More recently, LLM-based evaluators have gained popularity due to their ability to assess responses holistically. These methods score the input [Question, True-answer, Model-response] in a binary classification ([0, 1]) by using tailored prompts (Xia et al., 2024; Wang et al., 2024). However, these methods introduce new challenges, including sensitivity to prompt variations (Mao et al., 2023), evaluation inconsistencies across different LLM versions, and high computational costs (Shekhar et al., 2024). These constraints hinder their reliability and scalability, particularly for large-scale evaluation tasks where ef-

Question 1: What is the leftmost labeled tick on the x-axis?
Response: The leftmost labeled tick on the x-axis for the subplot at row 2 and column 1 (Original I2) is 50.
True Answer: 50
Relaxed-Accuracy: 0 ×
GPT-4o: 1 ✓
QAEval: 1 ✓

Question 2: Could you analyze this graph and tell me the x-coordinate of the peak?
Response: Yes, the x-coordinate of the peak is 390nm.
True Answer: 387.5nm
Relaxed-Accuracy: 0 ×
GPT-4o: 0 ×
QAEval: 1 ✓

Question 3: List the horizontal coordinates of all peaks.
Response: The horizontal coordinates: 595nm;612nm;630nm
True Answer: 595nm;612nm;630nm
Relaxed-Accuracy: 0 ×
GPT-4o: 0 ×
QAEval: 0.5 ✓

Figure 1: Comparison of complex QA evaluations using different methods, including our proposed QAEval.
ficiency and stability are critical. Typical errors presented in relaxed-accuracy and GPT-4o-based evaluation methods are shown in Fig. 1.

To address these challenges, we propose QAEval\footnote{https://github.com/yuetanbupt/QAEval}, a hybrid evaluation framework that combines the robustness of rule-based methods with the adaptability of language model-based evaluation. Unlike existing LLM-based methods, QAEval is designed to be lightweight, plug-and-play, and cost-efficient while maintaining high accuracy. Our method utilizes a Mixture of Evaluators (MOE) model, trained with Dynamic Load Balancing Optimization (DLBO), which leverages Kullback-Leibler (KL) dispersion to enhance evaluation robustness. Additionally, we introduce two manually labeled datasets, QAExtract with 9,889 samples and QAScore with 14,419 samples, to refine both extraction and scoring capabilities. Experimental results on QAScore test set demonstrate that QAEval achieves state-of-the-art performance (92.3%), surpassing models such as GPT-4o and Claude-3 while using only 0.6B parameters, significantly reducing computational overhead without compromising accuracy.

The contributions of this work are summarized as follows: 1) We introduce QAEval, a novel evaluation framework that efficiently and accurately assesses QA tasks by integrating an answer extraction model, a rule-based scoring model, and a scoring model utilizing MOE. The framework is lightweight, and highly robust, supporting a plug-and-play deployment approach. 2) To enhance the robustness of the MOE scoring model, we propose DLBO, which dynamically balances evaluator selection propensity and evaluation accuracy, ensuring more stable and reliable performance. 3) We construct two high-quality datasets: QAExtract for answer extraction and QAScore for scoring, by sampling data from 31 QA datasets and generating responses using 10 mainstream LLMs. All data is manually labeled and thoroughly proofread to ensure dataset quality and reliability.

2 Related works

With the rapid development of LLMs, QA tasks have become an important benchmark for testing model understanding (Yue et al., 2023; Kim et al., 2024) and reasoning capabilities.

Early QA datasets, such as FigureQA (Kahou et al., 2017) and DVQA (Kafle et al., 2018), primarily rely on fixed-answer formats, where responses are constrained to binary ("yes/no") or pre-defined categorical labels. These datasets are evaluated using accuracy, which directly measure the match between model outputs and ground-truth answers (see Fig. 2A). While effective for structured tasks, this rigid evaluation framework struggle to accommodate more complex QA tasks that involve open-ended or numerical responses. To address this, relaxed-accuracy is introduced (Methani et al., 2020), allowing for a tolerance range in numerical answers and improving flexibility. Beyond numeric relaxation, recent rule-based methods (Bulian et al., 2022; Li et al., 2024) further enhance equivalence judgment through manually defined transformation rules, enabling interpretable matching for paraphrased or reformatted answers. However, as QA tasks continue to grow in complexity (Ma et al., 2024), particularly with the emergence of LLMs capable of generating diverse and unconstrained responses, traditional rule-based methods fail to provide a comprehensive assessment of response quality. The inherent variability in LLM outputs, such as paraphrased responses or contextual interpretations (Yue et al., 2021; Zhu et al., 2024; Verga et al., 2024), makes it difficult for rule-based methods to accurately capture semantic equivalence (Wang et al., 2022; Wang, 2023; Yue et al., 2024).

To overcome these limitations, LLM-based evaluation methods (Achiam et al., 2023; Xia et al., 2024) have been developed, leveraging the contextual interpretation capabilities of LLMs (see Fig. 2B). These approaches instruct an LLM to assess responses by comparing the \([Question, True-answer, Model-response]\) tuple, where the evaluation task is often treated as a binary classification problem, assigning a score of "1" for correct answers and "0" for incorrect answers. Some methods, such as Charxiv (Wang et al., 2024), instruct LLMs to first extract key information from lengthy model responses and then score them. While LLM-based evaluation improves flexibility by handling diverse and unstructured responses, it also introduces significant challenges. LLMs are prone to hallucinations and inconsistent responses, leading to unreliable or unexplainable scoring decisions. Furthermore, their sensitivity to prompt variations, dependence on proprietary model versions, and high computational costs hinder their robustness and scalability for large-scale evaluation tasks (Chang et al., 2024).

In summary, rule-based evaluation approaches
Figure 2: Comparison of the proposed QAEval framework with existing methods. Figure 2A represents the rule-based method, Figure 2B represents the LLM-based method, and Figure 2C shows our QAEval framework.

struggle with model-generated responses that vary in structure and phrasing, leading to underestimation of correct answers due to limited semantic flexibility. While LLMs enable flexible response assessment (Vu et al., 2024), they frequently produce hallucinated errors and lack interpretability, making their scoring inconsistent and difficult to verify. LLM-based evaluators exhibit poor robustness, with significant fluctuations in results due to slight prompt variations (Xu et al., 2023; Chiang and Lee, 2023). Their reliance on proprietary models introduces version inconsistencies, while high API/deployment costs and latency make them impractical for large-scale QA evaluation.

3 Methodology

We propose QAEval, a novel framework designed for efficient and accurate evaluation of QA tasks. As illustrated in Fig. 2C, QAEval comprises three key components: an answer extraction model, a rule-based scoring model, and an MOE scoring model. By integrating extraction with a multi-stage scoring approach, QAEval enhances evaluation accuracy while maintaining a lightweight model architecture.

QAEval evaluates each input sample, consisting of [Question, True Answer, Model Response], via a multi-step process: 1) Answer Extraction. Short-form answers are extracted from lengthy model-generated responses to standardize the evaluation input. 2) Rule-Based Quick Scoring. The extracted answers are first assessed using a improved rule-based scoring method for rapid evaluation. 3) Flexible Scoring via MOE. If the rule-based approach fails to establish a reliable match between the extracted and true answers, an MOE scoring model is employed to provide a more flexible and context-aware assessment. The final evaluation output is a three-tiered scoring system ([0, 0.5, 1]), capturing varying degrees of answer correctness.

3.1 Answer extraction

The answer extraction model is designed to distill concise answers from lengthy QA responses, facilitating a more straightforward comparison with ground-truth answers. To achieve this, we fine-tune a Qwen2.5-0.5B model using our curated QAExtract dataset. Through instruction fine-tuning, the model learns to extract key information from complex and verbose responses, ensuring more efficient and accurate matching during evaluation.

The extraction model takes the input \([Q, R]\), where \(Q\) represents the question and \(R\) represents the model-generated response. The output \(A_{\text{pre}}\) is the extracted information from the QA system response. During training, the model learns to minimize the loss between the predicted answer \(A_{\text{pre}}\) and the true extracted answer \(A_{\text{ext}}\) with the following loss \(L_{\text{extract}} = -\sum_{t=1}^T \log P(A_t|A_{<t}, Q, R; \theta)\), where \(T\) is the length of the true extracted answer \(A_{\text{ext}}\); \(A_t\) is the \(t\)-th token of the true extracted answer; \(A_{<t}\) represents the sequence of tokens generated before \(A_t\) at step \(t\); \(P(A_t|A_{<t}, Q, R; \theta)\) is the probability of generating token \(A_t\) conditioned on the previous tokens, question \(Q\), response \(R\), and model parameters \(\theta\). During inference, the model extracts the short answer by finding the sequence \(A_{\text{pre}}\) that maximizes the joint probability of all tokens by \(A_{\text{pre}} = \argmax_A \prod_{t=1}^T P(A_t|A_{<t}, Q, R; \theta)\).
3.2 Rule-based quick scoring
Based on the relaxed-accuracy method (Methani et al., 2020), we propose an improved rule-based evaluation method (\( R_{acc} \)) that aims to measure the prediction results of the model more comprehensively. While retaining the error-tolerance mechanism for numerical answers, the method adds a case-insensitive matching algorithm for string answers, thus increasing the flexibility and applicability of the evaluation. In addition, to address the limitation that the relaxed-accuracy method only provides two scores, 0 or 1, we introduce an intermediate score of 0.5 to identify partially correct answer cases. When dealing with multiple-answer samples, we match each model-predicted answer individually with the corresponding true answer to ensure evaluation accuracy. The implementation details are shown in Algorithm 1.

The rule-based method provides a fast and reliable preliminary evaluation using predefined matching rules. In the initial scoring phase, it calculates a score \( S \), based on predefined matching rules. The scoring function is given by:

\[
S = \begin{cases}
1, & \text{if } R_{acc}(A_{pre}) = A_{true} \\
0.5, & \text{if } R_{acc}(A_{pre}) \in A_{true} \\
0, & \text{otherwise}
\end{cases}
\]

If the rule-based score \( S \) is 0.5 or 1, the extracted answer is deemed a match, and the score is finalized. If \( S = 0 \), indicating no match or an incorrect response, the filtered sample (denoted as \( F \)) is passed to the MOE model for further assessment.

\[
\text{FinalScore} = \begin{cases}
S, & \text{if } S = [0.5, 1] \\
\text{MOE}(F), & \text{if } S = 0
\end{cases}
\]

This two-stage scoring strategy merges the reliability and efficiency of the rule-based method with the flexibility of the MOE model, offering a balanced solution for complex QA tasks. It reduces computational costs while maintaining accuracy.

3.3 Mixture of evaluators
We develop a scoring model using MOE (Fan et al., 2024; Jiang et al., 2024; Cai et al., 2024; Guo et al., 2025) to handle complex QA responses that rule-based methods cannot match. Unlike LLM-based evaluators with a generative paradigm, we treat QA scoring as a classification task and train a classifier. The advantage is that the classification approach directly maps the evaluation outcomes to predefined categories, which enhances stability and transparency. It also avoids the unpredictability of generative models and reduces the risk of hallucinations. To address the evaluation accuracy drop caused by diverse QA types, we introduce an MOE architecture with DLBO, which improves scoring accuracy by assigning specific QA types to evaluators (see Fig. 3). The input is defined as \( E_{input} \) for evaluators and \( G_{input} \) for the gating network, where \( G_{input} = [..., [Q, A_{true}], ...] \), \( E_{input} = [..., [Q, A_{true}, R, A_{pre}], ...], \forall i \in [1, N] \). \( N \) is the batch size.

First, a gating network has been designed, which assigns a weight distribution based on the QA type of the input. It dynamically directs different sample types to the most suitable evaluators, ensuring specialization. Features \( \mathbf{H}_{CLS} = \mathcal{M}^A(G_{input}) \), are extracted from the ALBERT model (\( \mathcal{M}^A(\cdot) \)) (Lan, 2019) using the [CLS] token representation, where \( \mathbf{H}_{CLS} \in \mathbb{R}^{N \times d} \) is the [CLS] feature vector. \( d \) is the hidden layer size. The gating network takes \( \mathbf{H}_{CLS} \) as input and computes the weight distribution across evaluators. \( \mathbf{G} = \sigma(\frac{\mathbf{H}_{CLS}\mathbf{W}_g + \mathbf{b}_g}{2}) \), where \( \mathbf{W}_g \in \mathbb{R}^{d \times K} \) is the weight matrix of the gating network. \( \mathbf{b}_g \) is the bias vector. \( K \) denotes the

Algorithm 1 Rule-based Scoring Algorithm
1: Function Correctness(t, p, δ = 0.05)
2: Convert t and p to lowercase (for string comparison)
3: Define toFloat(x): parse x (original or lowercased) as float; if ends with '%', remove '%' and divide by 100
4: \( t_f \leftarrow \text{toFloat}(t), p_f \leftarrow \text{toFloat}(p) \)
5: if \( t_f \) and \( p_f \) are not None then
6:   if \( t_f == 0 \) then
7:     return (\( p_f == 0 \))
8:   end if
9:   Compute \( \Delta \leftarrow \frac{|p_f - t_f|}{|t_f|} \)
10:  return (\( \Delta \leq \delta \))
11: else
12:  return (\( t == p \))
13: end if
14: Function Score(R, A)
15: Trim leading/trailing spaces in R, A, then split R and A into lists (by delimiter ';')
16: if \( |R| > |A| \) then
17:   return 0
18: end if
19: Initialize match ← []
20: for r in R do
21:   m ← any(Correctness(a, r, δ) for a in A)
22:   Append m to match
23: end for
24: if all(match) and \( |R| == |A| \) then
25:   return 1
26: else if all(match) then
27:   return 0.5
28: else
29:   return 0
30: end if
Figure 3: The proposed MOE scoring model. The left side of the figure illustrates the model scoring process, and the right side represents the DLBO for the gating network.

number of evaluators. \( \mathbf{G} \in \mathbb{R}^{N \times K} \) is the weight distribution generated by the gating network. \( \sigma \) is a softmax function.

Next, we develop a gating network to select the best evaluators for processing the evaluation task. Based on the gating network’s output (the weight values), we select the Top2 evaluators with the highest weights. Each evaluator independently performs the classification. The outputs of the selected evaluators are fused with their weights: \( y_{\text{fused}} = \sum_{k=1}^K \sigma(\text{Top2}(\mathbf{G})) \cdot \mathcal{M}^4(E_{\text{input}_i}) \).

Finally, during MOE training, the model may over-rely on one evaluator, diminishing the utility of others. To address this, we introduce Kullback-Leibler (KL) divergence to create a DLBO function. By applying a weight decay mechanism, the gating network’s weights are dynamically adjusted across epochs, ensuring balanced load distribution and enhancing overall model performance. The KL divergence measures the difference between the evaluator distribution and a uniform distribution:

\[
\mathcal{L}_{\text{balance}} = \sum_{k=1}^K (\frac{1}{N} \sum_{i=1}^N G_{ik}) \cdot \log(\frac{\frac{1}{N} \sum_{i=1}^N G_{ik}}{\frac{1}{K}})
\]

where \( k \) is the indexes of evaluators. An improved weight decay function dynamically adjusts the balancing weight by

\[
w_{\text{balance}}(p) = w_{\min} + \frac{w_{\text{initial}} - w_{\min}}{1 + e^{j(p/P-c)}}
\]

where \( w_{\min} \) and \( w_{\text{initial}} \) are the minimum and initial weights. \( p \) is the current epoch, \( P \) is the total number of epochs. \( c = 0.5 - f \), where \( f \) is the early adjustment factor. \( j \) is the curve steepness factor. The overall loss is \( \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{ce}} + w_{\text{balance}}(p) \cdot \mathcal{L}_{\text{balance}} \).

The DLBO function aims to constrain the average evaluator weight \( \frac{1}{N} \sum_{i=1}^N G_{ik} \) to close to the target uniform distribution \( \frac{1}{K} \), avoiding over-reliance on specific evaluators. The function is defined as the weighted logarithmic difference between the average evaluator weight and the target distribution. During the optimization, load balancing is achieved by updating the gradient of each sample weight \( G_{ik} \) (we define \( \mu_k = \frac{1}{N} \sum_{i=1}^N G_{ik} \)):

\[
\frac{\partial \mathcal{L}_{\text{balance}}}{\partial G_{ik}} = \frac{\partial \mathcal{L}_{\text{balance}}}{\partial \mu_k} \cdot \frac{\partial \mu_k}{\partial G_{ik}},
\]
\[
\frac{\partial \mathcal{L}_{\text{balance}}}{\partial \mu_k} = \log \left( \frac{\mu_k}{\frac{1}{K}} \right) + 1,
\]
\[
\frac{\partial \mathcal{L}_{\text{balance}}}{\partial G_{ik}} = \frac{1}{N} \left( \log \frac{\frac{1}{N} \sum_{i=1}^N G_{ik}}{\frac{1}{K}} + 1 \right).
\]

In the early training stages, higher dynamic weights cause greater gradient adjustments when evaluator weights deviate from the target distribution. DLBO ensures that evaluators are exposed to more data initially. The weights decay with training. In the later stages, gradients focus on prediction differences, allowing better-performing evaluators to receive higher weights for their specialized QA tasks, ultimately improving model accuracy.

4 Dataset

4.1 Dataset collection

To train the extraction and scoring models, we build two quality, manually labeled datasets: QAExtract
<table>
  <tr>
    <th>Dataset</th>
    <th>Train</th>
    <th>Val.</th>
    <th>Test</th>
    <th>Total</th>
  </tr>
  <tr>
    <td>QAExtract</td>
    <td>7889</td>
    <td>2000</td>
    <td>-</td>
    <td>9,889</td>
  </tr>
  <tr>
    <td>QAScore</td>
    <td>12419</td>
    <td>1000</td>
    <td>1000</td>
    <td>14,419</td>
  </tr>
</table>

Table 1: QAExtract and QAScore statistics.

and QAScore. Both are collected from 31 widely-used datasets for QA, covering a wide range of domains, including figure/chart QA, textbook QA, visual QA, etc (see Table A.2 in Appendix A for details). In the initial data collection phase, 60,000 QA samples are drawn from 31 datasets. However, the prevalence of homogeneous and simple QA pairs pose a risk of ineffective training. To address this, similar samples within the same dataset are randomly selected, followed by manual filtering, reducing the dataset to 24,000 samples while ensuring quality and diversity (shown in Table 1). The selection process adhered to three key principles: 1) maintaining a balanced data distribution by controlling the proportion of similar QA samples; 2) limiting the inclusion of simple QA pairs to a small set of representative examples; 3) increasing the proportion of complex QA samples to enhance dataset difficulty and model generalization.

4.2 Response generation

To further improve the reality and diversity of the dataset, we use 10 popular LLMs (e.g., GPT-4o) to generate responses (model list shown in Appendix B). During the generation, we use manually constructed diverse instruction templates to guide the LLMs to generate rich answer types.

4.3 QAExtract-dataset

The QAExtract dataset is used for instruction fine-tuning the extraction model of QAEval to generate concise and precise answers from longer responses.

To construct the dataset, simple cases where short answers can be derived through rule-based matching or LLM-generated extraction are first processed preliminarily, followed by manual verification. For more complex QA samples, manual extraction is conducted to ensure accuracy and quality. Three independent annotators (Appendix A.2) with expertise in NLP and QA research participate in this process. In cases where manually extracted answers differ among annotators, discussions are conducted to reach a consensus. The final dataset consists of 9,889 samples, each containing [Question, Model Response] as input and [True Extracted Answer] as the ground truth output.

To standardize the annotation task, the extraction process adheres to strict guidelines: 1) Human annotators extract short answers by referring to the given context [Question, Model Response]. 2) Extracted answers must be strictly derived from the content of [Model Response] without adding external information. 3) Extracted answers should be as concise as possible, retaining only numerical values when applicable, with multiple answers separated by semicolons.

4.4 QAScore-dataset

The QAScore dataset is designed to train a scoring model that evaluates the alignment between [Model Responses] and [True Answers]. It comprises 14,419 samples, each structured as [Question, True Answer, Model Response] as input and [True Score] as the target output. To ensure high quality and accuracy, all samples are human-scored based on a standardized three-level scale: Complete Match (1.0) for responses identical to the true answer, Partial Match (0.5) for responses with partial correctness but some omissions or biases, and Mismatch or Excess Content (0.0) for entirely incorrect or extraneous responses. A rigorous annotation process is implemented to maintain consistency and reliability. Three expert annotators (Appendix A.2) with backgrounds in NLP and QA evaluation are selected and instructed with clear scoring guidelines and examples to minimize subjectivity. Each sample is independently scored by the three annotators, with discrepancies resolved through discussion to reach a consensus.

5 Experiment

5.1 Methods

We perform extensive experiments using the rule-based and LLM-based methods. 1) Rule-based: Accuracy, Relaxed-accuracy (Methani et al., 2020), BEM (Bulian et al., 2022), and PEDANTS (Li et al., 2024). 2) LLM-based: Qwen2.5 (Team, 2025), Claude-3 (Anthropic, 2024), Gemini-1.5 (Team et al., 2024a), GPT-4o-mini (OpenAI, 2024), ChartX (Xia et al., 2024), Charxiv (Wang et al., 2024), and GPT-4o (Hurst et al., 2024). More details are shown in Appendix C.

5.2 Settings

For the rule-based and open-source LLM-based methods, we use publicly available code and pre-
<table>
  <tr>
    <th colspan="3">Instruction</th>
  </tr>
  <tr>
    <td colspan="3">Given multiple question-answer pairs and the corresponding predictions, evaluate the correctness of predictions. The output should be only "0", "0.5", or "1". Note that if the groundtruth answer is a numeric value with/without the unit, impose 5% error tolerance to the answer. "0.5" should be given if multiple correct answers are partially correct, but incorrect or extra answers should be scored as "0".</td>
  </tr>
</table>

Figure 4: The instruction for proprietary LLM-based methods.

<table>
  <tr>
    <th>Name</th>
    <th>Variable</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Number of trained evaluators</td>
    <td>\( K \)</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Number of selected evaluators</td>
    <td>Top-\( k \)</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Minimum weight</td>
    <td>\( w_{\min} \)</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Initial weight</td>
    <td>\( w_{\text{initial}} \)</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Early adjustment factor</td>
    <td>\( f \)</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>Curve steepness factor</td>
    <td>\( j \)</td>
    <td>20</td>
  </tr>
  <tr>
    <td>Hidden layer size</td>
    <td>\( d \)</td>
    <td>768</td>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>\( N \)</td>
    <td>16</td>
  </tr>
</table>

Table 2: Hyper-parameter statistics.

<table>
  <tr>
    <th>Methods</th>
    <th>RT(s)</th>
    <th>Cost</th>
    <th>Acc. (%)</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>-</td>
    <td>-</td>
    <td><b>99.1</b></td>
  </tr>
  <tr>
    <th colspan="4">Rule-based</th>
  </tr>
  <tr>
    <td>Accuracy</td>
    <td>4.17 (O)</td>
    <td>Free</td>
    <td>47.5</td>
  </tr>
  <tr>
    <td>Relaxed-accuracy</td>
    <td>4.36 (O)</td>
    <td>Free</td>
    <td>47.6</td>
  </tr>
  <tr>
    <td>BEM</td>
    <td>56.34 (O)</td>
    <td>Free</td>
    <td>63.5</td>
  </tr>
  <tr>
    <td>PEDANTS</td>
    <td>15.89 (O)</td>
    <td>Free</td>
    <td>70.2</td>
  </tr>
  <tr>
    <th colspan="4">LLM-based</th>
  </tr>
  <tr>
    <td>CharX(GPT-3.5)</td>
    <td>1281.74 (P)</td>
    <td>$0.5(Q)+$1.5(A)</td>
    <td>64.4</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>1886.73 (P)</td>
    <td>$0.15(Q)+$0.6(A)</td>
    <td>84.5</td>
  </tr>
  <tr>
    <td>Claude-3</td>
    <td>1418.14 (P)</td>
    <td>$0.25(Q)+$1.25(A)</td>
    <td>85.5</td>
  </tr>
  <tr>
    <td>Gemini-1.5</td>
    <td>1172.81 (P)</td>
    <td>$0.15(Q)+$0.6(A)</td>
    <td>86.1</td>
  </tr>
  <tr>
    <td>CharX(GPT-4o)</td>
    <td>1381.60 (P)</td>
    <td>$2.5(Q)+$10(A)</td>
    <td>87.5</td>
  </tr>
  <tr>
    <td>Charxiv</td>
    <td>3024.51 (P)</td>
    <td>$2.5(Q)+$10(A)</td>
    <td>90.6</td>
  </tr>
  <tr>
    <th colspan="4">Fine-tune</th>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B</td>
    <td>49.69 (O)</td>
    <td>Free</td>
    <td>44.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-0.5B(FT)</td>
    <td>129.97 (O)</td>
    <td>Free</td>
    <td>58.1</td>
  </tr>
  <tr>
    <td>QAEval (0.6B)</td>
    <td>139.62 (O)</td>
    <td>Free</td>
    <td><b>92.3</b></td>
  </tr>
</table>

Table 3: Comparison of different methods in accuracy (Acc.), running time (RT), and cost. $0.5(Q) + $1.5(A) means that input 1 million tokens cost $0.5 and output (response) 1 million tokens cost $1.5. O=Open source (RTX4090D for running). P=Proprietary.

trained weights with default hyperparameters. For the proprietary closed-source LLM-based methods, we conduct experiments by calling the API through the official website (More details in Appendix D). As shown in Fig. 4, we use the same instructions to ensure a fair comparison. The QAEval framework comprises an extraction model (Qwen2.5-0.5B\(^2\)), a rule-based model (Algorithm 1), a mixture of evaluators model (ALBERT\(^3\) as evaluator). We train QAEval for 10 epochs with a learning rate of 1e-6. Hyper-parameter statistics are shown in Table 2.

5.3 Evaluation

In our experiments, “Accuracy” measures whether the final prediction scores match the true labels, while “Score” serves as a complementary metric to assess the gap between each method’s total evaluation score and the true total score, which is 51.6% in the test set. Since there is no publicly available test set, we evaluate performance using QAScore-Test with 1,000 samples.

6 Results

We aim to develop an accurate and efficient QA scoring method with minimal computational cost. Therefore, our evaluation focuses on accuracy, running time, and cost-effectiveness. As shown in

\footnotetext{2https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct}
\footnotetext{3https://huggingface.co/albert}

Table 3, QAEval achieves the highest 92.3% accuracy (with 97.06%/477 in our rule-based model and 87.95%/523 in our MOE model) among all methods, closely approaching human-level performance (99.1%). Unlike LLM-based methods, QAEval does not rely on external API calls, eliminating additional costs. Although its execution time surpasses LLM-based methods with comparable parameter sizes and rule-based methods due to the complexity of its hybrid framework, QAEval exceeds these methods in accuracy by a significant margin.

Rule-based methods like Accuracy and Relaxed-accuracy show high efficiency, completing a single evaluation in approximately 4 seconds without requiring an answer extraction model. However, their accuracy falls below 50%, highlighting their limitations in handling diverse QA tasks using predefined rules alone. The BEM and PEDANTS methods perform better, but there is still a significant gap with the QAEval (-22.1% PEDANTS, and -28.8% BEM). Both models perform significantly worse in the evaluation of complex QA samples and are not flexible enough to adapt to diverse QA samples, especially for samples with very long responses but short answers. LLM-based methods exhibit strong accuracy, with Charxiv achieving 90.6%. However, their high API costs and time constraints limit scalability. Smaller LLMs perform poorly under default settings, as seen with Qwen2.5-0.5B, which
Question: What happens if caterpillars decrease? Choices: (A) plants decrease (B) plants increase (C) nothing happens (D) none of the above
Answer: plants increase

with DLBO   w/o DLBO

Question: Can you help me determine the x-coordinate positions of each peak in the image?
Answer: 360

with DLBO   w/o DLBO

Figure 5: Visualization of the effect of DLBO on the gating network. X-axis denotes the indexes of evaluators; Y-axis denotes the number of training epochs. Red and yellow numbers indicate the top2 selected evaluators.

<table>
  <tr>
    <th rowspan="2">1</th>
    <th colspan="4">Instruction Settings</th>
    <th rowspan="2">Ground Truth</th>
  </tr>
  <tr>
    <th>2</th>
    <th>3</th>
    <th>4</th>
    <th>5</th>
  </tr>
  <tr>
    <td>Acc.</td>
    <td>79.9</td>
    <td>85.5</td>
    <td>77.1</td>
    <td>79.2</td>
    <td>79.4</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Score</td>
    <td>47.8</td>
    <td>52.6</td>
    <td>52.1</td>
    <td>50.1</td>
    <td>51.9</td>
    <td>51.6</td>
  </tr>
</table>

Table 4: Claude-3 performance by different instructions.

<table>
  <tr>
    <th>Version</th>
    <th>Accuracy</th>
    <th>Score</th>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <td>GPT-4o-0513</td>
    <td>81.7</td>
    <td>48.8</td>
  </tr>
  <tr>
    <td></td>
    <td>GPT-4o-0816</td>
    <td>87.5</td>
    <td>51.2</td>
  </tr>
</table>

Table 5: Different GPT-4o version comparison

attains only 44.5% accuracy. After fine-tuning, its accuracy improves to 58.1%, demonstrating the effectiveness of task-specific fine-tuning for smaller models in the evaluation task.

6.1 LLM-based solution analysis

LLM-based scoring lacks robustness to instruction variations. To assess the robustness and consistency of LLMs in QA scoring, we conduct experiments using the Claude-3 model with five semantically equivalent but differently phrased instructions (provided in Appendix E). As shown in Table 4, the model exhibits notable sensitivity to instruction phrasing, with accuracy varying between 77.1% and 85.5%, a gap of 8.4%. Similarly, the final scores ranged from 47.8% to 52.6%, differing by 4.8%. These results indicate that even minor modifications in instruction wording can significantly impact scoring outcomes, highlighting the lack of robustness in LLM-based scoring methods.

Scoring performance varies across LLM versions. LLM-based evaluation methods often depend on proprietary APIs, and different model versions can yield substantially different results. To examine this effect, we compare two versions of the GPT-4o model (0513 and 0816) on the QA scoring task, as detailed in Table 5. The GPT-4o-0816 version achieves an accuracy of 87.5%, outperforming the 0513 version (81.7%) by 5.8%. These findings underscore the importance of version stability in ensuring consistent evaluation results.

<table>
  <tr>
    <th>Methods</th>
    <th>RS (%)</th>
    <th>ES (%)</th>
    <th>Imp. (%)</th>
  </tr>
  <tr>
    <td>Accuracy</td>
    <td>47.5</td>
    <td>79.6</td>
    <td>(↑)32.1</td>
  </tr>
  <tr>
    <td>Relaxed-accuracy</td>
    <td>47.6</td>
    <td>87.9</td>
    <td>(↑)40.3</td>
  </tr>
  <tr>
    <td>ChartX(GPT-3.5)</td>
    <td>64.4</td>
    <td>67.1</td>
    <td>(↑)2.7</td>
  </tr>
  <tr>
    <td>Claude-3</td>
    <td>85.5</td>
    <td>88.1</td>
    <td>(↑)2.6</td>
  </tr>
  <tr>
    <td>ChartX(GPT-4o)</td>
    <td>87.5</td>
    <td>90.6</td>
    <td>(↑)3.1</td>
  </tr>
</table>

Table 6: Effect of key information extraction on scoring results (Acc.). RS denotes using response and true answer to score; ES denotes using extracted answer and true answer to score; Imp. denotes improvements.

6.2 Extraction and DLBO utility analysis

Answer extraction significantly enhances scoring performance. This experiment evaluates the impact of a paradigm that extracts concise answers from lengthy responses before scoring them against true answers. In Table 6, results demonstrate substantial performance gains across all evaluation methods. For LLM-based methods, extraction improves accuracy by 2–3%, while rule-based methods experience a notable 40.3% increase in Relaxed-accuracy, achieving a final accuracy of 87.9%, comparable to leading LLM-based methods. These findings indicate that integrating an answer extraction model with rule-based method enhances QA scoring accuracy, simplifies processing of lengthy responses, and improves overall reliability, validating the effectiveness of our method.
Figure 6: The impact of the number of evaluators, the number of selected evaluators (Top-\( k \)) for final decision-making, and different sizes of extraction models on scoring results and runtime.

DLBO enhances specialized knowledge distribution among evaluators. To prevent over-reliance on a single evaluator for all samples, we introduce DLBO, which encourages diverse evaluator specialization. A case study visualizing the gating network’s output during different training epochs (8 evaluators) highlights this effect in Fig. 5. DLBO assigns tasks to Evaluators 2 (left) and 4 (right), whereas without DLBO, the model consistently relies on Evaluator 1. This demonstrates that DLBO enables a more balanced and task-specific allocation of evaluators in our MOE architecture, improving the robustness of the evaluation process.

6.3 Hyperparameter analysis

Number of evaluators. Our findings indicate that increasing the number of evaluators improves performance when there are fewer evaluators (see Fig. 6 left). However, beyond 8 evaluators, the performance gains become marginal, while the computational cost increases. This occurs because, with more than 8 evaluators, the gating network faces difficulty in effectively distinguishing the roles of each evaluator, resulting in diminishing returns in performance.

Number of selected evaluators (Top-\( k \)). We further examine the impact of the number of evaluators selected by the gating network in generating the final output (see Fig. 6 middle). Selecting only one evaluator results in incorrect predictions for a small subset of complex samples, due to a reduction in robustness. However, due to the similarity and redundancy of the outputs, as more than two evaluators are selected, the accuracy stabilizes, with little improvement beyond this point. Conversely, the computational time increases substantially as the number of evaluators grows, highlighting a trade-off between accuracy and efficiency.

6.4 Model scaling analysis

Given that LLMs often enhance performance through an increase in parameters, we conduct experiments with the Qwen-2.5 model, testing variants with 0.5B, 1.5B, and 3B parameters. The results reveal that while a higher number of parameters yields some performance improvement, the gains are marginal relative to the increase in computational resource requirements, which negatively impacts the efficiency (see Fig. 6 right). These findings suggest that high-quality data annotations, like QAExtract, are more crucial than model size in improving QAEval performance, stressing the value of effective labeling over model complexity.

7 Conclusion

This work tackles the limitations of current QA evaluation methods by introducing QAEval, which effectively combines the reliability of rule-based approaches with the adaptability of LLM-based models. Our approach demonstrates superior performance relative to advanced LLMs, while significantly reducing computational costs. In addition, the QAExtract and QAScore datasets provide training and evaluation resources for QA evaluation.

Limitations

While QAEval solves QA tasks with unambiguous answers effectively, e.g., visual QA, math problem solving, textbook QA, it is limited when applied to tasks involving ambiguous or subjective responses, such as literary dialogues, multi-turn conversations, and descriptive generation. In these tasks, answers are often not unique and may vary depending on context or interpretation. As a result, the quality evaluation of such responses is more prone to subjective influences. Therefore, further works are needed for ambiguous QA tasks.
References

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3.

Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022. Tomayto, tomato. beyond token-level answer equivalence for question answering evaluation. arXiv preprint arXiv:2202.07654.

Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2024. A survey on mixture of experts. arXiv preprint arXiv:2407.06204.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1–45.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Intervl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198.

Cheng-Han Chiang and Hung-yi Lee. 2023. A closer look into automatic evaluation using large language models. arXiv preprint arXiv:2310.05657.

Dongyang Fan, Bettina Messmer, and Martin Jaggi. 2024. Towards an empirical understanding of moe design choices. arXiv preprint arXiv:2402.13089.

Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, and Tao Lin. 2025. Dynamic mixture of experts: An auto-tuning approach for efficient transformer models. Preprint, arXiv:2405.14297.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088.

Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656.

Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300.

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535.

Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Palioras. 2023. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170.

Zhenzhong Lan. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.

Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, and Jordan Lee Boyd-Graber. 2024. Pedants: Cheap but effective and interpretable answer equivalence. arXiv preprint arXiv:2402.11161.

Qian Liu, Rui Mao, Xiubo Geng, and Erik Cambria. 2023. Semantic matching in machine reading comprehension: An empirical study. Information Processing and Management, 60(2):103145.

Jie Ma, Pinghui Wang, Dechen Kong, Zewei Wang, Jun Liu, Hongbin Pei, and Junzhou Zhao. 2024. Robust visual question answering: Datasets, methods, and future challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence.

Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria. 2024. GPTEval: A survey on assessments of ChatGPT and GPT-4. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), page 7844–7866, Torino, Italia. ELRA and ICCL.
Rui Mao, Qian Liu, Kai He, Wei Li, and Erik Cambria. 2023. The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection. IEEE Transactions on Affective Computing, 14(3):1743–1753.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.

Nitesh Methani, Pritiha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536.

Bolanle Ojokoh and Emmanuel Adebisi. 2018. A review of question answering systems. Journal of Web Engineering, 17(8):717–758.

OpenAI. 2024. GPT-4o mini: advancing cost-efficient intelligence.

Shivanshu Shekhar, Tanishq Dubey, Koyel Mukherjee, Apoorv Saxena, Atharv Tyagi, and Nishanth Kotla. 2024. Towards optimizing the costs of llm usage. arXiv preprint arXiv:2402.01742.

Gemini Team, Petko Georgiev, Ying Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, et al. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

Qwen Team. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikael Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, and Zhihui Xie. 2024b. Reka core, flash, and edge: A series of powerful multimodal language models. Preprint, arXiv:2404.12387.

Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796.

Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. 2024. Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407.10817.

Heng Wang, Tan Yue, Xiang Ye, Zihang He, Bohan Li, and Yong Li. 2022. Revisit finetuning strategy for few-shot learning to transfer the emdeddings. In The Eleventh International Conference on Learning Representations.

Yifei Wang. 2023. Deciphering the enigma: A deep dive into understanding and interpreting llm outputs. Authorea Preprints.

Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. 2024. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406.18521.

Anuradha Welivita and Pearl Pu. 2023. A survey of consumer health question answering systems. Ai Magazine, 44(4):482–507.

Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. 2024. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185.

Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. A critical evaluation of evaluations for long-form question answering. arXiv preprint arXiv:2305.18201.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huaron Zhou, Zhenheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. MiniCPM-V: A GPT-4V level MLLM on your phone. arXiv preprint 2408.01800.

Tan Yue, Yong Li, and Zonghai Hu. 2021. Dwsa: An intelligent document structural analysis model for information extraction and data mining. Electronics, 10(19):2443.

Tan Yue, Rui Mao, Heng Wang, Zonghai Hu, and Erik Cambria. 2023. KnowleNet: Knowledge fusion network for multimodal sarcasm detection. Information Fusion, 100:101921.

Tan Yue, Xuzhao Shi, Rui Mao, Zonghai Hu, and Erik Cambria. 2024. Sarcnet: a multilingual multimodal sarcasm detection dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14325–14335.

Munazza Zaib, Wei Emma Zhang, Quan Z Sheng, Adnan Mahmood, and Yang Zhang. 2022. Conversational question answering: A survey. Knowledge and Information Systems, 64(12):3151–3195.

Luyao Zhu, Rui Mao, Erik Cambria, and Bernard J. Jansen. 2024. Neurosymbolic AI for personalized sentiment analysis. In Proceedings of International Conference on Human-Computer Interaction (HCII), pages 269–290.
A  Details of datasets

A.1 Collection
As shown in Table A.1, to efficiently train our extraction and scoring models, we build two high-quality manually labeled datasets: QAExtract and QAScore, which are sampled from 31 widely-used QA-related datasets covering a wide range of domains, including textbook question answering, visual question answering, figure question answering, etc (shown in Table A.2). We show the sample in QAScore dataset in Fig. A.1.

<table>
  <tr>
    <th>Dataset</th>
    <th>Que.</th>
    <th>Res.</th>
    <th>Ext.A</th>
    <th>Tru. A</th>
  </tr>
  <tr>
    <td>QAExtract</td>
    <td>19.71</td>
    <td>37.05</td>
    <td>1.65</td>
    <td>-</td>
  </tr>
  <tr>
    <td>QAScore</td>
    <td>18.16</td>
    <td>39.13</td>
    <td>-</td>
    <td>1.61</td>
  </tr>
</table>

Table A.1: The average token statistics.

A.2 Annotation
A total of 6 annotators participate in the dataset annotation (3 for QAExtract dataset and 3 for QAScore dataset). The annotators, all native Chinese speakers from top universities, possessed advanced English proficiency (CET-6). The annotators have expert experience in the field of NLP and QA, and we make the annotation rules for both QAExtract and QAScore datasets as detailed instructions for annotators. The average hourly payment for each annotator is 129 CNY, exceeding the minimum wage. Additionally, annotators are given a 10-minute break after 30 minutes of work.

The dataset and code will be open-sourced under the MIT License. The data used in this study are derived from publicly available datasets and it has been ensured that the relevant data use policies and privacy protections have been followed. The data will provide support to the academic field and the use has been explained in detail in the instructions.

B  Details of the models used to generate responses

Claude-3.5-Sonnet (Anthropic, 2024) model performs well in reasoning, knowledge acquisition, and coding skills. The model has strong visual processing capabilities and is able to interpret charts and graphs.

Gemini-1.5-Pro (Team et al., 2024a) is a multi-modal model from Google with a sparse mixture of experts (MoE) architecture with a contextual window for processing up to millions of tokens for complex tasks requiring advanced reasoning and analysis.

Qwen-VL-Plus (Bai et al., 2023) is an enhanced version of the visual language model introduced by Alibaba, which significantly improves the recognition of image details and text, and excels in a wide range of visual tasks.

Qwen-VL-Max (Bai et al., 2023) is an ultra-large-scale visual language model from Alibaba that excels in visual reasoning and command following capabilities, supports processing of ultra-megapixel high-resolution images, and is capable of accurately recognizing image details and text.

Phi-3-Vision (Abdin et al., 2024) is a lightweight, multimodal state-of-the-art model supporting 128K context length, trained on high-quality textual and visual inference data, and rigorously supervised fine-tuned and preference-optimized for excellent command adherence and security.

InternVL2-76B (Chen et al., 2024) is a multimodal

<table>
  <tr>
    <th>Dataset</th>
    <th>Year</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>GEOS</td>
    <td>2015</td>
    <td>Geometry Problem</td>
  </tr>
  <tr>
    <td>VQA-AS</td>
    <td>2015</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>AI2D</td>
    <td>2016</td>
    <td>Textbook Question Answering</td>
  </tr>
  <tr>
    <td>FigureQA</td>
    <td>2017</td>
    <td>Figure Question Answering</td>
  </tr>
  <tr>
    <td>TQA</td>
    <td>2017</td>
    <td>Textbook Question Answering</td>
  </tr>
  <tr>
    <td>VQA2.0</td>
    <td>2017</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>DVQA</td>
    <td>2018</td>
    <td>Figure Question Answering</td>
  </tr>
  <tr>
    <td>VizWiz</td>
    <td>2018</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>VQA-RAD</td>
    <td>2018</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>KVQA</td>
    <td>2019</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>TextVQA</td>
    <td>2019</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>PlotQA</td>
    <td>2020</td>
    <td>Figure Question Answering</td>
  </tr>
  <tr>
    <td>Geometry3K</td>
    <td>2021</td>
    <td>Geometry Problem</td>
  </tr>
  <tr>
    <td>IconQA</td>
    <td>2021</td>
    <td>Math Problem</td>
  </tr>
  <tr>
    <td>GeoQA+</td>
    <td>2022</td>
    <td>Geometry Problem</td>
  </tr>
  <tr>
    <td>UniGeo</td>
    <td>2022</td>
    <td>Geometry Problem</td>
  </tr>
  <tr>
    <td>CLEVR-Math</td>
    <td>2022</td>
    <td>Math Problem</td>
  </tr>
  <tr>
    <td>ChartQA</td>
    <td>2022</td>
    <td>Figure Question Answering</td>
  </tr>
  <tr>
    <td>MapQA</td>
    <td>2022</td>
    <td>Figure Question Answering</td>
  </tr>
  <tr>
    <td>DocVQA</td>
    <td>2022</td>
    <td>Figure Question Answering</td>
  </tr>
  <tr>
    <td>ScienceQA</td>
    <td>2022</td>
    <td>Textbook Question Answering</td>
  </tr>
  <tr>
    <td>A-OKVQA</td>
    <td>2022</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>ParsVQA-Caps</td>
    <td>2022</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>TabMWP</td>
    <td>2023</td>
    <td>Math Problem</td>
  </tr>
  <tr>
    <td>SciBench</td>
    <td>2023</td>
    <td>Textbook Question Answering</td>
  </tr>
  <tr>
    <td>TheoremQA</td>
    <td>2023</td>
    <td>Textbook Question Answering</td>
  </tr>
  <tr>
    <td>PMC-VQA</td>
    <td>2023</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>Super-CLEVR</td>
    <td>2023</td>
    <td>Visual Question Answering</td>
  </tr>
  <tr>
    <td>SciChart</td>
    <td>2024</td>
    <td>Figure Question Answering</td>
  </tr>
  <tr>
    <td>Charvix</td>
    <td>2024</td>
    <td>Figure Question Answering</td>
  </tr>
  <tr>
    <td>MathVista</td>
    <td>2024</td>
    <td>Figure Question Answering</td>
  </tr>
</table>

Table A.2: Datasets used for developing QAExtract and QAScore.
"question": "How does the peak of the probability density change as kurtosis value becomes higher?",
"response": "As the kurtosis value becomes higher, the peak of the probability density becomes taller and sharper.",
"extracted_answer": "taller and sharper",
"true_answer": "higher peak",
"score": 1

Figure A.1: The sample in QAScore dataset.

large language model with capabilities of document and chart comprehension, infographics QA, scene text understanding, etc.
MiniCPM-V2.5 (Yao et al., 2024) is the latest multimodal large model of MiniCPM-V series, supporting more than 30 languages, with excellent document understanding, OCR extraction, complex reasoning, etc., while optimized for efficient edge device deployment.
Reka-Core (Team et al., 2024b) is a 67 billion-parameter multimodal language model supporting 32 languages with the ability to process text, images, video and audio for complex application scenarios and model distillation.
GPT-4V-Turbo (Achiam et al., 2023) is a multimodal model from OpenAI that combines natural language processing and visual understanding capabilities to analyze images and answer related questions. The model offers a significant increase in processing speed over GPT-4.
GPT-4o (Achiam et al., 2023) is a SOTA multimodal model that can process multimodal inputs (e.g., text and images) as well as perform excellent understanding and reasoning.

C Methods

C.1 Rule-based methods

Accuracy is a metric that measures the consistency between predicted and target answers, calculated as the ratio of correctly predicted samples to the total number of samples. The formula is:

\[
\text{Accuracy} = \frac{N_{\text{correct}}}{N_{\text{total}}},
\]

where \( N_{\text{correct}} \) denotes the number of samples where the predicted answer matches the target answer exactly, and \( N_{\text{total}} \) represents the total number of samples.

Relaxed-accuracy provides an additional tolerance for errors in numerical answers. For example, with a tolerance of 5%, 102 (Predict answer) = 100 (True answer).

\[
\text{Relax-accuracy} = \frac{N_{\text{relaxed-correct}}}{N_{\text{total}}},
\]

where \( N_{\text{relaxed-correct}} \) denotes the number of samples where the predicted numerical answer \( \hat{y} \) satisfies:

\[
|\hat{y} - y| \leq \epsilon \cdot y,
\]

with \( y \) being the true answer and \( \epsilon \) the allowed tolerance fraction (e.g., \( \epsilon = 0.05 \) for 5%). \( N_{\text{total}} \) represents the total number of samples.
BEM (Bulian et al., 2022) introduces a rule-based evaluation method that maps answers into predefined equivalence classes based on transformations such as numerical formatting, unit conversions, and common paraphrases, enabling more flexible matching beyond token-level accuracy.
PEDANTS (Li et al., 2024) proposes a lightweight rule-based system that performs answer equivalence checking through manually designed transformations and pattern matching, aiming to achieve interpretable and cost-efficient evaluation.

C.2 LLM-based methods

Qwen2.5-0.5B (Team, 2025) is the base model of the Qwen2.5 family with 500 million parameters and a decoder-only architecture. The model is pre-trained on large-scale multilingual and multimodal datasets with multilingual support and structured data comprehension.
Claude-3-Haiku (Anthropic, 2024) model has multimodal capabilities and is able to fast and efficiently process visual and textual tasks.
Gemini-1.5-Flash (Team et al., 2024a) is an efficient multimodal pre-training model introduced by Google that aims to improve the performance of visual and language understanding tasks through fast reasoning and low computational cost.
GPT-4o-mini (OpenAI, 2024) is a lightweight version of the GPT-4o model, designed to provide multimodal capabilities in a more efficient model. It maintains a better ability to process multimodal data and greatly reduces the model size.
GPT-4o (Achiam et al., 2023) is a SOTA multi-modal model that can process multimodal inputs (e.g., text and images) as well as perform excellent understanding and reasoning.

ChartX (Xia et al., 2024) designs the GPT-acc evaluation method for tasks with clear answers such as QA, which uses the GPT-4 model to evaluate responses and true answers and outputs binary classification results (True/False). Similar to relaxed-accuracy, GPT-acc sets a 5% error margin for numerical answers.

Charxiv (Wang et al., 2024) uses the GPT-4o model to evaluate model responses and true answers, designed with instructions that first ask the model to extract short answers from long responses and then evaluate them with true answers.

D Experimental settings

In this study, for different types of methods, we have given careful consideration to the experimental setup to ensure that each method is comparable under a fair evaluation framework.

Rule-based methods: based on the formulas in Appendix C.1, we implement the Accuracy and Relaxed-accuracy assessment methods. We set a 5% tolerance for relaxed-accuracy.

Proprietary LLM-based methods: for the experiments of proprietary LLM-based methods, including Claude-3, Gemini-1.5, GPT-4o-mini and GPT-4o, we conduct experiments by calling the API through the official website. The official recommended hyperparameter settings are strictly followed for each model. ChartX4 and Charxiv5 are based on GPT-4 and GPT-4o, respectively. We conducted experiments using the code, instructions and parameters provided in their papers.

Open-source LLM-based methods: for Qwen2.5-0.5B, we use publicly available code and pre-trained weights (downloaded from Huggingface6) with default hyperparameters. And for Qwen 2.5-0.5B(FT), we use the LoRA method (Hu et al., 2021) to fine-tune the model on the QAScore training dataset with 10 epochs.

E Instructions for robustness testing

Given that current LLM-based methods typically exhibit high sensitivity to instruction input, i.e., instructions with different expressions may lead to significantly different evaluation results, we design and implement a multi-instruction experiment. Specifically, we design five semantically consistent instructions that differ in terms of expression structure, wording choice, and linguistic style to systematically evaluate the stability and robustness of the model under different instruction expressions.

These instructions cover different linguistic features, including concise, lengthy, and language, etc., so as to simulate as much as possible the diversified forms of instruction inputs that the model may face in real application scenarios. Examples of the instructions are shown in Fig. A.2, and the model outputs under different instructions are compared through multiple rounds of experiments to further analyze their impact on the model performance and potential adaptive capability.

<table>
  <tr>
    <th>Instruction1</th>
    <td>将每个回复与问题的真实答案进行比较，评分规则如下：正确记为“1”，预测答案部分符合真实答案记为“0.5”，无关或完全错误的答案记为“0”。特别提醒，数值型答案允许有最多5%的误差。</td>
  </tr>
  <tr>
    <th>Instruction2</th>
    <td>Given multiple question-answer pairs and the corresponding predictions, evaluate the correctness of predictions. The output should be only "0", "0.5", or "1". Note that if the groundtruth answer is a numeric value with/without the unit, impose 5% error tolerance to the answer. "0.5" should be given if multiple correct answers are partially correct, but incorrect or extra answers should be scored as "0".</td>
  </tr>
  <tr>
    <th>Instruction3</th>
    <td>对每个预测结果进行评分：完全正确得“1”，部分正确得“0.5”，完全错误得“0”。数值答案允许5%的误差范围内视为正确。</td>
  </tr>
  <tr>
    <th>Instruction4</th>
    <td>One of three categorization levels is output, depending on how well the answer and the predicted response match: “1” for a perfect match, “0.5” for partially correct, and “0” for completely incorrect. If the answer is of a numerical type, a margin of error of 5% is allowed.</td>
  </tr>
  <tr>
    <th>Instruction5</th>
    <td>According to the question and answer, please follow the steps below to evaluate the correctness of the response.(1) Score “1”. if the response is correct compared with the true answer.(2) “0.5” means that the response is partially correct when the true answer contains multiple answers.(3) “0” for a completely incorrect response. Note in particular that for numerical answers, the margin of error must not exceed 5%.</td>
  </tr>
</table>

Figure A.2: Different instructions for robustness testing.

Reviewer paper summaries: The paper proposes QAEval, a method that uses a three-step pipeline to evaluate QA tasks. Specifically, the authors uses a finetuned model to extract and rephrase answers, then use a quick scoring method to judge correctness and MoE as the final step for a more robust QA evaluation. QAEval uses 0.6B models that are more efficient and effective than expensive and slow API-based LLMs.
Summary of strengths: - The paper introduces an efficient and effective LLM-based QA evaluation pipeline.

- The paper studies an important problem to improve the efficiency and robustness of QA evaluations.
Summary of weaknesses: - The paper has many claims about the unreliability of LLM evaluators without citing any related work and papers in the related work section (particularly line 150-163).

- Method section is unclear and can be improved with greater clarity. The method specifically discusses rule-based quick scoring to capture varying degrees of answer correctness. However, the rules to determine quick correctness is not mentioned nor defined. Did you use relaxed exact match, ROUGE to do those quick scoring? What rules did you use to determine the scores. A score of 0.5 or 1 is deemed a match, how did you determine which score to give to the predicted answer? Adding formal definitions of rules and methods to can help clarify step 2 in the pipeline. 

- The paper discusses efficiency and performance of a metric a lot. The paper shows that naive rule-based methods is the most efficient but worse than LLM-based approaches. QAEval contains several 0.5B models, which is smallest compared to other LLM-based methods thus it is more efficient. However, there are two other related works using even smaller and faster non-LLM models and their models are trained to evaluate answer correctness [1][2]. It would be helpful to see the results to show their runtime, cost, and accuracy against naive rule-based methods and LLM-based methods. These two works also have formal definitions of answer correctness. Are these methods better or worst than the original 0.5B QWen model? Adding these two methods as additional baselines other than naive rule-based methods can make the argument 'finetuning a 0.6B LLM is necessary for a more efficient and robust evaluation' stronger.

- The answer extraction pipeline is relatively straightforward and an easy task, does using a more efficient model like distilbert affect the overall evaluation performance? 

[1] Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation: https://arxiv.org/abs/2202.07654

[2] PEDANTS: Cheap but Effective and Interpretable Answer Equivalence: https://arxiv.org/abs/2402.11161
Comments, suggestions and typos: - The curated QA-Extract dataset is not clearly discussed in line 194 as its first time to be discussed (Could at least refer to section 4.3 at the end of the sentence).

- I'm willing to defend and review my reviews.
Overall assessment scores: 3
Metareview score: 3.5
Metareview: The paper introduces QAEval, a three-step evaluation pipeline for question-answering (QA). The pipeline consists of: (1) extracting short answers from long generated responses using a fine-tuned Qwen2.5-0.5B model, (2) fast scoring these answers using rule-based approach, and (3) applying a Mixture of Experts (Evaluators) for robust evaluation. QAEval is demonstrated to be efficient and effective. Experiments highlight its high efficiency and robustness compared to using larger LLMs accessed via APIs.
Summary of reasons to publish: 1/ The proposed three-step evaluation pipeline is novel. The large-scale datasets, QAExtract and QAScore, can be valuable resources for the broader research community, both for QA evaluation and LLM development.

2/ The experimental design is comprehensive and robust, including dedicated ablation studies.